I wanted a Sentry I could open the hood on
A few years before ErrSight existed, I spent a bad afternoon staring at a grouped issue in a hosted error tracker, absolutely certain it was wrong. Two completely different exceptions, from two different code paths, had been folded into the same issue. The occurrences counter said fourteen thousand. The page was slow to load. And I had no way to know why the tool had grouped those two crashes together, because the grouping logic lived on someone else’s servers, behind a pricing page.
That is the moment I keep coming back to. Not the bill that shows up after your worst Tuesday (I wrote about that pricing problem separately) but the simpler, more stubborn itch of wanting an error tracker whose every decision I could read, change, and trust. I wanted Sentry-shaped ergonomics: send me your exceptions, group them into issues, alert me, let me tail them live. And I wanted to own the whole thing, top to bottom, so that when something grouped wrong, or counted wrong, or alerted twice, the answer was a file I could open instead of a support ticket.
So I built ErrSight. This is the long version of how: the motivation, the roadblocks that actually hurt, and the unglamorous pre- and post-deployment lessons that nobody puts on a landing page. I am going to be honest about the trade-offs, because the interesting parts of this system are all trade-offs.
The shape of the system
ErrSight is not one program. It grew into four cooperating ones, and the seams between them are where most of this post lives.
At the center is a Rails monolith: the product. It owns the dashboard, billing through Dodo Payments, the issues UI, the alert pipeline, retention and GDPR, and the original POST /api/v1/events ingestion endpoint. Error events land in a TimescaleDB hypertable partitioned on time, with denormalized issues and usages tables in front of the hot read paths. When that Rails ingestion path became the throughput bottleneck, I peeled it off into a Rust events-processor sidecar (axum, sqlx, tokio) that does nothing but ingest, writing into the exact same Postgres tables so the two paths coexist. The live tail started out riding Rails ActionCable; I later moved it into a small Rust/Loco service built on a tokio::broadcast channel and Server-Sent Events, with no WebSockets, no Node, and no Redis. And feeding all of it is a fleet of eight client SDKs (Ruby, browser JS, Node, Python, Go, Rust, React Native, and Swift), each a thin, non-blocking shipper for one wire protocol.
Rails stays the product; the hot paths got peeled off into Rust. The unifying idea is that one Rails controller is the authoritative wire parser, and everything else either feeds it or faithfully clones it. Hold onto that, because the hardest bugs were all about two systems disagreeing on something they were supposed to agree on.
Roadblock 1: the Rails ingestion path topped out, and the rewrite was scarier than the bottleneck
The first real wall was throughput. The Rails write path is correct and I am fond of it, but it is expensive. Per event it does somewhere between eight and eleven database round-trips: an auth query, a rate-limit UPSERT, a quota reservation that on its own is an advisory lock plus a SELECT plus a UPSERT plus a counter UPDATE, a Solid Queue job INSERT, and then roughly six more operations when that job dequeues and the event actually persists. Stack Ruby’s interpreter overhead and ActiveRecord object churn on top, and sustained ingestion caps out around 2,000–2,500 events a second.
The naive fix is “throw hardware at Postgres,” but the bottleneck was never Postgres: it was the number of times Ruby talked to Postgres. So I wrote events-processor in Rust to collapse those eight-to-eleven round-trips down to roughly one or two, amortised, per event. The general patterns (how a Rust service reads and writes a Rails-owned Postgres database without corrupting it, and how it authenticates against the Rails-owned identity) I wrote up separately in wrapping a Rails-owned Postgres database and sharing authentication between Rails and a Rust service. The ErrSight-specific tricks are below, and they’re the part that doesn’t appear anywhere else.
The first is to amortise the auth. A sixty-second in-process cache (fifty thousand entries) gets a hit rate north of 99.9%, and the single JOIN that fills it snapshots more than the API key: it also captures the drop-reason (paused, over events, over storage) and the active add-on credits, so a cache hit enforces quota with zero database operations.
The second, and the biggest win, is a local quota reservoir. Instead of one quota transaction per request, the Rust service pre-reserves chunks of a thousand events per organization against the monthly limit, then decrements a local counter under a mutex. On a cache hit, the reservation costs about a microsecond. It eliminates the advisory-lock transaction that dominated the Rails per-request cost.
The obvious objection to a local reservoir is: doesn’t that blow past the customer’s hard cap? It can’t, and the reason is the part I’m proudest of. Every database top-up reads the remaining headroom under the same org-and-period advisory lock the Rails request path uses, the no-overages lock from the other post, and grants only what’s actually left:
let headroom = events_limit - current_total;
if headroom <= 0 { return Ok(0); }
let granted = headroom.min(want);
The reservoir can over-_reserve_, charging up to one chunk early, but it can never over-_ingest_. On a clean shutdown it returns the unused remainder; an ungraceful kill -9 over-charges by at most one chunk per org until a Rails reconciliation job corrects it. I name that failure mode in the doc-comments rather than pretend it doesn’t exist.
The third trick is to batch the writes. A sharded set of writer tasks buffers five hundred events or fifty milliseconds, whichever comes first, and persists them in a single multi-row INSERT ... SELECT FROM UNNEST(...) over sixteen typed Postgres arrays: one round-trip instead of five hundred. And the fourth is to lock per project, not per event. Rails takes a pg_advisory_xact_lock per ingestion_id for idempotency; doing that for hundreds of events in one flush would hammer Postgres’s shared lock table, so the Rust path takes one lock per distinct project in the batch:
// Sorting the keys gives every concurrent flush one acquisition order,
// so two flushes can never deadlock against each other.
lock_keys.sort_unstable();
lock_keys.dedup();
sqlx::query("SELECT pg_advisory_xact_lock(t.k) FROM unnest($1::bigint[]) AS t(k)")
.bind(&lock_keys)
.execute(&mut *tx)
.await?;
The net result is roughly 20,000–25,000 sustained events a second, about ten times the Rails path, with p50 acknowledgement latency down from around ten milliseconds to under one.
The trade-off is real, and it has two halves. The first is that I now maintain the same domain logic (fingerprinting, regression detection, issue aggregation) in two languages, and they have to stay byte-identical or the same crash splits into two issues across the two write paths. The Rust service hand-rolls Ruby’s exact Zlib.crc32(s) % 2**31 so both runtimes serialize on the same advisory lock; get that hash wrong by one bit and the two paths stop agreeing about whose turn it is. The Rust writer even reproduces a Rails quirk on purpose: it drops blank-message events in its own pre-filter, because Rails validates message presence and silently drops them at save, and “faithfully clone Rails” has to include Rails’ silences.
The second half is the one I’d most want a reader to take away. The fast path is genuinely fast, but it is also partial: events ingested through events-processor currently fire no alerts and produce no live-tail broadcast, and bulk operations like resolve, mute, retention pruning, and GDPR erasure all remain Rails-only. The throughput didn’t come free; it came with a list of things the fast path doesn’t do yet, every one of which is a new seam to keep honest. And the bottleneck didn’t disappear so much as move, out of Ruby’s interpreter lock and into the write-ahead log. My own throughput doc says it plainly: the processor “will not magically extract 10x: Postgres needs to be sized appropriately,” and the 25k ceiling is per database, which at full tilt is 2.16 billion events a day, at which point retention pruning becomes its own problem.
A 10x rewrite that you have to keep in lockstep with the original forever is not a free lunch. It’s a second mouth to feed that happens to be very fast.
Roadblock 2: counting errors correctly is a concurrency problem in disguise
Grouping is the soul of an error tracker, and counting is where it quietly goes wrong.
Originally the issues page computed its aggregates (occurrences, first and last seen, severity, affected users) with a GROUP BY over the events hypertable on every render. That query gets linearly slower with each project’s event volume; on a busy project it was a multi-second page load. The fix was to stop counting at read time and start counting at write time: denormalize the aggregates onto an issues row per (project_id, fingerprint), maintained incrementally with one UPSERT per event.
The naive UPSERT has two bugs that only show up under load. Two threads ingesting the same fingerprint concurrently can double-count. And, this one is sneakier, a retried or backdated job carrying a stale occurred_at can overwrite a newer “last seen” message with older data. The fix leans entirely on Postgres semantics:
ON CONFLICT (project_id, fingerprint) DO UPDATE SET
occurrences_count = issues.occurrences_count + 1,
last_seen_at = GREATEST(issues.last_seen_at, EXCLUDED.last_seen_at),
first_seen_at = LEAST (issues.first_seen_at, EXCLUDED.first_seen_at),
severity = GREATEST(issues.severity, EXCLUDED.severity),
last_message = CASE
WHEN EXCLUDED.last_seen_at >= issues.last_seen_at
THEN EXCLUDED.last_message
ELSE issues.last_message
END
The row-level lock held during ON CONFLICT DO UPDATE serializes same-fingerprint upserts, so occurrences_count + 1 is race-free with no application mutex anywhere. GREATEST, LEAST, and the CASE make the whole statement order-independent: a late event can never clobber newer state. Unique-user counting uses the same family of trick: an INSERT ... ON CONFLICT DO NOTHING RETURNING id, where affected_users_count is bumped only when a row was actually inserted, so a duplicate user can’t inflate the number.
Out-of-order events are the silent killer of denormalized counters. If your UPSERT doesn’t reach for
GREATESTandLEAST, a retried job will eventually lie to you.
Then there’s the fingerprint itself, which is versioned. A v2 hash runs over the exception class, the top in-app frame’s filename, and the function, for modern SDKs that ship structured frames. A legacy hash runs over the digit-stripped message and the first backtrace line, for old ones. Both are SHA-256 truncated to thirty-two hex characters, and the v2| prefix means new and old SDKs coexist without colliding, at the cost that, mid-rollout, the same crash from an upgraded and a non-upgraded client lands in two issues until everyone’s on the new SDK.
Denormalization always owes a reconciliation debt, and this is where I pay it. Retention pruning, storage-cap eviction, and GDPR erasure all use delete_all, which skips the after_create_commit callback that maintains the aggregates, so the counts drift upward and point at events that no longer exist. Every bulk-delete path has to call Issue.reconcile_aggregates_for_fingerprints! to true them up. And one product decision is buried in that infra code: a fingerprint that drops to zero events is zeroed in place, not deleted, so a customer’s comments and triage assignment survive the underlying events being pruned. The most honest limitation lives in a comment in IssueMerger: merging two duplicate issues cleans up what you see now, but a new event with a merged-away fingerprint spawns a fresh issue again, because grouping happens at ingest from each event’s own hash. Making merges stick forward needs a fingerprint-alias table consulted on the ingest path. It’s a TODO, and the flash message says so. I’d rather ship the honest version with the caveat than pretend the button does more than it does.
Roadblock 3: a live tail, without dragging ActionCable into Rust
ErrSight has two realtime features: a global live feed on the dashboard and a per-project log viewer with a LIVE/pause toggle. In Rails these were two ActionCable channels riding solid_cable (a Postgres-backed pub/sub polling every tenth of a second) with two Stimulus controllers subscribing over WebSockets and building DOM rows from JSON on the client.
That whole stack is heavy: a pub/sub backend, the @rails/actioncable client, a Warden-authenticated WebSocket, and client-side rendering. When I moved the live tail into the Rust service, reproducing all of it felt absurd for what is, fundamentally, “push a row to a browser.” I wrote up the SSE approach in depth separately, so here I’ll stick to the ErrSight-specific wiring: I dropped ActionCable for Server-Sent Events over a single process-global tokio::broadcast channel.
fn bus() -> &'static broadcast::Sender<EventMsg> {
static BUS: OnceLock<broadcast::Sender<EventMsg>> = OnceLock::new();
BUS.get_or_init(|| broadcast::channel(512).0)
}
pub fn publish(msg: EventMsg) { let _ = bus().send(msg); }
The ingestion worker publishes each persisted event onto a 512-slot bus; two axum SSE handlers subscribe through tokio_stream::BroadcastStream, render pre-escaped HTML row fragments server-side, and the browser uses native EventSource with insertAdjacentHTML. No framework, no Node, no Redis, no separate cable database.
Two gotchas turned out to be load-bearing. The first is that authorization flipped inside-out. ActionCable rejects unauthorized subscribers up front, at subscribe time, via stream_for org. A single global bus has no per-subscriber routing (every SSE subscriber sees every event from every org), so authorization had to move from subscribe-time to stream-time, into a per-connection filter_map that drops rows the viewer isn’t allowed to see:
let stream = BroadcastStream::new(realtime::subscribe()).filter_map(move |res| match res {
Ok(msg) if Some(msg.org_id) == allowed =>
Some(Ok(Event::default().data(render_feed_row(&msg)))),
_ => None,
});
I tested that a user on a different active org receives exactly zero of another org’s events, because “every subscriber sees everything and the filter is the only thing standing between orgs” is the kind of sentence that should make you write the test. The second gotcha is that streaming server-rendered HTML reopened an XSS hole the Rails Stimulus controllers had closed by using textContent everywhere. So the Rust renderer hand-rolls a tiny esc() that escapes the five HTML-significant characters on every user-controlled field before it hits the wire.
The in-process bus bought enormous simplicity, and the price was one hard operational constraint: the publisher (the worker) and the subscriber (the web handler) must share memory, which forces a single run mode: cargo loco start --server-and-worker. Split them into separate processes and the publishes never reach the subscribers, silently. That’s exactly the kind of footgun you forget six months later, so it’s the first line of the module’s doc-comment.
Roadblock 4: TimescaleDB and Rails migrations do not get along
This one cost me staging deploys, which is the most embarrassing kind of cost.
Rails’ default :ruby schema format (db/schema.rb) cannot represent create_hypertable or compression DDL. So on a fresh database, db:prepare takes the schema-load path and assume_migrated_upto_version stamps the TimescaleDB migrations as already-applied without ever running them. You get a plain Postgres events table, no hypertable, no compression, and a migration report that’s entirely green. Your time-series table isn’t one, and nothing tells you.
The fix is an idempotent converger, TimescaleHypertable.ensure!, run on every web boot from the Docker entrypoint after db:prepare. It no-ops when TimescaleDB is absent (CI, a plain Postgres dev box), no-ops when the table is already a configured hypertable, and never raises, because a degraded database must not crash the web container. A boot-time health check logs loudly if the hypertable or compression policy is missing in production. ErrSight re-asserts its own hypertable shape every time the web process starts.
The gotcha that actually looped on me was CREATE INDEX CONCURRENTLY. It’s rejected on hypertables (“hypertables do not support concurrent index creation”) and my original trigram-search migration used CONCURRENTLY to avoid locking writes. Staging deploys retried that error indefinitely. The fix was a plain CREATE INDEX IF NOT EXISTS, which takes a brief per-chunk lock bounded by the number of recent uncompressed chunks rather than total event volume; acceptable at this scale, and a different conversation at a much larger one. There are a few more landmines in the same family (the primary key has to be the composite (id, occurred_at) because TimescaleDB wants the partitioning column in every unique key, and you can’t alter a column’s type while any compressed chunk exists) but the shape of the lesson is the same each time.
TimescaleDB gives me cheap, time-partitioned, compressible storage, and I’d choose it again. What it takes in return is that ordinary Rails migrations become multi-step operations and schema.rb quietly becomes a liar. I accepted a converger-on-every-boot and a health check rather than fight the schema dumper. It’s ugly. It’s also bulletproof, which on the deploy path beats elegant every time.
Roadblock 5: one wire protocol, eight SDKs, six concurrency models
Eight SDKs sound like eight HTTP clients. The actual hard part was making eight independently-evolving codebases agree on the same wire shape and the same failure semantics, while each language hands you a completely different concurrency model.
The contract is identical everywhere: POST /api/v1/events with an X-API-Key header (write keys are elp_ plus forty-eight hex characters, read keys are elr_, so a leaked write key can spam ingestion but never read stored data), a single event or an array of up to a hundred, half a megabyte max, levels debug | info | warning | error | fatal. Every SDK does the same four things its own way: non-blocking capture into a bounded queue, a background worker that batches and flushes every couple of seconds, recursive splitting of oversized batches to stay under the server cap, and on a 429, re-queueing the held events and pausing until Retry-After. The single source of truth is the Rails controller, to the point that the Go SDK’s event.go names it in a comment as the authoritative parser.
And the SDKs are emphatically not byte-identical: the server absorbs the drift. Go and Node serialize the field as timestamp; the other six emit occurred_at; and the controller just shrugs and takes either: parse_timestamp(data["timestamp"] || data["occurred_at"]). Eight SDKs, two spellings of the same field, and a one-line || that means none of it matters.
The Ruby gem paid for the lessons the other seven were designed to avoid. Its CHANGELOG reads like a crime-scene report. There was a real cross-user PII leak: set_user stored the current user on Thread.current and never cleared it, so on a long-lived Puma thread, request B’s exceptions got tagged with request A’s user, fixed by moving the scope onto a per-request frame pushed and popped by Rack middleware. There was total silent event loss under Puma’s cluster fork: the flush thread was started in Client#initialize in the primary process, POSIX threads don’t survive fork(), so every worker inherited a dead thread and a queue nothing drained, fixed by comparing Process.pid on the hot path and rebuilding the queue, mutexes, and HTTP connection on a mismatch. And there was the one that became a fleet-wide invariant: a 429 that parked the only worker for a full minute with a blocking sleep(retry_after), during which the queue overflowed and dropped the very events it was trying to protect.
@mutex.synchronize { @queue.unshift(*events) }
# A timestamp gate instead of a blocking sleep. flush! checks
# rate_limited? at the top and skips while paused, so the worker
# stays responsive and the queue keeps draining on the next window.
@rate_limited_until = Time.now + retry_after
Underneath, six concurrency models (Ruby threads, Go goroutines, Node’s AsyncLocalStorage, Python’s ContextVar plus os.register_at_fork, a bare Rust OS thread, a Swift serial DispatchQueue) all reduce to the same “queue, batch, flush, drain on exit” loop. They sit at wildly different maturity levels (most are pre-1.0, somewhere in 0.1.x to 0.3.x; Node is somehow already at 1.1.0; Ruby is 0.2.2) and they don’t share a version line. They even disagree on principle, on purpose. When a customer’s own before_send PII scrubber throws, Ruby, Python, JS, Node, and React Native fail open (a buggy filter shouldn’t silently lose production errors) while Rust fails closed by default, because there the scrubber is the privacy gate and shipping the unscrubbed original would leak, with an opt-in before_send_fail_open(true) for anyone who disagrees. That’s a one-line philosophical split between availability and privacy, and I let the fleet hold both opinions.
The React Native SDK draws the most honest boundary of all: it captures JavaScript errors only, and the README tells you to run Sentry or Crashlytics for native crashes, rather than ship a half-working native module and pretend it’s whole.
Pre-deployment: the stuff that isn’t a feature but you can’t ship without
None of what follows shows up on the pricing page. All of it would have sunk me on the public internet.
The deploy is split. The Rails monolith ships via Kamal to a single dedicated VPS (Namecheap), with TimescaleDB Community running on the box as a Kamal accessory and DATABASE_URL derived from POSTGRES_PASSWORD so the two can’t drift. The Rust events-processor runs on Railway. The one war story here: builds run off the box, in CI, because the VPS CPU lacks AVX and building on it crashed with SIGILL: the box only pulls and runs. A GitHub Actions pipeline is the thing standing between “rubocop passes locally” and “the box is on fire.”
There is also, deliberately, no Redis. Solid Queue runs inside Puma via plugin :solid_queue, and Solid Cache replaced the rest. An all-Postgres stack is one fewer thing to operate, monitor, and have go down at 3 a.m.
The security work is where I spent more time than I’d have guessed. Notably, event ingestion is not throttled by Rack::Attack (only login and signup are) because ingestion limits have to hold across many Puma workers and replicas, which a per-process limiter cannot. On top of that: Cloudflare Turnstile on signup, a per-plan hard rate-limit cap, and a dedicated program-abuse path. The one that needed the most care was webhooks. A user-configurable webhook URL is a loaded SSRF gun pointed at 169.254.169.254, the cloud metadata endpoint, so URLs are validated against private ranges at save time and re-resolve DNS at send time, dialing the raw IP to defeat DNS rebinding, with IPv4-mapped IPv6 normalized so ::ffff:127.0.0.1 can’t sneak past. Slack webhooks are hard-allowlisted to exactly hooks.slack.com. None of that is visible to a customer; all of it is the difference between a webhook feature and a proxy for attacking my own infrastructure.
GDPR got the same treatment. There’s a ninety-day discard window before a hard purge; org discard is wrapped in a transaction so a failing callback can’t leave a half-discarded org still ingesting; and purges use bulk delete_all to avoid instantiating millions of rows. My favorite detail is the smallest one: per-user erasure logs only a twelve-character SHA-256 hash of the identifier, because logging the raw email you were just asked to delete would smuggle the PII straight back into logs the erasure can’t reach.
And ErrSight watches its own health without an external stack. A pure Solid Queue QueueHealth service reads Solid Queue’s own tables (no Datadog, no Prometheus) and reports every minute. The metric that actually matters isn’t raw backlog; it’s the age of the oldest ready job, because a ten-job backlog with a ten-minute-old job (one job wedged) is scarier than a fifty-thousand-job backlog that’s one second behind.
Post-deployment: dogfooding, deletions, and ops reality
Here’s the part that’s genuinely fun: ErrSight watches its own errors with ErrSight. The Gemfile contains gem "errsight", "~> 0.2.2" (the same published SDK I sell to customers, pinned by SHA-256 in the lockfile) and the app forwards its own production exceptions to an ErrSight project.
That’s a chicken-and-egg risk. The moment you most need error reports is during an outage, which is exactly when the thing receiving them might be down, and an error in the error-reporting path could loop forever. The defense is two layers. The SDK swallows all of its own delivery failures (rescue StandardError, flush! rescue nil) so reporting to a dead server fails silently instead of cascading into the host app. And the self-instrumentation is capped at errors and switched off entirely unless LOG_APIKEY is set, so ErrSight’s own routine info/warn firehose never floods the pipeline it would be reported into:
if ENV["LOG_APIKEY"].present?
Errsight.configure do |config|
config.api_key = ENV["LOG_APIKEY"]
config.min_level = :error
end
else
Errsight.configure { |config| config.enabled = false } if defined?(Errsight)
end
There is no Sentry, no Rollbar, no Bugsnag gem anywhere in the Gemfile. No hedge, no antidote on the shelf.
Which is its own story, because there briefly was one.
For about a day, I added Sentry, to monitor my own queue health. ReportQueueHealthJob called Sentry.capture_message when the backlog crossed a threshold. I was, in other words, using a competitor’s error monitor to monitor my error monitor. The next morning I deleted it and replaced the call with a plain Rails.logger severity switch driven by the same logic. That was the moment I started eating my own dog food in earnest, and it’s the most on-the-nose thing in the whole project.
The bigger excision was Mixpanel. I built an analytics service, a TrackMixpanelEventJob, and a tracking concern threaded through about twenty controllers, then reverted the entire thing in one afternoon. Twenty-seven files, three hundred and eighty-two lines deleted, fourteen kept. Google’s gtag survived; nothing else did.
That kind of churn was the texture of the whole build. I deleted and rebuilt the background-job queue more than once during the Sidekiq-versus-Solid-Queue indecision before committing to Solid Queue for good. Two months. A hundred and eighty-seven commits. A great many additions that turned, eventually, into subtractions.
On what broke and what I’d do again: the TimescaleDB-versus-Rails friction broke staging more than once, and I’d still choose TimescaleDB. I’d just write the converger and the health check on day one instead of after the third looping deploy. The Ruby SDK’s fork bug and PII leak were genuinely alarming in hindsight, and they’re the entire reason the other seven SDKs were built correctly up front; paying for a lesson once is fine as long as you actually transfer it. The Rust rewrite was the highest-risk thing I did, and I’d do it again, while being more upfront, sooner, that it doesn’t remove the bottleneck so much as relocate it to the write-ahead log.
The ops reality of running this solo is that the unglamorous support tooling earns its keep daily. There’s an ActiveAdmin panel and a “become user” impersonation flow that stores the operator’s real id in session[:true_admin_id], shows a persistent banner, refuses to impersonate other admins, and audit-logs every become and restore. Being able to literally see the app as a specific customer is how you debug a “works on my machine” ticket when the machine is the only one there is. And the open-source community edition, under AGPLv3, is the same engine with billing, quotas, and analytics stripped out (you can watch the business model vanish in a directory diff), which I treat as a no-lock-in trust signal as much as a gift.
Wrapping up
ErrSight started as an itch: I wanted a Sentry-shaped tool whose every decision I could open and read. It turned into four programs that have to agree with each other under load: a Rails monolith, a Rust ingestion sidecar that’s ten times faster on the same database, an SSE service that replaced WebSockets with a 512-slot channel, and eight SDKs speaking one protocol in six concurrency models. The features were the easy part. The hard part was the seams: a CRC32 that has to match across two languages, a fingerprint that has to match across two write paths, an UPSERT that has to be order-independent across retries, a hypertable that Rails will silently forget to create.
If I had to compress everything above into a single line to hand the next person building something like this, it’s this:
When two systems have to agree on something, the agreement is the feature. Build it so you can check it, name the failure mode out loud, and assume the disagreement will arrive at the worst possible millisecond.
Honestly, the plumbing turned out more interesting than the product. But if it’s the product you came for, it’s running at errsight.com, watching its own errors, like everything else.