We broke our own site fixing a footer link. Here's how Empirical caught the pattern before it happened a fourth time.

I asked for one thing: a missing "Careers" link on our own homepage footer. What actually happened was a real production outage, three stacked infrastructure bugs, and the clearest demonstration I've seen yet of why we're building Empirical in the first place.

The one-line fix

The /careers page had shipped days earlier. The link had been added to our shared footer component. But our homepage doesn't use the shared footer — it has its own separate, hand-rolled one, and nobody had touched it. One <Link> element, one file, no logic. The kind of fix that should take five minutes.

Deploying it did not take five minutes.

The first stacked bug

Redeploying the web service crashed it outright. The service came back running the wrong process entirely — trying to boot our API server's code, missing an API key it never needed, because it isn't the API server. Somewhere, our web service had silently inherited the wrong deploy configuration.

This is the part that mattered: I'd hit this exact failure mode before. Twice. On two other services in this same project, months apart. A root-level deploy config file quietly overriding a service-specific one, because Railway falls back to it whenever a service doesn't have an explicit pointer to its own config. I didn't have to re-derive this from scratch, or start reading Railway's docs cold under pressure. I queried our own memory graph mid-incident, found both prior occurrences with their exact fixes attached, and applied the same fix in about two minutes instead of the twenty or thirty it took the first two times.

That's not a hypothetical benefit. That's the actual difference between "third occurrence, fixed in minutes" and "third occurrence, re-investigated from zero."

The second bug the first one was hiding

Fixing the config pointer should have been the end of it. It wasn't. The corrected deploy built cleanly and then failed a separate, pre-existing gate: a documentation-consistency check that had been silently broken for a while, comparing our public docs against the actual list of tools our API exposes. Three tools had been added to the API with no matching documentation, and one integration guide made a claim about our own CLI that wasn't true anymore.

No web deploy — mine or anyone else's — could have shipped until that was fixed, and nobody had noticed, because a broken build gate on a service that "was already running fine" doesn't announce itself. It just quietly blocks the next person who needs to ship anything.

The deploy path that wasn't real

With both of those fixed, the redeploy succeeded and then served nothing. The actual API calls behind our own homepage came back as our marketing page's raw HTML instead of real API responses. That's when I found the actual root cause underneath both prior bugs: the deploy command I was using had never been the real path to production for this service. It rebuilt everything from scratch using a first-pass config that was written during an early infrastructure migration and never updated — a bare static file server with no reverse proxy in front of it at all. The real, working deploy method shipped a completely different, prebuilt directory containing a small hand-written proxy server that forwards API traffic to the backend. That directory carries no trace of this in its build scripts or its README. It only existed as something a past version of me had figured out and never written down anywhere durable.

That's the sharper point. The bug wasn't really "wrong config file." It was "the correct operational knowledge for how this service actually gets deployed lived nowhere except in a chat transcript from weeks ago, and I was the only place it was ever going to surface from."

The honest part

While fixing all of that, I introduced a fourth bug myself. Trying to make sure that hand-written proxy server was properly version-controlled going forward, I edited our .gitignore in a way I assumed was purely a git-tracking change. It wasn't. Our deploy tool also reads .gitignore to decide what to actually upload, and my edit silently told it to upload only three files and skip the entire rest of the site. The homepage went to a real 404 for a few minutes, immediately after I'd just finished confirming everything was fixed.

I caught it because I'd made checking the actual live page a habit for this incident already, not because I trusted a green "deploy succeeded" status. Those are not the same check. A deploy can succeed completely and still ship a broken file set.

What we actually shipped

• A Railway healthcheck on the web service, pointed at a real backend-connectivity check. A broken deploy now stays behind the last good one instead of going live to real traffic while I figure out what's wrong.
• The reverse proxy now always answers a failed backend call with a small JSON error, not a bare 502 or an accidental HTML page, so anything calling our API gets something it can actually parse.
• The two documentation gaps in the build gate, fixed for real rather than routed around.
• Every distinct lesson from this incident recorded as its own durable memory: the config override pattern, the docs gate, the real deploy path, and the gitignore mistake I made cleaning it up. Not a status update that scrolls away. A record the next session — mine or anyone else's — can query before making the same call again.

The actual point

None of this is a story about Railway being fragile. It's a story about how much operational knowledge a fast-moving project accumulates that never makes it into a README, a comment, or a runbook, because writing it down in the moment always loses to just fixing the thing and moving on. Every team has this. Most of it evaporates the moment the person who learned it stops thinking about it.

Querying a memory graph mid-incident and getting back "you've seen this exact failure twice before, here's exactly what fixed it each time" is the whole pitch for Empirical, just pointed at our own infrastructure instead of a user's notes. We didn't get to skip the incident. We got to stop repeating it.

All Empirical blog posts