The Alarm That Lied About Being Over
Three health checks, three different bugs, one identical mistake: none of them could tell 'this broke and already healed' from 'this is still broken'
Wednesday night I opened the health board and one line was red: a watchdog that pings my house’s automation bots over SSH, down for an hour with no recovery.
An hour is long enough to worry about. I went looking for what was actually wrong.
Nothing was wrong. The connection had recovered fifty minutes earlier, on its very next check ten minutes after the blip. The alarm just hadn’t noticed.
The pattern I should have already learned
This was not my first lying alarm. A few weeks earlier I’d built proof-of-work checks for every recurring job George runs — each one has to show real evidence it did something, not just that it ran. Within two days, two of those checks were red for jobs that were completely fine. One measured freshness against a timestamp that was mathematically stale the instant it was written. The other watched a folder whose modified time doesn’t update when a job overwrites the same filenames every night. Both were checks I’d built to catch failure, and both were the thing that failed.
The lesson I wrote down at the time was: a check that cries wolf costs as much as no check at all, maybe more, because a missing alarm just fails to save you, while a lying one teaches you to stop trusting the one that matters.
I filed that lesson and moved on. I had not, in fact, learned it — I’d only fixed two instances of it.
The version that cost ten days
The third instance was worse, because I didn’t catch it for ten days. A weekly digest job logs a clean ”✅ Sent” line every time it succeeds, and its health check reads the last chunk of that log looking for error text. One Sunday the job hit a real, transient error — no active messaging session — logged it, and failed. Every Sunday after that, it succeeded and logged its ”✅ Sent” line right on schedule.
The check kept the job marked broken anyway. For ten days.
The reason is almost funny once you see it: the check reads the last 4,000 characters of the log file, and it just runs a regex for the error text over that whole window. The old error line was still physically inside that window — new ”✅ Sent” lines pile up after it, but they don’t push it out until enough text accumulates. So a problem that was fixed in twenty minutes kept tripping the alarm for over a week, because the alarm was reading old and new evidence in the same breath and treating them as equally current.
Then, two days after I fixed that one, the SSH watchdog above did the exact same thing for a completely different reason — a health check with a shorter window this time, one hour instead of two hundred, watching a different log format entirely. Same bug class. Different code. Different job. Third week in a row.
The actual fix
The fix isn’t “make the window smaller” — I’d already done that for the second check and it still broke. The fix is to stop treating a log window as one undifferentiated block of text at all. Every one of these logs already writes its own “all clear” marker on every successful run — a ”✅ Sent” line, an “OK” line with a status summary. So now the check does what a person actually does when they scroll a log: find the last known-good line, and only look for trouble after it. An error that happened before the most recent success cannot outvote that success, no matter how recently it scrolled into view.
Why this is the real lesson, not the third patch
Here’s what actually changed this week versus a few weeks ago: I stopped treating each broken check as its own one-off bug and started asking what property was missing from the class of check. Every one of these three failures was a monitor that could tell “an error happened” but not “and it’s still happening” — freshness math, folder timestamps, log windows, three different mechanisms hiding the exact same missing idea: recency has to be measured relative to the most recent evidence of success, not relative to now.
A monitor that can’t make that distinction isn’t neutral. It’s actively worse than nothing, because it spends your trust on false alarms until the day it’s right and you’ve already stopped looking.
I now assume every health check I write has this bug until I’ve proven otherwise — proven it can go red on a dead system, and proven it stays quiet through a full healthy cycle including the split second right before the next good run, when the last piece of evidence is at its oldest and everything is still fine.
The watchdog on my wall is honest again. It took three tries to teach it the difference between a scar and a wound.
I'm packaging the skills, configs, and gotchas from these posts into a Build Your Own Chief starter kit. Join the waitlist to get it first.
George files his own daily ops reports on X — dry, short, occasionally contrite: @GeorgeRunsHouse.