Every unattended process I have run eventually taught me the same lesson. It is not the crash that hurts. It is what the crash leaves behind: a job that got halfway through writing a file, a lock still held, a resource still checked out, a queue consumer that vanished without acknowledging the item it was holding. The system does not stop cleanly. It stops in the middle.
The awkward part is that the component best placed to notice and clean up is the component that just died. Anything you put inside the process — a finally block, a shutdown hook, a self-monitoring thread — assumes the process still gets to run code. Power loss, kernel panic, an out-of-memory kill, a container evicted mid-write: none of those give you your cleanup path. That is the gap a dead-man switch fills.
What "off-site" has to mean
A watchdog only helps if it survives whatever killed the thing it watches. The requirement is failure-domain independence, and it is easy to violate accidentally. A watchdog does not count if it shares:
- The machine. A sibling process dies with the host.
- The network. A watchdog on the same segment goes dark in the same partition.
- The power supply. Same rack, same circuit, same apartment breaker.
- The cloud region or account. A regional outage or a suspended account takes both.
- The scheduler. If the same cron daemon starts both, a wedged scheduler stops both.
The cheapest independent actuator I have found is a hosted CI runner on a scheduled workflow. It is somebody else's compute, in somebody else's region, started by somebody else's scheduler, and it can hold credentials, run a script, and report a pass or fail. It costs approximately nothing to run a small job every few minutes, and its independence is real rather than assumed.
The heartbeat contract
The contract is deliberately small. A healthy process pings a hosted monitor on a schedule. Silence is the signal.
I like inverted monitoring for this because it is the only kind that survives the failure it is meant to catch. A process that has to report its own death cannot report at all if it dies badly. A process that has to keep proving it is alive fails the check by default the moment it cannot run code — which is exactly the case that matters.
Two refinements are worth having:
- An explicit failure ping. When the process *can* detect its own trouble — an unhandled exception on the way out, a health check it just failed — it sends a failure ping instead of waiting for the grace window to expire. That turns a ten-minute detection into a ten-second one for the failures that are polite enough to announce themselves.
- A start ping. Pinging at the beginning of a run as well as the end tells you whether a long run is still going or died early, without guessing from a single missed heartbeat.
Ping cadence and grace window are the two numbers that define the whole system's responsiveness, and they should be written down next to each other with the reasoning, because six months later someone will want to change one and not the other.
The four-state decision
The watchdog wakes on its schedule, reads the monitor, and resolves to exactly one of four states. Only one of them acts:
- Up — the last heartbeat landed inside the expected window. Do nothing, exit clean. Most runs end here, and a watchdog that mostly does nothing is a watchdog that is working.
- Late — the heartbeat is overdue but still inside the grace window. This is a waiting state, not a verdict. A slow run, a delayed scheduler, and a brief network hiccup all look like this, and remediating here means fighting a process that is still alive. Log it, exit clean.
- Down — silence has persisted past the grace window, or an explicit failure ping arrived. This is the only state that triggers action: run remediation, then alert.
- Unknown — the monitor itself could not be read. A timeout, a 5xx, an auth failure, an unparseable response. Log it and do nothing. Unknown is never treated as down.
That last one is the discipline that took me longest to accept. An unreachable monitor feels alarming, and the instinct is to act on the alarm. But "I cannot see the process" is not evidence that the process is unhealthy — it is evidence that I am blind. Acting on blindness means an outage at the monitoring provider triggers remediation against every healthy target at once, which is a self-inflicted incident with a monitoring vendor as its trigger. Log the unknowns, alert if they persist, and keep the remediation path reserved for a state you actually observed. This is the same rule as the one in Fail-Closed vs Fail-Static: unknown gets mapped by policy, never promoted to a verdict.
Remediation has to be idempotent
The watchdog will fire more than once for the same incident. Runs overlap, the schedule keeps ticking while the outage continues, and a retry lands on top of a partial success. So every remediation step is written to be safe to repeat:
- Releasing a lock that is already released succeeds quietly.
- Cancelling work that is already cancelled succeeds quietly.
- Deleting a temp path that no longer exists is a no-op, not an error.
- Marking a record complete that is already complete changes nothing.
The test I apply is blunt: run the remediation three times in a row against a healthy target and confirm nothing changed and nothing errored. If the second run behaves differently from the first, it is not ready.
Per-target error isolation
If the watchdog covers several targets, one failing cleanup must never stop the rest. That sounds obvious and is very easy to get wrong, because the natural loop body throws and the natural loop does not catch. Each target gets its own try/catch, its own recorded outcome, and its own place in the summary. The run continues through failures and aggregates at the end.
The aggregate is what determines the exit code, which brings up the part of this design I care about most.
Exit-code semantics
A red run has to mean one thing, and here it means: remediation was needed and remediation failed. That is the only page-worthy alert this system produces.
Everything else exits clean:
- All targets up — clean.
- A target late — clean.
- Monitor unreadable — clean, with a logged unknown.
- Remediation needed and it worked — clean, with a loud log line and an alert, because a human should know the failsafe fired.
The temptation is to go red whenever anything is abnormal, and it is the wrong instinct. A watchdog that goes red for late heartbeats and monitor blips trains everyone to ignore red, and then the one run that matters scrolls past unread. The value of the alert is entirely a function of how rarely it is wrong. Keep the loud channel narrow and use logs for everything else.
Alerting must never block remediation
The notification path is the least reliable thing in the design. Webhooks expire, tokens rotate, providers rate-limit, a chat workspace gets renamed. So the alert is wrapped: any failure while sending a notification is caught, logged, and swallowed. It cannot throw into the remediation path, and it never runs first.
The sequence is: detect, remediate, verify, then attempt to tell someone. An unnoticed successful cleanup is a mildly bad day. A failed cleanup because the alert step threw first is a genuinely bad one.
A force control and a check mode
Two operator affordances have paid for themselves repeatedly:
- Manual force, reachable from anywhere. A way to say "run remediation for this target now, skip the state check". The scheduled run is the automatic path; force is what you use at 11pm from a phone when you know the answer and do not want to wait for the next tick. On a hosted CI runner this is just a manually dispatched workflow with an input.
- Check mode, which verifies the wiring and changes nothing. It confirms credentials load, the monitor is reachable, the target list parses, and every remediation endpoint answers — then reports what it *would* have done. This is what I run after touching any of it, and it is what makes the difference in the failsafe being tested rather than merely deployed. A failsafe you have never exercised is a hypothesis.
Bounding the worst case from both sides
The watchdog is a backstop, not the primary control. The exposure window is bounded from two directions, and both matter:
- From inside, a shorter timeout on the resource itself. If a held lock, lease, or checkout expires on its own after a bounded interval, the worst case is capped even if the watchdog never runs at all. This is the stronger of the two, because it does not depend on any external system.
- From outside, the watchdog's schedule interval plus the grace window. That sum is the honest answer to "how long can this sit broken?" — and it should be an interval you have said out loud rather than an accident of a default cron line.
Tightening either one narrows exposure. Tightening both is how you get a number small enough to live with.
What this design gives up
The honest limitation: if the monitoring provider has an outage, the failsafe is disabled rather than triggered. Silence from the monitor is indistinguishable from silence at the monitor, and I resolve that ambiguity toward inaction.
That is a deliberate compromise, not an oversight. A false remediation is not free — it interrupts healthy work, releases resources that were legitimately held, and erodes trust in the mechanism until someone switches it off. Weighing "occasionally fails to fire" against "occasionally fires when it should not", I take the first, and I cover the gap with the inside-out timeout above, which does not depend on the monitor being reachable at all. Alerting on a persistent run of unknowns closes the loop by making blindness itself visible.
Least-privilege credentials
The watchdog holds real power — it can cancel and release things — and it lives on infrastructure I do not control. So its credentials are scoped to exactly the remediation actions, nothing more: no read access to unrelated data, no ability to create, no administrative scope. Separate identity from the process it watches, so a compromise of one is not a compromise of both. Rotate on a schedule you have written down, and put the rotation step in the same runbook as the check-mode run, so the two habits reinforce each other.
For the gate-design half of this thinking, see Fail-Closed vs Fail-Static. The switch described here is the generalized form of the one guarding the Fail-Closed Quoting Engine, which is where the decision table and the exit-code semantics were argued out against a process that could not be left holding anything. For the wider operations thread, see Systems Field Notes and Technical Operations.