Most unattended automation has at least one gate in it: a check that decides whether the next step is allowed to run. A backup runner asks whether the destination volume is mounted. A deploy job asks whether the smoke test passed. A home-lab irrigation controller asks whether the soil sensor says dry. A nightly batch job asks whether yesterday's load actually finished.
The interesting question is not what a gate decides when it can answer. It is what the gate does when it cannot.
Two failure stances
Fail-static keeps the last known answer. If the sensor stops reporting, the controller keeps doing what it was doing. Fail-closed treats an unanswerable question as "no". The gate blocks, nothing proceeds, and work is deferred rather than performed on stale grounds.
I default to fail-closed for anything that grants permission, and I think "keep doing what you were doing" is the wrong default there because of what it quietly assumes: that the world has not changed since the last successful reading. That assumption fails hardest during exactly the incidents worth caring about. The reason the reading stopped arriving is often the same reason the answer has changed. A monitoring agent goes silent because the host it watches is sick. A config service becomes unreachable because a network segment is degraded. A status file stops updating because the upstream job crashed. Fail-static reads the loss of signal as "everything is fine, continue" — the one interpretation the evidence does not support.
Fail-static still has a place. It suits gates where continuing is inherently harmless and stopping is disruptive: a dashboard that keeps showing the last good reading, a rate limiter that holds its current ceiling while its config service is unreachable. The mistake is applying that stance to a gate whose job is to say whether an action is safe.
Split gates by the question they answer
Once I stopped writing "the gate" and started writing gates, plural, each named after the question it answers, the stances sorted themselves out.
Take a nightly report generator. It has a timing gate — "are we inside the window when this job may run?" — derived from a clock and a schedule. If the calendar service that supplies holiday exceptions is unreachable, falling back to the last known schedule is defensible. Schedules change slowly, and the worst case is a report that runs on a day nobody needed it.
The same job has a safety gate: "is the destination the one I think it is, and does it have room?" That one must fail closed. The cost of "I could not tell" is nowhere near the cost of proceeding.
So the code reads:
withinSchedule()— a timing question. May fail static.destinationVerified()— a safety question. Fails closed.noConflictingRunActive()— a safety question. Fails closed.upstreamDataComplete()— a correctness question. Fails closed.
Naming the question makes the stance obvious to whoever reads the code next, and it stops a single generic isOkToRun() from smuggling a timing fallback into a safety decision.
Missing, stale, ambiguous, and erroring all resolve to no
There are four distinct ways a gate fails to get an answer, and collapsing all four into one branch is the highest-value habit I have picked up here:
- Missing — the value was never written. The status file the upstream job should have produced does not exist.
- Stale — a value exists, but it is old.
- Ambiguous — the value is present and fresh but does not parse into a case we handle. An enum gained a member since this code deployed.
- Erroring — the lookup threw. Timeout, permission denied, malformed JSON, DNS failure.
The shape I use is boring on purpose:
if (!reading) return block("missing")if (age > freshnessBudget) return block("stale")if (!KNOWN_STATES.has(reading.state)) return block("ambiguous")
and the whole function wrapped so that any thrown error returns block("error") rather than escaping to a caller that might interpret an exception as "skip the check". The recorded reason matters as much as the block. A gate that blocks without saying why produces an operator who concludes the gate is broken and disables it.
Unknown is never promoted to a verdict
The gate's action is binary, but its data model should not be. Internally I keep three states — allow, block, unknown — and map unknown to a block at exactly one place, the caller. Unknown is mapped by policy; it is never *recorded* as a block, and it is certainly never recorded as a pass.
Logs get read later. If unknown is stored as "block", the history shows a healthy gate rejecting things, and nobody can tell an intact gate from a blind one. The rate of unknown is one of the more useful signals I have: it climbs before a gate goes wrong, and it is the metric that tells you a source is dying while the gate is still nominally working.
The inverse anti-pattern is the same mistake wearing a different shirt. if (sensorSaysUnsafe) block() reads the absence of an unsafe signal as safe, so a dead sensor is silently a green light. Invert it: if (!sensorSaysSafe) block(). The gate should demand affirmative evidence, not the absence of a complaint.
Latching, so a flapping input cannot cause churn
A gate reading a flapping input will thrash — block, allow, block, allow — and if each transition kicks off work, the flapping gets amplified into churn: half-finished jobs, partial writes, cleanup running against cleanup.
Latching fixes that. Once the gate blocks for a reason serious enough that a person should see it, it stays blocked no matter what the input does next. Three details make the difference:
- Persist the latch before acting on it. Write the latch, confirm the write landed, then perform the stop. If the process dies mid-stop, the restart reads the latch and stays stopped. Do it the other way round and a crash produces a machine that resumes straight into the condition that just tripped it.
- The latch outlives the process. Store it where a restart will find it — a file on the persistent volume, a row in the local database — not a variable in memory or a flag on a container about to be replaced.
- Only a person clears it. No auto-clear timeout. A latch that clears itself after thirty minutes is just a slower flapper. Clearing should be a deliberate act with a name attached: a command run, a file removed, a click recorded.
Latches want a scope. Latch the smallest unit that makes sense — one failing backup destination should latch that destination, not the whole runner.
The asymmetry test
Before writing the stance, I write both directions down in plain language:
- A wrongly blocked run costs one skipped nightly report, noticed the next morning, rerun by hand in ten minutes.
- A wrongly allowed run costs a partial dataset written over a good one, noticed a week later when the numbers look strange, recovered from a backup that is now a week stale.
Written like that, the stance picks itself, and — this is the part that actually matters — over-blocking becomes an accepted cost rather than a defect waiting to be filed. Teams that skip this step end up "fixing" the gate the third time it blocks unnecessarily, and the fix is nearly always to loosen it. I keep the asymmetry in a comment directly above the gate, so the argument for strictness sits where the person tempted to relax it will read it.
The asymmetry can genuinely point the other way. A gate guarding a cache refresh on a read-only pipeline might cost more by blocking for six hours than by refreshing from slightly stale input. That is fine. The discipline is naming the cheap direction and the expensive one out loud, then biasing toward the cheap one — not assuming strict is always right.
Freshness belongs to the source, not the reading
The age of a single reading is a weak freshness test. Sources have their own cadence, so a sensor reporting every thirty seconds and one reporting every fifteen minutes cannot share a threshold. Worse, a source can emit perfectly fresh timestamps while being wrong: a stuck agent replaying a cached value with a new timestamp looks healthier than a source that honestly went quiet.
So I key freshness to the health of the source and ask, in sequence:
- What cadence does this source declare — declared, not inferred from whatever it happened to do last week?
- Have the last several readings arrived at roughly that cadence?
- Is the value moving in a way a live source would move? A temperature sensor repeating an identical value to four decimals for six hours is not reporting.
- Does the source's own health endpoint agree that it is healthy?
The cheap version of this is one timestamp and one counter per source: last seen, plus consecutive on-time readings. Require a minimum streak before the value counts at all. A gate that asks "is this source healthy?" and only then "what does it say?" survives a class of failure that a plain age check walks straight into.
A component that is built is not a component that is enabled
Every gate carries an explicit arming flag, defaulting to off, and the process logs its armed state on startup — one line listing every gate and whether it is live. The reasoning:
- Deploying a gate and enabling it are separate risks and deserve separate moments.
- A new gate should spend its first stretch in observe-only mode: evaluate, record the verdict, take no action. That is how you learn its false-block rate without paying for it.
- A disarmed gate must be loudly disarmed. The failure I most want to prevent is a gate everyone believes is protecting them that has quietly been off since a config refactor.
Missing configuration means the gate is off *and says so*, not that it defaults to armed. Silent defaults in either direction are how a system ends up with a safety story nobody has verified. That startup line is the first thing I read when something got through that should not have.
What I keep from this
Name the question each gate answers. Make missing, stale, ambiguous, and erroring land in the same branch. Keep unknown in the data even though the action is binary. Latch, persist the latch first, and make a person clear it. Write the asymmetry down before choosing the stance. Judge freshness by the health of the source. And print the armed state on every start, so "we have a gate for that" is a claim anyone can check.
These are notes on the pattern; the system they came out of is the Fail-Closed Quoting Engine, where every gate on this page had to earn its place against something that would keep acting if nobody stopped it. The companion note, The Off-Site Dead-Man Switch, covers what to do when the process holding the gate is itself the thing that died. For the wider operations thread, see Systems Field Notes and Technical Operations. Runbook Composer is useful for writing down the human half — the step where somebody clears the latch.