Status: supervised production. This is the most technically demanding system I have built, and it is the one that runs with real consequence attached. It is a set of engineering notes rather than a product page, because the reliability work is the transferable part: how to run unattended automation that holds live obligations on somebody else's system and still fails safe when the feed, the process, the machine or the operator is not there.
What it is
A supervised, fail-closed automated quoting system for thin two-sided markets on two CFTC-regulated US event-contract venues, run with personal capital. These are legal, regulated US markets under federal commodities oversight; there is nothing offshore or grey about the venues, and the system holds no client money.
Three properties define what the rest of this page is describing, and they are worth stating exactly before anything else.
It posts resting orders only. It is a maker and never a taker: every order it can emit is a post-only limit that goes into the book and waits to be hit, and nothing in it crosses the spread. Taker-shaped orders are not avoided by convention — they are refused at the wire, at three independent layers, before a request can reach the venue. That enforcement is described in full below; the point here is that it is enforced rather than intended.
It builds its quotes from streaming data. Fourteen WebSocket connections carry full order-book snapshots into a book cache, and every quote is derived from that cache rather than from a request issued at decision time; a dropped shard degrades its own markets to a single throttled read and touches nothing else. That is what keeps resting quotes honest against a live book, and it is why the streaming design further down is not an implementation detail — the freshness of the stream is the freshness of the quote, which makes shard liveness, rather than the age of any individual book, the thing the system has to reason about.
It fails closed. Doubt resolves to cancel. Missing data, stale data, ambiguous data, a hung process, a dead machine or an absent operator all end in the same place: resting orders cancelled, placement stopped, inventory held. Why that stance is not the default, and what it costs when it is wrong, is set out below.
It runs on a home machine as a small fleet of Python processes under a supervisor, with an independent dead-man switch on separate infrastructure. The work was operator-directed engineering with an AI pair: I set the constraints, wrote the specs, ran the incidents and made every call that had money behind it; the pair helped implement and red-team. Every safety decision on this page has a date and an incident behind it, because almost none of them were designed in advance. They were written the day something went wrong.
Where it came from
The interest is older than the system. I came up through data analytics, and long before any of this I spent years on sports data — mostly on the question underneath it: what can actually be predicted, and what only looks predictable once you already know the answer. That curiosity is why this project started. It was never a view about a particular market.
So it began as a research question rather than a build. The first version was a read-only fills dashboard, and the research phase behind it closed with a written negative result: measured properly, the naive approach had no edge. That was the correct answer to the question I had asked, and it is the reason the system that exists now looks nothing like the one I set out to build.
What survived was everything around the prediction. When you cannot be confident about the forecast, the part that decides when *not* to act carries the weight — the wire-level guards, the fail-closed gates, the latching governor, the supervisor, the off-site canceller. I did not set out to do reliability engineering; that is simply what the durable work turned out to be. Someone who starts by asking what is predictable and ends up writing a dead-man switch has at least learned where the defensible part of the problem was, and it is the half of the system I can publish in full.
What is public here, and what is not
The engineering is public. The strategy is not.
That split is deliberate and I would rather state it than be vague about it. This is a live personal system, and an edge that is described in public stops being an edge — a portfolio page is not worth handing competitors a map. So this page carries the architecture, the failure modes, the tests and the post-mortems in full, and it deliberately omits the two venues by name, the market families, sizing, bankroll, P&L, pricing inputs, fair-value sources, incentive mechanics, thresholds and market selection. None of that is missing because it is embarrassing. It is missing because it is the commercially valuable half, and the reliability half is the half that is worth reading anyway.
Everything below can be evaluated on its own terms. If the interesting question is "can this person build something that stays safe unattended," the answer is entirely in the parts I am willing to publish.
Why fail-closed
The whole system rests on one stance: quote only on fresh, positive evidence that quoting is safe, and when in doubt, cancel. That sounds obvious and it is not the default. Most automation fails static — when a data feed goes stale it keeps doing whatever it was doing, because the last known state is still "known." That is fine for a clock that answers "when does the event start." It is the wrong shape for a gate that answers "is it safe to quote right now," because stale data there means the honest answer is no.
So the gates are split by the question they answer. Timing gates may fail static. Play-state gates fail closed: missing, stale, ambiguous or erroring data all resolve to not quotable — one branch, four causes, so no failure mode gets its own accidental exemption. Every failure mode in the hardening work resolves the same direction: cancel resting orders, hold inventory, stop placing. Holding is always safe under the operator's constraints; selling never happens automatically. And every arming is an explicit operator action behind its own flag, so a component that is built is not a component that is live.
The generalised version of this argument, with the venue and the money taken out of it, is written up separately as Fail-closed vs fail-static. That note is the portable pattern; this page is the system it came from.
The sharded WebSocket book cache
The first version read order books by polling the venue's REST endpoint per market. That worked at a handful of markets and stopped working the moment the watched set grew: the venue's edge rate-limiter started returning blocks, and because the limit is applied per source address and shared across every process on the machine, one impolite loop could get all of them throttled at once. Polling harder was not an option, and polling less made the books too stale to quote against.
The replacement is a cache fed from 14 independent WebSocket connections. Several design decisions in it are the transferable part:
- Full snapshots, not deltas. Every market-data message carries a complete book, with no sequence numbers to track and no delta stream to replay. That makes the cache a last-write-wins replace and lets it be a true drop-in for the polling call it replaced — the calling code never learned that its data source had changed.
- Stable hash partitioning. Each connection owns a slice of the watched set chosen by a hash of the market identifier modulo the shard count. A market always lands on the same shard, so a watched set that changes slightly between cycles does not reshuffle subscriptions across connections. Round-robin or index-based assignment would have churned every subscription every time one market settled.
- A scarce, shared connection budget. A single connection is safe to a few hundred subscriptions and fails hard past a measured ceiling, and the account itself has a measured ceiling on concurrent connections shared by every process. That made connections a budgeted resource: the shard count, the private order stream and the audit stream all draw on one envelope, and any proposal to add a stream has to say which process gives one up. Treating a shared external quota as a first-class budget with named claimants is not a trading idea; it is the same discipline as file descriptors or database connections, and skipping it is how a "capacity-only" change silently starves an unrelated process.
- Freshness keyed on shard liveness, not per-book age. This is the design point I would most want to explain in an interview. A thin market only emits a snapshot when its book changes, which can be minutes apart, so a book's age tells you almost nothing about whether it is current. A five-minute-old snapshot on a quiet market is correct; a five-second-old snapshot on a shard that has since died is not. So freshness asks about the *source*, not the *value*: a book is fresh if and only if its own shard is connected and has produced traffic recently. Age-based staleness would have thrown away good data on quiet markets and trusted bad data on dead ones — both directions wrong at once.
- Graceful per-shard degradation. A read checks the liveness of that market's shard only. One shard dropping makes its own markets unavailable and the caller falls back to a single throttled REST read for them; every other shard keeps serving. Degradation costs coverage, never correctness.
- Fail-drop publishing, never fail-blind. A value is published only if its shard is connected *and* its cached snapshot postdates that shard's current connection. Data captured before a reconnect is withheld until a fresh snapshot lands, because the venue re-sends full snapshots on subscribe — which means the backfill after a reconnect is free, and the only thing that has to be engineered is the discipline not to serve the pre-reconnect copy in the meantime. A withheld value degrades to the safe path. A stale value that looks live does not.
- Self-referential state gets its own defences. One derived value in the pipeline is computed partly from the system's own resting state, which creates a feedback hazard: if the self-exclusion is wrong, the system can chase itself. The defences are layered and all point the same way — every doubtful case resolves toward the conservative side or drops the value entirely; staleness in the self-state invalidates the whole map wholesale rather than guessing per item; and a guard freezes any series that moves monotonically in the direction a wrong exclusion would produce, since genuine two-way flow essentially never sustains that signature. Wholesale invalidation over per-item guessing is the general lesson: when the correction input is untrustworthy, the corrected outputs are all untrustworthy together.
A REST verification sweep survives as an audit lane rather than the primary path: it re-reads a small random sample on a slow cadence and records agreement, so the streaming path has continuous evidence that it still matches the source of truth instead of a one-time validation from the day it shipped.
Wire-level order-policy enforcement, at three layers
The policy stated at the top of this page is enforced as code: every order this codebase can emit must be a resting, post-only limit with a resting time-in-force. Anything else — market orders, immediate-or-cancel, fill-or-kill, close-position, singular modify — is refused before it reaches the venue. Missing fields fail; they do not default-pass, because a policy that treats absence as consent is not a policy.
The reason this is enforced at three layers rather than one is worth spelling out, because "we validate the order before sending it" is what everybody says and it is not sufficient:
- The instance layer patches the specific client object the calling code holds. It is the cheapest guard and the easiest to bypass, because any code path that constructs its own client gets none of it.
- The class layer patches the client type program-wide, so a second client constructed anywhere in the process inherits the policy — including in a module that was written months later by someone who never read the guard.
- The raw-POST layer sits at the transport, below every convenience method the vendor SDK offers. It rejects the forbidden endpoints outright and asserts the resting shape on the body of every order-creating request. This is the layer that catches the case the other two cannot: a new SDK version adding a method nobody has audited, or a call that bypasses the typed helpers entirely.
Each layer alone is defeatable by an ordinary mistake. Together they mean no code path in the process can construct a taker-shaped order by accident, and the check the transport performs is on the bytes actually going out rather than on an intermediate object that something might still mutate. Every construction site that can write was audited and wrapped; the read-only paths were audited and documented as deliberately unwrapped, so "why isn't this one guarded" has a written answer instead of being an open question.
There is a rule attached to the guards, and it is a documentation rule rather than a code rule: any function that constructs, modifies or cancels an order must be justified against a local mirror of the venue's own documentation, by line reference, before it ships. That rule exists because the incident that created the guards was caused by a primitive whose dangerous semantics were documented in a mirror that was already sitting in the repository, and nobody read it. Making the citation a merge requirement is what turned "we have the docs" into "the docs are load-bearing."
Fail-closed play-state gates
Some of these markets track events that go periodically live, and quoting into a book while the underlying event is in progress is exactly the wrong time to have resting orders out. The gate that governs this answers "is it safe to quote right now," so it fails closed: quoting is allowed only when a fresh read from an upstream feed positively says play is not live and the event is not over. In-progress, suspended, delayed, unknown, stale and errored all resolve to not quotable, and suspensions are deliberately excluded because play may resume and those books are jumpy.
This gate was rebuilt twice, and the sequence is the most instructive thing on this page.
The first version keyed on a single status field. That field lingered on a "play complete" value after the final segment finished and the outcome was decided, so "the day's play is done" and "the event is over" were indistinguishable — and the gate re-opened quoting on a market whose result was already known.
The second version keyed on two other fields and asserted in a code comment that they were authoritative. A live measurement weeks later showed that the upstream feed set those fields per segment, not per event. So the fixed gate fired between segments — precisely the window it was supposed to open — and the affected markets could never be quoted after the first segment ended. The gate was not wrong about safety; it was wrong about the semantics of somebody else's data model, and the assertion in the comment was doing the work that a measurement should have done.
The third version keys on a primary completion signal with an independent time-based backstop, and falls back to the earlier reading when the backstop's input is absent — a fallback that over-blocks and can never over-quote. Across all three passes the rule "no data means not quotable" was never relaxed, and that is why two design errors cost quoting time instead of costing money. Over-blocking was the acceptable failure; over-quoting was not, and knowing which direction is cheap before you start is the whole game.
The gate also polls adaptively — a slow baseline cadence with a faster cadence inside the windows where the state can change — and never trusts a read past a staleness bound, so every fetch failure simply ages the last reading toward "not quotable" rather than needing its own error branch.
The risk governor: latch before acting
A separate process enforces hard limits whether or not anyone is watching. On a breach it cancels all open orders and disarms trading, and it never sells inventory.
The important word is latched, and the important ordering is persist the latch before acting on it. The latch is written and the write confirmed, and only then does the halt run. If the process dies mid-halt, the restart reads the latch and stays halted; done the other way round, a crash produces a process that comes back up and walks straight into the condition that just tripped it. A latched halt whose cancel could not be confirmed retries the cancel and stays halted; status remains halted for as long as the latch is set; and the latch clears only when the operator re-arms, which makes re-arming the deliberate, named "I have investigated this" action rather than a timeout that quietly forgives.
The halt path also cannot be aborted by its own instrumentation: the logging call inside it is IO-safe, so a disk problem cannot prevent a stop. The rule generalises — a safety action must not be able to fail because something non-safety-critical failed first.
The supervisor: venue truth and command-line liveness
Detached processes get killed by system updates and session teardowns, and a dead maker stops managing its resting orders — which means quotes sit into an event start and get adversely filled. That happened twice before the supervisor existed.
Two details make it work:
- Liveness by command-line match, never a tracked PID. A PID recorded at launch goes stale the moment anything restarts a process out of band, and a supervisor that trusts a stale PID either double-launches a maker that is already running (two processes quoting the same book) or believes a dead process is alive. Matching on the command line asks the operating system what is actually running, which is the only source that cannot drift.
- Venue-truth resync before every relaunch. On each restart the supervisor first syncs local state from what the venue actually holds, and only then arms. Local state written before a hard kill is a claim, not a fact; the venue's own view of resting orders and positions is the fact. Without the resync a restart can re-arm into a phantom state — believing it holds orders it does not, or not tracking orders it does — and both directions produce duplicate or orphaned orders.
It runs under a logon-scheduled task so a reboot recovers the whole fleet without a human, and it has a self-test mode that reports what is running and what it *would* launch while launching nothing.
The hardening pass added an escalation ladder above this, because the supervisor only recovers processes that are *absent* — and the failure that actually hurt was processes that hung while still alive, invisible to every recovery mechanism in the system. The ladder alerts first, then, past a much wider budget, cancels that process's resting orders, kills it, and lets the supervisor relaunch it, with a warm-up grace period so a restart cannot immediately re-trigger. Its details are all about not making things worse: the cancel is selective rather than account-wide, because the processes share one account and an account-wide cancel would destroy the other processes' books; escalations are serialised behind a global lock so two recoveries never coincide; a storm cap turns repeated auto-recovery into an alert that says a human is needed; and the whole escalation must complete inside the dead-man switch's grace window, so the off-site canceller never fires in the middle of a recovery it cannot see. That last constraint — a recovery mechanism has to be legible to the *other* recovery mechanism — is the kind of thing you only find by drawing both timelines on the same axis.
The same audit found that the health watcher, the component whose entire job was catching hangs, was itself not supervised. The watcher that catches failures is a component like any other, and asking "what watches this?" one level further out is nearly free and almost always finds something.
Bounded calls, breakers and clock integrity
One incident night produced a specification that is mostly about removing unbounded waits. Processes hung alive several times, leaving resting quotes unmanaged for tens of minutes each time.
The root cause was not where it looked. The WebSocket stacks run on daemon threads and can only degrade caches, never freeze the main loop. The freezes lived in the REST read layer, and specifically in two habits that are easy to write and hard to see:
- A per-socket-read timeout is not a total bound. The HTTP client's timeout applies to each read from the socket, so a response that trickles a byte just inside the timeout window can hang forever without ever tripping it. Layered under a silent retry loop, one call could consume minutes with no log output at all.
- Unbounded pagination on top of that. A "fetch everything" loop with no page cap, calling a degraded endpoint, turns a slow venue into a stall of indeterminate length — and it was the first call of the main cycle, which is exactly what the freeze signature showed.
The fix set is unglamorous and worth copying: a wall-clock deadline on every logical call that stops attempts when the budget is spent regardless of attempt count; a log line on every retry, because the silence was what made a slow venue undiagnosable; a page cap on every pagination loop sized to a multiple of observed need, logging loudly and returning partial results that callers already tolerated; explicit connect and send timeouts on the WebSocket layer so a flow-control stall feeds the existing reconnect-and-backoff path instead of permanently killing a shard's supervisor coroutine; a maximum snapshot age on the order cache that self-marks it dirty rather than letting it be quietly trusted forever; and an explicit rule that a deadline expiry must never trigger extra retries anywhere, so bounding a call cannot amplify load.
The whole set adds zero steady-state load and is strictly load-*reducing* under degradation, which is the property that made it safe to ship into a live system: bounded attempts and bounded pages where the previous behaviour was unbounded.
The same specification adds a per-venue connectivity circuit breaker with the usual three states, and one detail that is specific to this domain: the open state cancels the book first. Open means flat — nothing resting, so nothing to go stale and nothing to churn while the venue is degraded — and the breaker emits a heartbeat line while it is open, so the health watcher and the dead-man switch can tell a deliberate pause from a hang.
It also adds clock integrity for free. Every venue HTTP response already carries a server date header, so the client stacks record the difference against local time into a rolling median, ignoring responses that took long enough for latency to contaminate the reading. Small skew warns; large skew fail-closes placement for that process. The reasoning: order expiry, cancel windows and every other time-based backstop price time using the local clock, so a bad clock silently weakens every one of them at once, and nothing else in the system would notice. Zero additional API calls, an entire class of silent failure covered.
Fail-open telemetry
Instrumentation records what the makers do into SQLite in WAL mode through a bounded queue and a batching daemon thread. Three properties, in priority order: every record call enqueues and returns immediately; a full queue drops silently rather than blocking a caller; and any internal error degrades the writer to a permanent no-op instead of raising. Call sites still wrap their hooks in exception handlers anyway — belt and braces, because a live book is not the place to find out that an assumption about the observability layer was wrong.
This is the deliberate inverse of everything else on this page. The safety machinery fails closed; the observability machinery fails open. Telemetry can be lost. It can never take down a live book. Getting that inversion the wrong way round — instrumentation that can raise into a trading path — is one of the classic ways a monitoring improvement becomes an outage.
The same layer feeds read-only operator dashboards: a standard-library HTTP server rendering ledger, health, incident and snapshot views with inline SVG charts and no framework, bound locally, with a client that structurally cannot write to a venue.
Testing: what coverage does not prove
- pytest batteries. Roughly 800 automated tests across the main tree and the streaming rewrite, from guard behaviour and gate logic to dashboard telemetry.
- A hand-built mutation battery. Twenty-two deliberately broken copies of the maker — a guard deleted, a threshold removed, a predicate forced true, a comparison inverted — plus a clean baseline run. The harness injects each defect into a generated *copy*, runs the whole offline battery against it in a subprocess, and asserts a non-zero exit for every single mutant; then it runs clean and asserts zero. The original tree is never touched.
This is the check I would keep if I could keep only one, because it answers a question coverage cannot ask. Coverage proves a line executed. It says nothing about whether any assertion would have *noticed* if that line were wrong. A suite can execute a safety guard on every run, report full coverage of it, and still pass with the guard deleted — and until you delete it and watch the suite fail, "the tests protect this" is a belief rather than a measurement. A green battery proves nothing until it has been shown to go red on a wrong patch.
- Replay and soak harnesses. Recorded tape and shadow counts replay offline; new feeds are built with zero live calls against fixture batteries before any soak; soaks run in dry or shadow mode with dated candidate directories and release checkpoints. A fixture battery for one component parses the live module's source and fails on any drift in the constants it mirrors, so the two cannot silently diverge.
- Shadow-mode cutover behind a hash-bound arm authority. The streaming rewrite can only go live if an arm file exists that satisfies all of: a SHA-256 digest matching the exact source set that was soaked — the rewrite's own modules plus every named safety module it depends on; an explicit acknowledgement string; and an expiry that has not passed. The file is absent by default, a dry run can never create it, and editing any covered source invalidates it immediately. Provenance manifests and an isolated interpreter launch keep shadow builds separate from the live tree. The property this buys is precise: what was soaked is what is armed, enforced by a hash rather than by a human remembering that a one-line "harmless" edit landed after the soak began.
- Self-test and dry modes everywhere. The supervisor, watchdog and off-site canceller each have a mode that reports what they would do and does nothing. Anything that can act in anger should be runnable in a mode where it cannot.
The off-site dead-man switch
The venue-side order expiry bounds the worst case; the local watchdog is the sensor; the off-site canceller is the actuator on independent infrastructure. The local half pings a hosted heartbeat monitor while healthy, pings a failure endpoint the instant the governor halts or state goes stale, and goes silent if the box dies. A private push topic lets a "STOP" from my phone cancel everything and disarm. Both halves are outbound-only: no inbound ports, no remote shell exposure on the home machine, which means adding a remote kill switch did not add a remote attack surface.
The actuator is a scheduled CI runner in a different location that polls the monitor's status and acts on the reading:
- up — heartbeat arriving on schedule → no action; every scheduled run is a no-op.
- late — inside the grace window → no action; grace exists for restarts and blips and is not a verdict.
- down — the monitor has declared silence → cancel every resting order on each venue, independently, so one venue erroring never blocks the other; push a phone alert; exit red if any venue's cancel failed, so a red run means "a cancel failed while the box was down — investigate."
- unknown — the status is unreadable, the monitor is unreachable, or nothing is configured → no action, logged. Unknown is never promoted to down.
- manual force — a button in the CI interface cancels everything on both venues immediately, from anywhere I can reach it, even if the home machine is unreachable.
- check mode — verifies secret wiring and constructs a request signature offline, and cancels nothing.
Cancelling is idempotent, so re-cancelling an empty book is a no-op — which is what makes it safe to run on a schedule rather than exactly once. Alerting is wrapped so a failed notification can never block remediation. Credentials are trade-only, scoped to exactly the remediation actions, and cannot withdraw; they are a separate identity from the process being watched, so compromising one is not compromising both. Exposure is bounded by venue-side order expiry on the outside and by the polling interval on the inside.
The design accepts one honest tradeoff and states it rather than hiding it: an outage at the heartbeat provider disables the failsafe rather than firing it, because silence *from* the monitor is indistinguishable from silence *at* the monitor, and a false cancel-all is not free either. The portable version of this whole mechanism, with no venue in it, is The off-site dead-man switch.
Incidents and post-mortems
The incidents are the most useful part of the record, so they are here in abstract.
A halt path that liquidated inventory
An automated severe-halt path called a venue primitive that closes an entire position at market price, in an unlatched loop, on thin books. It realised far more loss than the positions owed, because market-dumping an illiquid book sells at whatever is resting, and dumping repeatedly in a loop sells into the hole it just made.
Three separate defects had to line up: a primitive whose documented semantics were "sell the whole position at market" being used as if it were a risk *reduction*; no latch, so one bad decision became many; and no policy that said auto-selling was forbidden in the first place.
The same day the policy became code. No code may ever auto-sell, close or reduce a held position — positions ride to settlement, where binary downside is already bounded by the collateral paid, and the operator's risk envelope is the account balance, managed by withdrawing what is not at stake rather than by a program that "protects" the account by dumping it. The reasoning is about the book, not about conviction: dumping a held position into a thin market realises a loss created by the exit itself, at a moment when nothing about the underlying probability has changed. The book is too shallow to absorb the size, so the fill price reflects the depth available rather than the odds of the outcome. A program that sells to protect an account converts a temporary absence of buyers into a permanent loss, which is the opposite of protection. Halts cancel and disarm only, once, latched. Selling is permitted only through an explicitly planned exit path with its own enable flag, its own stricter data-freshness gates, and its own floors — and even that path emits resting post-only orders, never a market close. The wire-level guards described above are the enforcement of that policy, and the documentation-citation rule was added at the same time because the primitive's semantics had been sitting in the repo's own mirror of the venue docs the whole time.
Built, deployed, enabled, and reverted 27 minutes later
A change to the order-reconciliation guards was validated in replay and shipped. Its detector was correct and never misfired once. What the validation did not test was the downstream consequence of enabling it: with the change live, drift in the private order cache made resting orders look vanished, they were adjudicated as not-fills, freed for re-quote while still resting at the venue, and duplicate orders compounded across consecutive cycles. It was reverted 27 minutes after being enabled and the duplicate count collapsed one cycle later. The revert was a single command because the change was gated on a flag file read live each cycle — no restart, no deploy, no window in which the system was neither the old thing nor the new thing.
The written lesson was a component was validated, not the system behaviour it unlocked. The detector's own correctness was never the question; what mattered was what re-enabling adjudication would do to an order source that carried drift, and nothing in the test plan looked at that seam.
The superseding specification was adversarially red-teamed before anything was written, and it rejected both obvious fixes with evidence:
- Raising the capacity ceiling was disqualified, not deferred. The guards turned out to self-track the cap already, so the premise that they needed decoupling was simply wrong. Worse, a count ceiling was being read elsewhere as an input to a *safety* check — so a change advertised as "capacity only, no behavioural change" would in fact have enlarged the population of positions taking a less-checked path. And raising the ceiling would have lifted the governor's halt line above anything its sensor had ever been observed to return, blinding the very component that exists to catch a runaway. "Behaviour-neutral" is a claim that has to be proven against every reader of the value, not asserted from the intent of the change.
- Fixing the obviously-broken cache-eviction predicate first was disqualified too, which is the counter-intuitive one. The bug was real, correctly located, and a one-line fix. It was also load-bearing: its constant divergence was what forced a full truth refresh from the venue on a regular cadence, and that involuntary heartbeat was the main thing bounding a *second*, independent drift mechanism — a snapshot race that could resurrect already-cancelled orders permanently. Fix the visible bug alone and the cache stays "trusted" far longer, the invisible drift persists, and a resurrected dead row can make a recorded fill look like it never happened. The instrumentation to tell the two mechanisms apart had to come first, because the counters in the log emitted totals but not identities — which is precisely why the ambiguity had survived for days.
The actual defect, once named, was one sentence: the system freed an order slot on single-source evidence. The fix required an order to be observed absent from both independent sources before its slot could be released, using data the cycle already fetched, so it added no API calls. Its failure mode is holding a dead key too long — a bounded capacity cost — rather than duplicating a live order, which is an exposure cost. That asymmetry is what made it safe to ship even with the underlying drift unfixed.
The validation plan attached to it is the part I reuse everywhere now. Before the fix could be trusted, the drift-injection replay had to be run against the *old* code first and shown to reproduce the bug — a test that has not been proven to fail on the broken version is not evidence about the fixed one. A cancel-all shape was added specifically to guard against over-correcting into "never frees anything." And the rollout named its kill signals and their thresholds in writing, in advance, with the revert being the deletion of a flag file.
Play-state gates rebuilt twice
Covered above. The short version: a gate keyed on a status field that lingered, then a gate keyed on fields *asserted* to be authoritative which live measurement showed were not, then a gate keyed on a primary completion signal with a time backstop. The fail-closed default survived all three passes, which is why two wrong models of somebody else's data cost quoting time rather than money.
The incident night
Processes hung alive several times, leaving resting quotes unmanaged for tens of minutes. The root cause analysis and the hardening specification it produced are described above; the meta-lesson is that the first suspect was wrong. The WebSocket layer looked guilty because it is the noisy, asynchronous, network-facing part — and it was structurally incapable of causing the freeze. The culprit was the boring synchronous read path, where an unbounded retry loop wrapped an unbounded pagination loop and neither logged anything. Silence in a subsystem is not evidence that it is healthy; it is an absence of evidence, and the fix list started with making the silent parts noisy.
What transferred
Failure-domain thinking. The safety action must not share a machine, a network or a power supply with the thing it protects. Everything else in the reliability stack follows from taking that seriously.
Honest negative results are kept as records. The negative verdict that closed the research phase is still in the repository, and so is a later gap analysis that returned "do not arm" with numbered blockers. A decision log that only contains the decisions that worked is not a decision log.
Direction of failure before the mechanism. Write down what a wrong block costs and what a wrong allow costs, in plain language, before choosing the stance — and keep that comparison next to the gate, where the person tempted to loosen it will read it.
Safety before scale. Capacity changes are labelled behaviour-neutral only after every reader of the value has been checked, and every raise waits for an explicit operator go.
Precedent before exposure. The whole system had an earlier one-day design scaffold on a sandbox, never armed, which fixed the shape of the rate limiter, the exposure guard and the latching kill switch before anything touched a live venue.
The two patterns that generalise cleanly beyond this domain have been written up on their own, deliberately with the venue and the money removed, because they apply to backup runners and deploy gates and scheduled jobs just as well: Fail-closed vs fail-static and The off-site dead-man switch. They are the portable patterns; this page is the system that produced them, with the dates and the incidents attached.
Boundaries
This is a personal system operated with personal capital and no client funds, on two CFTC-regulated US event-contract venues. It offers no signals, no strategy, no performance claims and no code; the venues, market families, pricing inputs, sizing and results are deliberately absent, and no repository link or live dashboard capture appears here. It was built as operator-directed engineering with an AI pair, and it is described here as an engineering record.
Related
The portable patterns extracted from this system are Fail-closed vs fail-static and The off-site dead-man switch. The research thread behind why these markets were interesting in the first place is the Market Intelligence Field Notes. The wider operations posture is Technical Operations and Systems Field Notes. For a completed operating archive built on the same "operate under constraint, exit deliberately" discipline, see CipherG. All projects are listed under Ventures.
Non-advice note: This material is background material only. It is not investment, trading, financial, tax, legal, or compliance advice, and it is not a recommendation to trade, invest, speculate, wager, allocate capital, use leverage, or copy a strategy. This system was operated with personal capital, no client funds; it offers no signals, strategy, or performance claims; and nothing here is a recommendation to trade or wager.