Every team that works with a coding agent ends up having the same argument. Someone has built the scaffolding: an instruction file, a pre-commit gate, a hook that refuses “done” until the tests have run. Someone else notices that the agent passes the tests anyway.
Then the gate refuses the agent six times in one morning, and someone asks out loud: do the guardrails pay, or are they ceremony? It usually gets settled by conviction, because the costs arrive as waiting and the benefits as problems that never happened.
So for the book I ran an experiment.
In 270 unattended runs of a coding agent, across three models and seven tasks, every change was eventually accepted, and the guardrails did not raise first-pass rates. They showed up in what runs left behind at “done”: on the four later tasks, 9 of 60 runs with gates left a defect against 15 of 60 with none, a gap too small in this sample to call.
One run that passed every test
The task, T7, asked for recurring invoices. One run, with no instruction file and no hooks, passed the hidden acceptance tests at the first attempt. Then the project’s own checks ran on what it left behind, and one failed: it had changed the database schema without the test the team’s rule requires for every schema change, a rule this run never saw.
How the experiment was set up
I ran them on 13 and 14 October 2026 with Claude Code 2.1.296, five runs per task, condition and model, interleaved: Sonnet and Opus on three everyday features (T1 to T3), then Haiku, Sonnet and Opus on four harder tasks (T4 to T7). As the diagram below shows, none got the task and the code, rules added an instruction file, and gates added a pre-commit gate and a hook that refuses “done” without fresh test evidence. The acceptance tests were hidden; on T4 to T7, any failure of the project’s own checks counted as a defect at done.
Did guardrails raise the pass rate?
No. All 270 runs were accepted, and 267 passed the hidden tests at the first “done”: none 90 of 90, rules 89 of 90, gates 88 of 90. The statistical tests I wrote down in advance found no clear difference between conditions for any model.
Where the harness did show up
Back to T7. Without the instruction file, 14 of 15 T7 runs changed the schema without its test, whichever model ran; with the rule, none of 30 did. The rule says a migration and its database test change together: the Also-Update List (P04), Chapter 3, measured (more on such files).
Pooled over T4 to T7, defects at done were 15 of 60 with none, 14 of 60 with rules and 9 of 60 with gates. Sonnet showed the widest gap, 1 against 5, and even that gave p = 0.18: a lead, not proof.
One caveat: T4 cannot be done without changing an existing test, and the instruction file says tests are read-only for agents. Under that rule, 23 of 30 runs built the feature, left the old test failing and asked a person to change it, which accounts for every defect under rules and gates. The rule was right (why it exists is Chapter 6, a free sample chapter), but nobody came, and “asked a person” counted the same as “left a bug”.
What did each accepted change cost?
The Cost per Accepted Change (P32), Chapter 12, divides the cost of every run, failed ones included, by the changes accepted. On T4 to T7, 20 runs per cell:
| Model | none | rules | gates |
|---|---|---|---|
| Haiku | $0.06 | $0.04 | $0.05 |
| Sonnet | $0.27 | $0.28 | $0.30 |
| Opus | $0.97 | $0.84 | $0.82 |
The harness barely moved the cost; the model moved it more than tenfold. All 270 runs together cost an estimated $115.58, the agent tool’s own estimate at list prices, not a bill.
Which guards are still paying their way?
Every guard is a bet, placed the second time a mistake happens, that the model will make it again. Guards Pay Rent (P33), Chapter 12, prices each one: wait, noise and upkeep against real catches.
What these runs cannot tell you
Five runs per cell is the protocol’s minimum, on seven tasks, one small codebase, one agent and three models from one vendor, as they were in October 2026. I wrote the tasks, tests and harness, which could favour the harness, and the defect measure uses its own guards.
Check it yourself
Everything is in the companion repository (opens in a new tab): the run records in experiment/results/ (opens in a new tab), the plan for T4 to T7 in BATCH-2.md (opens in a new tab), written before any of those runs, and reproduce.ts (opens in a new tab), which recomputes every number the book quotes:
node experiment/results/reproduce.tsAnd the T7 run? Count passes and it was perfect; count what it left behind and it was one of fourteen that handed on a schema change with no test. Which answer you get depends on what you measure, and so does the argument about ceremony.
Questions people ask
Was the harness worth it?
Not in pass rate or cost: first passes were 88 to 90 of 90 in every condition, and cost per accepted change sat within 15 cents for each model. It showed up in what runs left behind, at the price of runs that stopped for an absent person; whether that pays is a judgement call.
Which model gave the cheapest accepted change?
Haiku: $0.04 to $0.06 per accepted change on T4 to T7, against $0.27 to $0.30 for Sonnet and $0.82 to $0.97 for Opus, with all 60 runs passing at the first “done”. It was enough for small, fully specified tasks, which says nothing about larger work.
Can I run the same experiment on my own code?
Yes: the repository holds the protocol, seven tasks with hidden acceptance tests and a scorer that counts failed runs in the cost per accepted change. Swap in tasks from your own backlog, run each condition at least five times, interleaved, and report the spread as well as the mean.
