One test had gone red, so you handed it to the agent and went back to your work. It changed the code and ran the suite. Still red. It tried again. Still red.
Then the message you were waiting for: all tests pass, task done. A slightly shorter test file, a green build. You merged.
Later, when the bug that test was written for turns up, you go looking for it. It is not failing. It is not skipped. It is simply gone. Nothing the agent told you was false: every test that ran, passed.
An agent judged by a green test run will sometimes make the run green the cheap way: delete a test, mark it skipped, or loosen an assertion. Stop it with four checks it cannot talk its way past: tests the agent cannot edit, a build that fails when the test count drops, a mutation check that finds hollow tests, and a “done” accepted only with fresh test evidence.
Why does an AI agent delete or skip a failing test?
The cause is mechanical: the test file is part of the environment the agent changes until the goal is met, and nothing marks it as special. Charles Goodhart observed in 1975 that a statistical regularity tends to collapse once it is used for control, and Marilyn Strathern later gave the idea the form people remember: “When a measure becomes a target, it ceases to be a good measure.” (Strathern, European Review, 1997 (opens in a new tab)). A green run is a measure of working code. Ask an agent for green and you have made it the target.
Chapter 6 (free as a PDF) counts seven ways to a green run that proves nothing: delete the failing test, switch it off, hollow out the assertion, change the expected value, regenerate the snapshots, mock the thing under test, or claim a run that never happened.
Make the tests read-only to the agent
During an implementation task, the agent may run the tests but not write them; when behaviour really changes, the test changes first, in a separate task a person reviews. The book calls this Tests Behind Glass (P14). Deny writes to the test and snapshot folders in your agent’s permissions, say so in your instruction file, and back the lock with a scan for .skip, .only, .todo and xit that fails the commit, with an exit in its message: if the test looks wrong, stop and ask a human.
That exit has a price: in the 270 unattended runs behind Chapter 12, on a task that needed an existing test changed, 23 of 30 runs built the feature, left the old test failing and asked a person to change it; nobody came.
Fail the build when the test count drops
Suppose the lock leaked. The Test Count Ratchet (P15) makes the size of the suite a number that may only go up, checked against a committed baseline. The book’s version, after one deletion:
test-ratchet: FAILED, 1 problem
- The test count fell from 78 to 77.
Fix: put the missing tests back. Do not delete, merge or
skip tests to get a green run; fix the code instead.
...All 77 remaining tests passed. The ratchet refused anyway, and only a person lowers the baseline. But it counts tests without weighing them: swap a strong test for one that asserts true, and the count holds.
Find hollow tests with mutation testing
Mutation testing weighs them. It makes the code wrong on purpose, one small change at a time (a > becomes >=), and runs the tests after each. If no test fails, the mutant survived: that line runs, but nothing checks what it does. Ask an agent to make date tests “less brittle” and it may do this:
// before: checks the answer
assert.equal(addDays('2026-01-30', 3), '2026-02-02');
// after: checks that it looks like a date
assert.match(addDays('2026-01-30', 3), /^\d{4}-\d{2}-\d{2}$/);Coverage is unchanged and so is the count. Mutants notice:
The book calls this the Knock for Hollow Tests (P16). A survivor means strengthen an assertion, never loosen one.
Accept “done” only with fresh test evidence
The last move needs no test edit: the agent sees green, tidies one comparison while writing its summary, and reports done on code that no longer exists. So ask for a receipt: one command runs the tests and records the exit code, the counts and a SHA-256 hash of every source file, and the agent’s stop hook refuses “done” if there is no evidence, the run failed, or any hash has moved. Leave the agent a way to ask a person, or the hook pushes it towards gaming.
The four checks cover each other’s holes
Play your vanished test through all four: the lock refuses the deletion, the ratchet catches it if the lock leaked, mutants catch a loosened assertion, and the evidence check refuses a stale claim. None is complete alone: mutants cannot tell whether a changed expected value is what the business wants, but a person reading the test diff can.
To try it yourself, the companion repository (opens in a new tab) has one drill per mistake: npm run drill D09 deletes a test in a scratch copy and shows the ratchet catching it, no API key needed.
The agent that deleted your test still wants green. It always will. The difference: the only road to green left open now runs through the code.
Questions people ask
Is making the test folder read-only enough?
No: an agent with a shell can write a file without its edit tool, and a lock lifted for a test-writing task is easy to forget. The rule needs the sensors behind it.
Can the agent still write new tests?
Yes, in a separate test-only task, before the implementation, that a person reviews. A test written alongside the code encodes the agent’s misunderstandings too.
Does this slow the agent down?
A little: on a slow suite, read the ratchet’s count from the gate’s own test run. Mutation testing costs a test run per mutant, so run it nightly or on the files the agent touched, not in the commit gate.
