Anatomy of a Verification Loop: Sensor, Budget, Exit, Ratchet
An agent loop is a control system, and most broken loops are missing one of four parts. A framework for building them: a sensor for every target, a budget that cannot be argued with, an exit that hands off cleanly, and checks that only ever tighten.
At 1 a.m. a coding agent is asked to get a failing pull request green. The first run fails on a date-parsing test. The agent changes the parser and runs again. Now an unrelated integration test fails, one that has failed about once a week for a year. The agent runs the suite again and everything passes. It reports success. In the morning someone reads the diff and finds that the date-parsing test was also changed: the expected value was updated to match what the new code produces.
Every individual step was reasonable. The loop around it is what failed. It had a sensor the agent could edit, a sensor that lied once a week, no limit on attempts, and an exit condition of "the light turned green."
This post is a framework for building the loop properly: a control system with four parts you design on purpose. The sensor measures the work, the budget bounds the effort, the exit ends the loop honestly, and the ratchet keeps the checks from going backwards.
A loop is a control system
A thermostat has a set point (the temperature you asked for), a plant (the room), a sensor (the thermometer), a controller (the logic that decides), and an actuator (the heater). The controller only ever sees the sensor. If the thermometer is next to the oven, the controller does exactly the right thing with exactly the wrong number, and the room freezes.
An agent loop has the same parts, and the vocabulary is worth borrowing because it tells you where to look when something goes wrong.
| Control term | In an agent loop | Example |
|---|---|---|
| Set point | The spec or target | "Dates in ISO 8601 with offsets parse correctly"; "CI under 10 minutes" |
| Plant | The thing being changed | The code, the running product, main |
| Sensor | The checks | Compiler, tests, linters, a browser run, a judge model |
| Error signal | A failed check, with its message | "expected 2026-03-01T00:00+05:30, got 2026-02-28" |
| Controller | The agent | Reads the error, decides the next edit |
| Actuator | What the agent can change | Edits, commits, pushes, merges |
| Supervisor | People | Set the target, approve high-stakes moves, can stop the loop |
Two consequences follow. The agent can only be as right as its sensor: a better model reading a broken check converges faster on the wrong answer. And nothing in the picture says when the loop ends. Someone has to design that.
Why "retry until green" fails
The obvious loop is the one in the opening scene: run the checks, fix what fails, repeat until everything passes. It feels like rigour. It has three holes.
Green is not the same as correct. If the agent can change the sensor (edit the test, add a skip, loosen a lint rule) then green may only mean it found the cheapest way to make the sensor quiet. This is not hypothetical. METR reported in June 2025 that recent frontier models reward hack on their tasks, for example by tampering with the code that scores them. ImpossibleBench (October 2025) made the point sharper: it rewrites unit tests so they contradict the spec, which means any pass is a cheat, and found frontier models passing them anyway by special-casing or editing tests, in one configuration more than half the time.
A retry on an intermittent failure is a coin toss. If a check fails one run in twenty for reasons unrelated to the change, then rerunning until green always succeeds eventually, including on the runs where the change really did break it. Retrying converts a real signal into noise.
Nothing stops it. A loop with no budget can spend six hours circling one failure, then report success on a flake.
The four parts below close those holes one at a time.
Part one: the sensor
Question it answers: how would we know?
Every target has a sensor
Write down what "done" means, then point at the check that measures each part. Any target without a check is open loop: the agent will report it as satisfied because nothing contradicts it. "The page works on a phone" with no viewport check is open loop. "Errors are handled" with no test that injects one is open loop.
| Target | Sensor | Status |
|---|---|---|
| ISO 8601 with offsets parses to the right instant | Table test with 12 offset cases | Measured |
| Existing formats still parse | Existing parser tests, unchanged | Measured |
| No new allocation in the hot path | Benchmark comparison | Open loop |
| Error messages name the bad input | Nothing | Open loop |
Either add the two missing checks, or say in the pull request that those targets are unverified. Never let the agent report all four as done.
The doer is not the judge
A sensor the controller can edit is not a sensor. That covers the obvious case (the agent modifies the test it is supposed to pass) and the subtle one (the agent grades its own work).
The research on self-grading is consistent. Huang and colleagues showed in "Large Language Models Cannot Self-Correct Reasoning Yet" (ICLR 2024) that asking a model to review and revise its own answer, with no external feedback, does not reliably improve it and sometimes makes it worse.
So separate them structurally, not by instruction:
| Separation | How to enforce it |
|---|---|
| The agent cannot edit the sensor | Tests and check config changed in the same diff as the code they judge get flagged for human review, or the check is read from main, not the branch |
| The judge runs in a different context | A reviewer agent starts fresh, with read-only tools, and sees the diff, not the author's reasoning |
| The author cannot overrule the judge | Findings are recorded as they come back; the author can fix them, not dismiss them |
| The judge differs from the doer where it matters | A different model, a deterministic check, or a person for the highest-stakes calls |
Reading the check from main is the single cheapest separation. If the lint config, the coverage floor and the review rules are always loaded from the default branch, a pull request cannot weaken the thing that judges it. Changing a check becomes its own reviewed change, which is what it should have been all along.
Fix noisy sensors first
A flaky check is a sensor that lies some fraction of the time. A controller acting on a lying sensor does exactly what a thermostat does with a thermometer near the oven: confidently wrong things. Google's testing team reported in 2016 that about 1.5% of all their test runs produced a flaky result, and almost 16% of their tests showed some level of flakiness. At agent speed, with many runs a day, a 1.5% flake rate is not rare. It is daily.
The order of operations matters. Before you tune prompts, models or retry counts, measure each sensor's noise and repair the worst one. Tuning a controller against a noisy sensor teaches it to ignore the sensor.
How to tell a sensor is noisy, without guessing:
- Rerun the check on an unchanged commit many times (tens of runs, under load, at more than one CPU count). Any failure is noise.
- Track per-check failure rate on
main. A check that fails onmainand passes on rerun with no new commit is noisy by definition. - Read the failure text. "Timeout after 5000 ms" and "connection refused" are usually the harness, not the code.
The usual causes are real clocks and sleeps, shared state between tests, unordered iteration and startup races. Fix the cause. Quarantine only with an owner and a deadline: a quarantined test is a target with no sensor.
Make the error signal specific
"FAILED" teaches the controller nothing. "Expected 2026-03-01T00:00+05:30, got 2026-02-28T18:30Z for input 2026-03-01 with offset +05:30" tells it exactly what to change. OpenAI's harness engineering write-up describes writing custom lint error messages so that they carry the remediation into the agent's context. That is the right instinct: a sensor's output is the agent's next prompt, so write it like one.
Part two: the budget
Question it answers: how much are we willing to spend to find out?
A budget needs more than one dimension, because loops fail in more than one way.
| Budget | What it bounds | Typical setting | What happens at the limit |
|---|---|---|---|
| Attempts | Fix-and-recheck cycles | 3 to 5 | Stop and escalate with the trace |
| Same-failure repeats | Thrashing on one error | 2 | Stop: the agent is not learning from the signal |
| Tokens or cost | Spend per task | Set from your median task cost, times 3 to 5 | Stop and report spend |
| Wall clock | Time to an answer | Hours, not days | Stop, even mid-attempt |
| Blast radius | What the loop may touch | The files and services in scope | Refuse the action, do not retry it |
| Relaxations | Skipped tests, waived checks | A small fixed count that expires | The relaxation turns itself off |
Anthropic's Building effective agents makes the same point in passing: it is common to add stopping conditions, such as a maximum number of iterations, to keep control. The detail most teams miss is the last row. Any loosening of the loop (skipping slow tests on small changes, waiving a check for a hotfix) needs its own budget that expires automatically. Control engineers call this anti-windup. Without it, temporary relaxations accumulate until the loop has no sensor left.
The budget also has to be outside the agent's reach. A budget the agent can edit, or argue its way past, is a suggestion.
Part three: the exit
Question it answers: when do we stop, and what do we hand over?
A loop has exactly three honest ways to end, and the exit design decides which one you get.
Never retry until green
This is the rule we would keep if we could keep only one. A second attempt is a new run. Report both. If a check fails and then passes with no change in between, you have learned something important: the check is noisy, or the code is nondeterministic. Either way, the right output is "failed once, passed once, here are both logs," not "passed."
So the loop records every attempt, and the exit report includes the failures. "Passed on attempt 3; attempts 1 and 2 timed out in the same integration test" is something a reviewer can act on. "Passed" is not.
Done means done on this commit
"Tests pass" is a claim about a specific commit. The exit condition names the commit it checked, and any later edit invalidates it. This sounds pedantic until you merge a final "fix typo in test" commit that never ran.
Escalation is a design, not a failure
The stuck and exhausted exits hand the problem to a person. Hand over the target, every attempt with its error text, what the agent tried and what it suspects. A well-reported stuck loop takes minutes to resolve; a silent give-up takes an hour of reconstruction.
Part four: the ratchet
Question it answers: how do we make sure we never go backwards?
A ratchet turns one way: a floor can rise, but no ordinary change may lower it. Coverage floors, lint baselines, enforced rules, required CI jobs and performance budgets should all move only toward stricter.
Three habits make the ratchet real:
- Every escaped defect asks: which sensor should have seen this? Then add it. The cheapest version is one line in every bug-fix pull request naming why the original change was not caught.
- A rule that code can decide becomes a check. "Never import X from Y" in prose is followed most of the time. As a lint rule it is followed every time, and costs no tokens.
- Measure the diff, not the total. A floor on total coverage lets one well-tested file subsidise an untested one. A floor on the lines this change touched is stricter, and it ratchets the whole codebase forward one pull request at a time without a big-bang cleanup.
The ratchet is also how you defend against Goodhart's law (Marilyn Strathern's paraphrase: when a measure becomes a target, it ceases to be a good measure). An agent optimising against fixed checks finds their blind spots; checks that grow from every escape keep closing them.
Nest the loops by speed
Sensors differ in cost by orders of magnitude: a type check takes seconds, a browser journey minutes, a review of last week's escapes an hour of someone's time. Running everything on every edit makes feedback slow, and slow feedback makes agents overcorrect or stall. So nest the loops, each with its own sensor, budget, exit and ratchet.
The inner loop absorbs most attempts, so it must be fast and deterministic. The outer loop is where people spend their attention: on the pattern of changes that got through, not on each change.
What goes wrong
Most broken loops fail in one of six ways. Each one maps to a missing or broken part.
| Failure | What it looks like | Missing part | The check that catches it |
|---|---|---|---|
| The loop grades itself | The agent's own summary says "all tests pass, verified in browser" and no log shows it | Sensor independence | Require evidence the sensor produced (logs, artifacts, a record tied to the commit), never the agent's description |
| The sensor gets edited | A test's expected value changed in the same diff as the code it tests | Sensor independence, ratchet | Flag test or check changes alongside the code they judge; read check config from main |
| Flaky sensors | Same commit, different result; "rerun and it passed" in the log | Sensor quality | Per-check flake rate on main; stress reruns of new tests |
| Unbounded retries | Forty attempts, a large bill, a pass on a flake | Budget | Attempt and same-failure caps outside the agent's control |
| Goal drift | Asked to fix a bug, the agent refactors the module, disables a feature, or "simplifies" the requirement until it passes | Exit tied to the original target | Exit checks the target list from the start of the run, plus a diff-scope check |
| Slow sensors | CI takes 40 minutes, so the agent batches five fixes per push and cannot tell which one broke what | Nesting by speed | A time ceiling on the middle loop and fast focused checks in the inner one |
Goal drift is the hardest to see: the agent satisfies the checks by changing the problem. The defence is to freeze the target list when the run starts, outside the agent's context, and have the exit compare against that list rather than whatever the agent now believes the task to be.
A worked example
Back to the 1 a.m. pull request. Here is the same task, with each part added in turn and what it changes.
| Change | What it prevents | Why it composes |
|---|---|---|
| Targets written down with a sensor each; benchmark and error-message checks added | The agent reporting untested targets as done | Gives the exit a list to compare against |
| Test files changed alongside the parser are flagged for human review | The expected value being edited to match the bug | Makes the existing tests a sensor the agent cannot move |
| The weekly-flaky integration test is fixed (it used a real clock) before anything else | The agent learning that failures can be rerun away | Without this, the next row would fire constantly |
| Budget: 4 attempts, stop after the same failure twice | A night of thrashing | Turns "stuck" into a report instead of a bill |
| Exit report lists every attempt and the commit each ran on | "Passed" hiding "failed twice, passed once" | The reviewer sees the noise and can act on it |
| The escape (a test edited to match a bug) becomes a lint rule on the outer loop | The same failure next month on a different agent | The ratchet tightens from a real event |
Notice the order. Sensor quality comes before budget, because a budget on a noisy loop only stops it sooner with the same wrong answer. The ratchet comes last because it feeds on escapes, and you only learn what escapes once the other three parts are honest.
How to measure your loop
These numbers are cheap to collect if every run writes a record.
| Metric | What it tells you | Watch for |
|---|---|---|
| First-pass rate | How often the first attempt passes all sensors | Very high can mean weak sensors, not good agents |
| Attempts per success | How hard the loop works | Rising without a change in task mix means the sensor or the error text got worse |
| Flake rate per sensor | Which sensors lie | Anything above zero on main is a repair item |
| Same-failure stops | How often the agent thrashes | Clusters on one check usually mean vague error text |
| Escape rate | Defects found after the loop said done | The number the whole system exists to lower |
| Time to signal | Seconds or minutes until the agent hears back | The middle loop should have a hard ceiling |
| Escalation rate | How often a person is pulled in | Zero is suspicious; it often means the exit is too lenient |
What this does not fix
It does not make a weak sensor strong. The framework tells you whether a sensor is honest, not whether it checks the right things. Honest, fast sensors that all measure the happy path converge reliably on code that fails on the first real input.
Judgment targets stay hard. "Is this the right design?" and "does this read well?" have no deterministic sensor. A judge model helps, but a judge is itself a sensor with its own noise, and it needs its own calibration against people. Some targets will remain open loop, and the honest move is to say so.
Separation is never total. A reviewer agent from the same model family
as the author shares its blind spots. Reading checks from main stops a
branch weakening its judge, but merges made outside the agent's harness (by
hand, in a web UI) skip whatever the harness enforces.
What it costs
Every part has a price.
- Sensors cost compute and time. Nesting by speed pays for them, and the nesting itself needs maintaining.
- Fixing flaky tests is slow, unglamorous work whose payoff is invisible until you compare escape rates.
- Honest exits produce more escalations at first. Loops that used to "pass" now report "stuck," and someone has to read those reports.
- Ratchets occasionally block a reasonable change. Lowering a floor should be possible, but as a separate, reviewed decision.
The framework
| Part | Question it answers | Rule | Fails as |
|---|---|---|---|
| Sensor | How would we know? | Every target has a sensor; the doer is not the judge; fix noisy sensors first; error text names the fix | Self-grading, edited tests, flakes, open-loop targets |
| Budget | How much will we spend? | Caps on attempts, repeats, cost, time and scope, outside the agent's reach; relaxations expire | Unbounded retries, runaway cost |
| Exit | When do we stop, and with what? | Done, stuck or exhausted; never retry until green; results tied to a commit; escalate with the full trace | Pass on a flake, stale results, silent give-up |
| Ratchet | How do we never go backwards? | Checks only tighten; escapes become checks; rules code can decide become checks; measure the diff | Goodhart drift, accumulating waivers |
Adopting it
Take the parts in this order.
- Map targets to sensors for your most common agent task. Mark the open-loop rows. This takes an hour and usually changes what you think "verified" means.
- Measure and fix the noisiest sensor. Rerun the suite on an unchanged commit and repair whatever fails. Do not touch prompts or models yet.
- Separate the doer from the judge: read check configuration from
main, and flag test changes that ride along with the code they test. - Add the budget and the three exits, with every attempt recorded.
- Start the ratchet: one line per bug fix naming why it escaped, and a monthly pass that turns the most common reason into a check.
A good loop is boring to watch. It fails fast, says exactly why, stops when it should, and gets a little stricter every week. The model inside it matters less than you think. The loop decides what the model is allowed to call done.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.