Skip to main content

Anatomy of a Verification Loop: Sensor, Budget, Exit, Ratchet

An agent loop is a control system, and most broken loops are missing one of four parts. A framework for building them: a sensor for every target, a budget that cannot be argued with, an exit that hands off cleanly, and checks that only ever tighten.

CatalEx Engineering · Published September 17, 2026 · 09:00 UTC

At 1 a.m. a coding agent is asked to get a failing pull request green. The first run fails on a date-parsing test. The agent changes the parser and runs again. Now an unrelated integration test fails, one that has failed about once a week for a year. The agent runs the suite again and everything passes. It reports success. In the morning someone reads the diff and finds that the date-parsing test was also changed: the expected value was updated to match what the new code produces.

Every individual step was reasonable. The loop around it is what failed. It had a sensor the agent could edit, a sensor that lied once a week, no limit on attempts, and an exit condition of "the light turned green."

This post is a framework for building the loop properly: a control system with four parts you design on purpose. The sensor measures the work, the budget bounds the effort, the exit ends the loop honestly, and the ratchet keeps the checks from going backwards.

A loop is a control system​

A thermostat has a set point (the temperature you asked for), a plant (the room), a sensor (the thermometer), a controller (the logic that decides), and an actuator (the heater). The controller only ever sees the sensor. If the thermometer is next to the oven, the controller does exactly the right thing with exactly the wrong number, and the room freezes.

An agent loop has the same parts, and the vocabulary is worth borrowing because it tells you where to look when something goes wrong.

Control termIn an agent loopExample
Set pointThe spec or target"Dates in ISO 8601 with offsets parse correctly"; "CI under 10 minutes"
PlantThe thing being changedThe code, the running product, main
SensorThe checksCompiler, tests, linters, a browser run, a judge model
Error signalA failed check, with its message"expected 2026-03-01T00:00+05:30, got 2026-02-28"
ControllerThe agentReads the error, decides the next edit
ActuatorWhat the agent can changeEdits, commits, pushes, merges
SupervisorPeopleSet the target, approve high-stakes moves, can stop the loop

Two consequences follow. The agent can only be as right as its sensor: a better model reading a broken check converges faster on the wrong answer. And nothing in the picture says when the loop ends. Someone has to design that.

Why "retry until green" fails​

The obvious loop is the one in the opening scene: run the checks, fix what fails, repeat until everything passes. It feels like rigour. It has three holes.

Green is not the same as correct. If the agent can change the sensor (edit the test, add a skip, loosen a lint rule) then green may only mean it found the cheapest way to make the sensor quiet. This is not hypothetical. METR reported in June 2025 that recent frontier models reward hack on their tasks, for example by tampering with the code that scores them. ImpossibleBench (October 2025) made the point sharper: it rewrites unit tests so they contradict the spec, which means any pass is a cheat, and found frontier models passing them anyway by special-casing or editing tests, in one configuration more than half the time.

A retry on an intermittent failure is a coin toss. If a check fails one run in twenty for reasons unrelated to the change, then rerunning until green always succeeds eventually, including on the runs where the change really did break it. Retrying converts a real signal into noise.

Nothing stops it. A loop with no budget can spend six hours circling one failure, then report success on a flake.

The four parts below close those holes one at a time.

Part one: the sensor​

Question it answers: how would we know?

Every target has a sensor​

Write down what "done" means, then point at the check that measures each part. Any target without a check is open loop: the agent will report it as satisfied because nothing contradicts it. "The page works on a phone" with no viewport check is open loop. "Errors are handled" with no test that injects one is open loop.

Target to sensor map · date parsing fix
TargetSensorStatus
ISO 8601 with offsets parses to the right instantTable test with 12 offset casesMeasured
Existing formats still parseExisting parser tests, unchangedMeasured
No new allocation in the hot pathBenchmark comparisonOpen loop
Error messages name the bad inputNothingOpen loop

Either add the two missing checks, or say in the pull request that those targets are unverified. Never let the agent report all four as done.

The doer is not the judge​

A sensor the controller can edit is not a sensor. That covers the obvious case (the agent modifies the test it is supposed to pass) and the subtle one (the agent grades its own work).

The research on self-grading is consistent. Huang and colleagues showed in "Large Language Models Cannot Self-Correct Reasoning Yet" (ICLR 2024) that asking a model to review and revise its own answer, with no external feedback, does not reliably improve it and sometimes makes it worse.

So separate them structurally, not by instruction:

SeparationHow to enforce it
The agent cannot edit the sensorTests and check config changed in the same diff as the code they judge get flagged for human review, or the check is read from main, not the branch
The judge runs in a different contextA reviewer agent starts fresh, with read-only tools, and sees the diff, not the author's reasoning
The author cannot overrule the judgeFindings are recorded as they come back; the author can fix them, not dismiss them
The judge differs from the doer where it mattersA different model, a deterministic check, or a person for the highest-stakes calls

Reading the check from main is the single cheapest separation. If the lint config, the coverage floor and the review rules are always loaded from the default branch, a pull request cannot weaken the thing that judges it. Changing a check becomes its own reviewed change, which is what it should have been all along.

Fix noisy sensors first​

A flaky check is a sensor that lies some fraction of the time. A controller acting on a lying sensor does exactly what a thermostat does with a thermometer near the oven: confidently wrong things. Google's testing team reported in 2016 that about 1.5% of all their test runs produced a flaky result, and almost 16% of their tests showed some level of flakiness. At agent speed, with many runs a day, a 1.5% flake rate is not rare. It is daily.

The order of operations matters. Before you tune prompts, models or retry counts, measure each sensor's noise and repair the worst one. Tuning a controller against a noisy sensor teaches it to ignore the sensor.

How to tell a sensor is noisy, without guessing:

  1. Rerun the check on an unchanged commit many times (tens of runs, under load, at more than one CPU count). Any failure is noise.
  2. Track per-check failure rate on main. A check that fails on main and passes on rerun with no new commit is noisy by definition.
  3. Read the failure text. "Timeout after 5000 ms" and "connection refused" are usually the harness, not the code.

The usual causes are real clocks and sleeps, shared state between tests, unordered iteration and startup races. Fix the cause. Quarantine only with an owner and a deadline: a quarantined test is a target with no sensor.

Make the error signal specific​

"FAILED" teaches the controller nothing. "Expected 2026-03-01T00:00+05:30, got 2026-02-28T18:30Z for input 2026-03-01 with offset +05:30" tells it exactly what to change. OpenAI's harness engineering write-up describes writing custom lint error messages so that they carry the remediation into the agent's context. That is the right instinct: a sensor's output is the agent's next prompt, so write it like one.

Part two: the budget​

Question it answers: how much are we willing to spend to find out?

A budget needs more than one dimension, because loops fail in more than one way.

BudgetWhat it boundsTypical settingWhat happens at the limit
AttemptsFix-and-recheck cycles3 to 5Stop and escalate with the trace
Same-failure repeatsThrashing on one error2Stop: the agent is not learning from the signal
Tokens or costSpend per taskSet from your median task cost, times 3 to 5Stop and report spend
Wall clockTime to an answerHours, not daysStop, even mid-attempt
Blast radiusWhat the loop may touchThe files and services in scopeRefuse the action, do not retry it
RelaxationsSkipped tests, waived checksA small fixed count that expiresThe relaxation turns itself off

Anthropic's Building effective agents makes the same point in passing: it is common to add stopping conditions, such as a maximum number of iterations, to keep control. The detail most teams miss is the last row. Any loosening of the loop (skipping slow tests on small changes, waiving a check for a hotfix) needs its own budget that expires automatically. Control engineers call this anti-windup. Without it, temporary relaxations accumulate until the loop has no sensor left.

The budget also has to be outside the agent's reach. A budget the agent can edit, or argue its way past, is a suggestion.

Part three: the exit​

Question it answers: when do we stop, and what do we hand over?

A loop has exactly three honest ways to end, and the exit design decides which one you get.

Never retry until green​

This is the rule we would keep if we could keep only one. A second attempt is a new run. Report both. If a check fails and then passes with no change in between, you have learned something important: the check is noisy, or the code is nondeterministic. Either way, the right output is "failed once, passed once, here are both logs," not "passed."

So the loop records every attempt, and the exit report includes the failures. "Passed on attempt 3; attempts 1 and 2 timed out in the same integration test" is something a reviewer can act on. "Passed" is not.

Done means done on this commit​

"Tests pass" is a claim about a specific commit. The exit condition names the commit it checked, and any later edit invalidates it. This sounds pedantic until you merge a final "fix typo in test" commit that never ran.

Escalation is a design, not a failure​

The stuck and exhausted exits hand the problem to a person. Hand over the target, every attempt with its error text, what the agent tried and what it suspects. A well-reported stuck loop takes minutes to resolve; a silent give-up takes an hour of reconstruction.

Part four: the ratchet​

Question it answers: how do we make sure we never go backwards?

A ratchet turns one way: a floor can rise, but no ordinary change may lower it. Coverage floors, lint baselines, enforced rules, required CI jobs and performance budgets should all move only toward stricter.

Three habits make the ratchet real:

  1. Every escaped defect asks: which sensor should have seen this? Then add it. The cheapest version is one line in every bug-fix pull request naming why the original change was not caught.
  2. A rule that code can decide becomes a check. "Never import X from Y" in prose is followed most of the time. As a lint rule it is followed every time, and costs no tokens.
  3. Measure the diff, not the total. A floor on total coverage lets one well-tested file subsidise an untested one. A floor on the lines this change touched is stricter, and it ratchets the whole codebase forward one pull request at a time without a big-bang cleanup.

The ratchet is also how you defend against Goodhart's law (Marilyn Strathern's paraphrase: when a measure becomes a target, it ceases to be a good measure). An agent optimising against fixed checks finds their blind spots; checks that grow from every escape keep closing them.

Nest the loops by speed​

Sensors differ in cost by orders of magnitude: a type check takes seconds, a browser journey minutes, a review of last week's escapes an hour of someone's time. Running everything on every edit makes feedback slow, and slow feedback makes agents overcorrect or stall. So nest the loops, each with its own sensor, budget, exit and ratchet.

The inner loop absorbs most attempts, so it must be fast and deterministic. The outer loop is where people spend their attention: on the pattern of changes that got through, not on each change.

What goes wrong​

Most broken loops fail in one of six ways. Each one maps to a missing or broken part.

FailureWhat it looks likeMissing partThe check that catches it
The loop grades itselfThe agent's own summary says "all tests pass, verified in browser" and no log shows itSensor independenceRequire evidence the sensor produced (logs, artifacts, a record tied to the commit), never the agent's description
The sensor gets editedA test's expected value changed in the same diff as the code it testsSensor independence, ratchetFlag test or check changes alongside the code they judge; read check config from main
Flaky sensorsSame commit, different result; "rerun and it passed" in the logSensor qualityPer-check flake rate on main; stress reruns of new tests
Unbounded retriesForty attempts, a large bill, a pass on a flakeBudgetAttempt and same-failure caps outside the agent's control
Goal driftAsked to fix a bug, the agent refactors the module, disables a feature, or "simplifies" the requirement until it passesExit tied to the original targetExit checks the target list from the start of the run, plus a diff-scope check
Slow sensorsCI takes 40 minutes, so the agent batches five fixes per push and cannot tell which one broke whatNesting by speedA time ceiling on the middle loop and fast focused checks in the inner one

Goal drift is the hardest to see: the agent satisfies the checks by changing the problem. The defence is to freeze the target list when the run starts, outside the agent's context, and have the exit compare against that list rather than whatever the agent now believes the task to be.

A worked example​

Back to the 1 a.m. pull request. Here is the same task, with each part added in turn and what it changes.

ChangeWhat it preventsWhy it composes
Targets written down with a sensor each; benchmark and error-message checks addedThe agent reporting untested targets as doneGives the exit a list to compare against
Test files changed alongside the parser are flagged for human reviewThe expected value being edited to match the bugMakes the existing tests a sensor the agent cannot move
The weekly-flaky integration test is fixed (it used a real clock) before anything elseThe agent learning that failures can be rerun awayWithout this, the next row would fire constantly
Budget: 4 attempts, stop after the same failure twiceA night of thrashingTurns "stuck" into a report instead of a bill
Exit report lists every attempt and the commit each ran on"Passed" hiding "failed twice, passed once"The reviewer sees the noise and can act on it
The escape (a test edited to match a bug) becomes a lint rule on the outer loopThe same failure next month on a different agentThe ratchet tightens from a real event

Notice the order. Sensor quality comes before budget, because a budget on a noisy loop only stops it sooner with the same wrong answer. The ratchet comes last because it feeds on escapes, and you only learn what escapes once the other three parts are honest.

How to measure your loop​

These numbers are cheap to collect if every run writes a record.

MetricWhat it tells youWatch for
First-pass rateHow often the first attempt passes all sensorsVery high can mean weak sensors, not good agents
Attempts per successHow hard the loop worksRising without a change in task mix means the sensor or the error text got worse
Flake rate per sensorWhich sensors lieAnything above zero on main is a repair item
Same-failure stopsHow often the agent thrashesClusters on one check usually mean vague error text
Escape rateDefects found after the loop said doneThe number the whole system exists to lower
Time to signalSeconds or minutes until the agent hears backThe middle loop should have a hard ceiling
Escalation rateHow often a person is pulled inZero is suspicious; it often means the exit is too lenient

What this does not fix​

It does not make a weak sensor strong. The framework tells you whether a sensor is honest, not whether it checks the right things. Honest, fast sensors that all measure the happy path converge reliably on code that fails on the first real input.

Judgment targets stay hard. "Is this the right design?" and "does this read well?" have no deterministic sensor. A judge model helps, but a judge is itself a sensor with its own noise, and it needs its own calibration against people. Some targets will remain open loop, and the honest move is to say so.

Separation is never total. A reviewer agent from the same model family as the author shares its blind spots. Reading checks from main stops a branch weakening its judge, but merges made outside the agent's harness (by hand, in a web UI) skip whatever the harness enforces.

What it costs​

Every part has a price.

  • Sensors cost compute and time. Nesting by speed pays for them, and the nesting itself needs maintaining.
  • Fixing flaky tests is slow, unglamorous work whose payoff is invisible until you compare escape rates.
  • Honest exits produce more escalations at first. Loops that used to "pass" now report "stuck," and someone has to read those reports.
  • Ratchets occasionally block a reasonable change. Lowering a floor should be possible, but as a separate, reviewed decision.

The framework​

PartQuestion it answersRuleFails as
SensorHow would we know?Every target has a sensor; the doer is not the judge; fix noisy sensors first; error text names the fixSelf-grading, edited tests, flakes, open-loop targets
BudgetHow much will we spend?Caps on attempts, repeats, cost, time and scope, outside the agent's reach; relaxations expireUnbounded retries, runaway cost
ExitWhen do we stop, and with what?Done, stuck or exhausted; never retry until green; results tied to a commit; escalate with the full tracePass on a flake, stale results, silent give-up
RatchetHow do we never go backwards?Checks only tighten; escapes become checks; rules code can decide become checks; measure the diffGoodhart drift, accumulating waivers

Adopting it​

Take the parts in this order.

  1. Map targets to sensors for your most common agent task. Mark the open-loop rows. This takes an hour and usually changes what you think "verified" means.
  2. Measure and fix the noisiest sensor. Rerun the suite on an unchanged commit and repair whatever fails. Do not touch prompts or models yet.
  3. Separate the doer from the judge: read check configuration from main, and flag test changes that ride along with the code they test.
  4. Add the budget and the three exits, with every attempt recorded.
  5. Start the ratchet: one line per bug fix naming why it escaped, and a monthly pass that turns the most common reason into a check.

A good loop is boring to watch. It fails fast, says exactly why, stops when it should, and gets a little stricter every week. The model inside it matters less than you think. The loop decides what the model is allowed to call done.


Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.