Skip to main content

What Teams Actually Use for Verification Loops in 2026

A field guide to the sensors teams put in coding agent loops, from compilers and linters to Playwright, screenshots, judge models, evals, mutation testing and human review. What each one catches, what it misses, what it costs, how fast it answers and how noisy it is, plus a playbook for which to stack on which kind of change.

CatalEx Engineering · Published September 15, 2026 · 09:00 UTC

A coding agent finishes a change to a signup form and reports: "Implemented the new field, all tests pass, verified the form renders correctly." Every word of that is true. The type checker is happy, the unit tests for the validation function pass, and the component renders without throwing. On a phone, the new field pushes the submit button below a sticky footer, and nobody can sign up. No check in the loop could have seen it. None of them looked at a screen.

The agent did not lie. It reported what its sensors told it. The problem is that "verified" meant three cheap checks, all of which measure whether the code is well formed, and none of which measure whether a person can use the page.

Every team building with agents ends up assembling a stack of checks around them, some decades old (compilers, tests, CI) and some new (agent-driven browsers, judge models, evals). Each sees a different slice of reality at a different cost, speed and noise level. This post is a field guide to that stack as teams use it in 2026, and a playbook for which sensors to put on which kind of change.

What a sensor is, and the five things to know about one​

A sensor is anything that looks at the agent's work and returns a signal the agent (or a person) can act on: a compiler error, a failing test, a lint finding, a described screenshot, a judge's verdict, a review comment.

For each sensor you put in a loop, you want to know five things:

  1. What it catches. The class of defect it can see at all.
  2. What it misses. The classes it is blind to, however well you tune it.
  3. Cost. Compute, tokens, and human attention per run.
  4. Latency. Time from "agent finished an edit" to "agent hears back." Slow sensors run less often and get batched, which blurs the signal.
  5. Noise. How often it fails for reasons unrelated to the change, or passes when it should not. A noisy sensor teaches an agent to ignore it.

Most stacks go wrong on the fifth. A sensor right 95% of the time sounds fine until an agent runs it forty times a day.

The sensor taxonomy​

Costs and latencies are rough orders of magnitude for a mid-sized codebase, not benchmarks. Your numbers will differ; the ranking rarely does.

SensorCatchesMissesCost per runLatencyNoise
Compiler and type checkerWrong types, missing symbols, broken imports, unhandled cases in exhaustive matchesAny behaviour that type-checksNear zeroSecondsAlmost none
Linters and static analysis (ESLint, golangci-lint, Semgrep, CodeQL)Banned patterns, known bug shapes, security smells, your own architectural rulesLogic errors that look like normal codeNear zero to lowSeconds to a few minutesLow for precise rules, high for broad ones
Unit testsBehaviour of a function on the inputs someone wrote downInputs nobody wrote down; integration between parts; tests that assert the wrong thingLowSecondsLow, unless they touch clocks, network or shared state
Integration and contract testsParts that disagree: schemas, APIs, database queries, migrationsReal third-party behaviour when the fake differs from the real thingMediumMinutesMedium
CIEverything above, in a clean environment, against the merge baseWhatever the suites don't cover; anything only true on a laptopMedium (runner minutes)Minutes to tens of minutesInherits every flaky test it runs
Scripted browser tests (Playwright, Cypress)User journeys that break: buttons that don't submit, pages that don't load, flows that dead-endVisual and layout problems unless explicitly asserted; journeys nobody scriptedMedium to highMinutesMedium to high: timing, animations, test data
Agent-driven browser (Playwright MCP, computer use)Problems a person would notice exploring: broken flows, confusing states, errors on screenAnything it doesn't think to try; it can believe an action worked when it didn'tHigh (tokens)MinutesHigh run to run
Screenshots as evidenceLayout, overflow, contrast, wrong state on screen; before and after comparisonBehaviour between the captured states; anything off-screenLow to capture, attention to reviewSeconds to capturePixel diffs are noisy across platforms; described captures less so
LLM judge and reviewer agentsRule violations that need judgment, missing tests, suspicious logic, mismatches between intent and diffShared blind spots with the author model; anything outside the diff it was shownMedium to high (tokens)MinutesMedium: verdicts vary run to run and have known biases
EvalsRegressions in model-driven behaviour: prompts, tools, agent flowsInputs outside the eval set; drift after the set was writtenHighMinutes to hoursMedium, needs repeated runs
Property-based tests (Hypothesis, fast-check)Edge cases nobody listed: empty, huge, unicode, ordering, round-tripsProperties nobody statedLow to mediumSeconds to minutesLow; failures shrink to a minimal reproducing input
Mutation testingTests that run code without checking itNothing about the product directly; it grades the testsHigh if run on everything, low on a diffMinutesSome equivalent mutants are unavoidable
Human reviewWrong intent, bad design, risk, things nobody thought to checkWhatever the reviewer skims; large diffsThe most expensive minute in the loopHoursVaries by reviewer and by day

Read down the "Misses" column: every sensor is blind to something a later row can see, and later rows are slower and dearer. So teams layer them.

Cheap, deterministic sensors run on every edit and absorb most attempts. Expensive, judgment-based ones run once per change, on work that already passed everything below. A judge reviewing code that does not compile is money spent on the wrong layer.

Why "ask the agent to verify it" is not a sensor​

The first thing everyone tries is an instruction: "test your changes before you say you're done." It helps a little. But an agent's description of its verification is not verification.

Anthropic's own write-up on harnesses for long-running agents names the failure directly: Claude tended to mark a feature complete without proper testing, and did much better once explicitly prompted to use browser automation and test as a human user would. Their computer use documentation carries the same warning in a different form: Claude sometimes acts as if its actions succeeded without explicitly checking the results.

The fix is not a sterner instruction. Every claim of verification must point at something a sensor produced: a test log, a CI run on the exact commit, a described screenshot. No artifact, no verification. Claude Code's best practices put it simply: give Claude something it can run that comes back pass or fail.

Layer one: compilers, types and linters​

These are your best sensors: deterministic, nearly free, and fast enough to run on every edit.

Make the type system do more work. Exhaustive matches, branded id types and non-nullable fields turn runtime bugs into compile errors an agent fixes in one attempt. Strict mode is worth more with agents, because the agent pays the cost of satisfying it.

Write linters for your own rules, not just the community's. Semgrep is the common choice because its rules look like the code they match. If "handlers never call the database directly" lives in an instruction file today, it should be a Semgrep rule. An instruction is followed most of the time; a lint rule, every time.

Write lint messages for the agent. A finding that says "forbidden call" leaves the agent guessing. A finding that says "use store.Get instead; handlers must go through the store so tests can fake it" gets fixed on the first attempt. OpenAI's harness engineering post describes doing exactly this: lint error messages written to carry the remediation into the agent's context.

The limit: none of these sensors know what the code is for.

Layer two: tests and CI​

Agents write tests that pass. The risk is tests that assert whatever the code currently does, check a mock instead of the real thing, or had their expected value edited to match a bug. Anthropic's Claude Sonnet 4.5 system card lists tests that verify mocks rather than real implementations among the reward hacks it still sees.

Three cheap counters:

  1. Tests that drive the real thing: the real database in a container, the real HTTP stack. A fake that diverges from production is a sensor calibrated to the wrong instrument.
  2. Changed-line coverage. A floor on the lines this change touched stops untested branches without demanding a whole-codebase cleanup.
  3. Flag test edits that ride along with code edits, so a person looks.

CI is the clean room. Its job is to run the full suites somewhere the agent cannot influence: a fresh environment, against the current merge base. That is why CI noise is so expensive. Google's testing team reported in 2016 that about 1.5% of their test runs gave a flaky result, and that about 84% of the pass to fail transitions they observed involved a flaky test. An agent that sees CI fail and pass on rerun often enough learns that red means "rerun," which is the worst lesson it can learn.

SWE-bench is a useful reminder of what tests can certify. It grades agents by running the repository's own tests, which is strong evidence of "fixed this without breaking that" and none at all that a maintainer would accept the fix.

Layer three: browsers, screenshots and evals​

This is the layer that would have caught the signup form. It is also where most of the new tooling is, so it is worth separating the options.

Three ways to put a browser in the loop​

ApproachHow the agent sees the pageGood forWatch out for
Scripted tests (Playwright, Cypress)It doesn't. A script asserts on the DOMJourneys that must never break; running in CI on every changeOnly checks what someone scripted; brittle selectors
Playwright MCPStructured accessibility snapshots of the page, no vision model neededExploratory checks, reproducing a bug, writing the scripted test afterwardsToken cost per step; the README itself notes many coding agents now prefer a CLI for token efficiency
Computer useScreenshots, mouse and keyboardAnything with a screen and no DOM: desktop apps, canvases, native dialogsSlowest and most expensive; can misjudge whether an action worked

The pattern teams converge on: use the agent-driven browser to explore and reproduce, then turn what it found into a scripted test that runs in CI forever. Pay for the noisy sensor once; keep the cheap one.

Agent-driven browsing has blind spots. Anthropic's harness write-up notes Claude could not see browser-native alert dialogs through the Puppeteer MCP, and accessibility snapshots show no colour, overlap or canvas. Visual defects need pixels.

Screenshots as evidence​

Screenshots work best as evidence attached to a change, not as a pixel-diff gate. Playwright's visual comparisons are real and useful, but the docs are candid that screenshots differ between browsers and platforms because of rendering and fonts. Pixel-perfect gates across machines become a flake factory.

What works better for agent loops:

Three rules make screenshots trustworthy:

  1. Before and after, under comparable conditions: same viewport, theme and seed data. The new state alone proves nothing about what changed.
  2. Whoever captures it describes it ("submit button visible at 390 px, above the footer"). The text is checkable, and it forces the capturing agent to actually look.
  3. Never fabricate a baseline. A missing "before" is a known gap. A reconstructed one is false evidence.

The limit: an image proves a state, not a behaviour. A screenshot of a success page does not prove the payment went through.

Evals for model-driven behaviour​

When the change is to a prompt, a tool description, or an agent's instructions, unit tests stop being the right sensor because the output is not deterministic. That is what evals are for. Anthropic's Demystifying evals for AI agents is the most practical guide we know: 20 to 50 simple tasks drawn from real failures is a great start, grade the final state the agent produced where you can, and read the transcripts, because you won't know whether your graders work otherwise.

Run each case more than once, report pass rates rather than verdicts, and keep held-out cases the agent tuning the prompt never sees.

Layer four: judges, reviewers, and testing the tests​

LLM judges and reviewer agents​

A reviewer agent reads the diff and reports problems. It is the only automated sensor that can check rules requiring judgment ("this error message leaks an internal path," "this test doesn't test the behaviour its name claims"). It is also a sensor with known biases. The original MT-Bench study found strong judges agree with human preferences over 80% of the time, about the same rate humans agree with each other, and also documented position, verbosity and self-enhancement biases.

What makes reviewer agents useful rather than decorative:

  • Fresh context, read-only tools. The reviewer sees the diff and the rules, not the author's reasoning.
  • One concern per reviewer. One rule against a few hundred lines beats everything against everything.
  • Structured findings (rule, file, line, evidence) that can be counted.
  • A verifier pass that re-checks each finding, cutting the false positives that teach authors to ignore reviewers.
  • The author cannot dismiss findings. Fix them, or escalate.

The biggest limit is independence: a reviewer from the author's model family shares its blind spots. And mechanical rules ("no magic numbers") are a poor use of a judge; they are cheaper and more reliable as lint rules.

Property-based testing​

A property-based test states a rule that should hold for all inputs ("parsing then formatting returns the original," "the sorted list has the same elements") and lets a library such as Hypothesis or fast-check generate hundreds of inputs, then shrink any failure to a minimal case. It catches exactly the edge cases an agent's example-based tests skip. A 2025 paper on agentic property-based testing ran a Claude Code based agent that wrote Hypothesis tests across 100 Python packages and found real bugs.

Mutation testing​

Mutation testing changes the code on purpose (flip a comparison, delete a line) and checks that some test fails. A surviving mutant is code the tests run but do not check: the most direct sensor for "the agent's tests assert nothing." On a diff it is cheap, and that is how it runs at scale: Google's mutation testing system mutates only changed code during code review and surfaces surviving mutants to the author. Meta's ACH goes a step further, using an LLM to generate both the mutants and the tests that kill them.

Layer five: human review​

People are the slowest, most expensive and least consistent sensor, and the only one that can answer "is this what we meant?" Spend them on intent, design, data model, security boundaries and anything irreversible, not on formatting or rules a linter can enforce.

Make the human's job reading evidence, not gathering it. A pull request with a CI run on its head commit, described screenshots, resolved reviewer findings and a list of what was not verified takes minutes to review well. One that says "all tests pass" takes an hour, or gets approved unread.

A worked example: the signup form, again​

Here is the opening change with a sensor stack chosen for it. Each row adds a sensor, and the third column shows why it was not redundant with the ones above it.

Sensor addedWhat it caught or would catchWhy the earlier rows could not
Types and lintThe new field's value was optional in the form type but required by the API clientBaseline: always on
Unit tests on validationAn empty string passed validationTypes allow empty strings
Property test on the validatorA name made only of combining characters passed, then failed on the serverNobody wrote that example
Scripted Playwright journey: fill the form, submit, land on the welcome pageWould catch the hidden submit button only if the test ran at phone widthThe journey at desktop width passes
Before and after screenshots at 390 px, described by the capturing agentSubmit button hidden under the sticky footerNothing above looks at a rendered screen
Reviewer agent against the UI rulesThe new field had no accessible labelRequires reading the markup with a rule in mind
Human reviewAsked whether the field should exist at all on the first stepOnly a person owns that decision

The stack composes: cheap rows absorbed most attempts, so expensive rows ran once, on code that already passed. And the screenshot finding became permanent and cheap: the Playwright journey now also runs at phone width.

How to measure your sensors​

MetricHow to collect itWhat it tells you
Flake rate per sensorRerun on unchanged commits; count failures on main that pass on rerunWhich sensor to fix first
Time to signalTimestamp from push to result, per sensorWhich sensors are too slow for the inner loop
Catch rateSeed known bugs, or use mutation score on the diffWhether the sensor can see what you think it sees
Escapes by missing sensorFor each bug found after merge, record which sensor should have caught itWhere to add the next sensor
Cost per changeRunner minutes plus tokens per merged changeWhether a layer is earning its place
Judge agreementSample judge verdicts and have a person label themWhether a reviewer agent can be trusted on a rule

What this does not fix​

A sensor stack cannot verify a target nobody wrote down. If the spec never says the form must work on a phone, no layer is obliged to check it, and the stack will report green with complete honesty.

Agent-driven sensors (browser agents, judges, evals) are themselves nondeterministic. Stack too many and the loop spends more time arguing with its sensors than fixing code.

And independence is never complete. The more of your stack is model-driven, the more it matters that the bottom layers are deterministic and the top one is a person.

What it costs​

Browser journeys and evals cost runner minutes and tokens on every change. Reviewer agents cost tokens in proportion to diff size and rule count. Mutation testing is cheap only on a diff. Property tests take thought to write. And every sensor is one more thing that can flake, which someone has to own. The discipline is not to add every sensor to every change. It is to choose.

The playbook​

Pick the row that matches the change. "Always" means the bottom of the stack (types, lint, focused tests, CI) runs regardless.

SituationAdd these sensorsSkip or deferHuman looks at
Pure function or library logicUnit tests, a property test, mutation on the diffBrowser, screenshotsOnly if the interface changed
UI changeScripted journey at desktop and phone width, before and after screenshots with descriptions, reviewer agent on UI rulesMutation testingScreenshots and intent
Bug fixA test that fails before the fix and passes after, plus one line naming why the bug escapedBroad exploratory runsThe root cause note
Refactor with no behaviour changeFull existing suite, changed-line coverage, reviewer agent for accidental behaviour changeNew tests beyond coverage gapsOnly on large diffs
Database migrationUp, down and up again on the real engine, integration tests against the migrated schemaScreenshotsData-loss risk and rollback plan
Prompt, tool description or agent instructionEvals with repeated runs and held-out casesUnit tests on the prompt textTranscripts from failed eval cases
Security-sensitive pathSemgrep or CodeQL rules for the pattern, reviewer agent with a security rule setNothingAlways
Exploratory bug report from a userAgent-driven browser to reproduce, then convert to a scripted testJudgesThe reproduction
Change to the checks themselvesRun the old and new check on the same commits; compareNothingAlways: this is changing the judge

The tools will change. The principle will not: put the cheapest sensor that can see the defect closest to the agent, and save people for the questions only people can answer.


Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.