Skip to main content

The Coverage Ratchet: How We Took Every Module to 90 Percent in a Week, and What Coverage Doesn't Tell You

In one week we went from measuring coverage to gating every module at 90 percent and every changed line at 95, with coding agents writing most of the tests. This is the four-part ratchet we used, why each part exists, and the limit we keep in view: coverage counts lines that ran, not behaviour that was checked.

CatalEx Engineering · Published October 20, 2026 · 09:00 UTC

On the afternoon of October 14, a pull request turned on coverage floors in our relay codebase: 85 percent of production statements for the two big Go modules, Local Relay and Hosted Relay. CI passed. About four hours later two more pull requests landed that raised both floors to 90 percent. Between them they carried roughly 7,800 lines of new tests, across command handlers, a console server, a controller and two chat connectors. Coding agents wrote almost all of them.

That speed is the interesting part, and also the dangerous part. When tests cost almost nothing to write, a coverage number stops being a measure of effort and becomes a target that something will optimise. You need the number to move in one direction only, and you need it to mean what you think it means.

This post describes the four parts of what we call the coverage ratchet: what we measure, the module floors, the changed-lines gate, and the direction rule. Each part answers one question. We also say plainly what the ratchet does not tell you, because a green coverage gate is easy to mistake for a correct system.

A target versus a ratchet​

A coverage target is a number you hit once. Somebody spends a sprint getting a module to 80 percent, the dashboard turns green, and from then on every change is free to erode it a little. Nobody notices a module drifting from 84 to 81 because no single change caused it.

A ratchet is a rule about direction, applied at every change. The floor can go up. It cannot go down. Every pull request is measured against what it touches, so erosion has a name and a diff attached.

With agents in the loop the difference matters more. An agent that is told "keep coverage above 80" will do exactly that, and no more. An agent that is told "never lower a package's coverage, and cover 95 percent of what you change" carries the bar into every change it makes, on every module, at whatever rate it merges.

Here is the week, as it happened.

DateChange
Oct 13Every Go module reports coverage, with production statements counted separately from generated mocks and test support
Oct 14, afternoonModule floors enforced in CI: Local and Hosted Relay at 85, the shared gateway library, the CLI client and the dashboard backend at 90
Oct 14, eveningLocal and Hosted Relay raised to 90, with the tests that meet it in the same pull requests
Oct 16Every package of the CLI client brought to 90. Pull requests gate changed lines at 90
Oct 17Written goal of 95 above the 90 floor; never lower a package; convert tests you edit to tables
Oct 19Changed-lines gate raised to 95. A write-tests skill for agents

Part one: measure the right denominator​

The first commit did not set any threshold. It changed what we counted.

Go's coverage profile lists every block of statements the instrumented packages contain, and whether a test ran it. If you divide covered by total over that whole profile, you get raw coverage. Raw coverage includes two kinds of code that are not product:

  • Generated mocks. Our tests use generated mocks for every interface. A mock's methods run constantly in tests, because that is their job. Counting them inflates the number, and the inflation grows as you add more mocks.
  • Test support. Fixtures, fake servers and helper packages that exist only for tests. They are close to 100 percent covered by construction.

So coverage_report.py splits the denominator. A file with Go's generated-code header, and anything under a dedicated testutil, testsupport or tests directory, leaves the production denominator. It is still listed, file by file, in the JSON output, so nothing disappears silently. The report prints both numbers.

What the coverage report counts
CodeRaw coverageProduction coverageWhy
Product packagesCountedCountedThis is what ships
Generated mocks (Code generated ... DO NOT EDIT.)CountedListed, not countedRuns in every test, proves nothing about the product
testutil, testsupport, tests directoriesCountedListed, not countedCovered by construction
A source path in the profile that no longer existsFails the reportFails the reportA stale profile is worse than none

Two details turned out to matter once agents started working against the number.

A combined percentage can hide a module. Hosted Relay's profile merges its own packages with the shared gateway library. One module at 97 and another at 84 can average above 90. The check therefore takes a minimum per module, and fails if a module it was asked about is missing from the report altogether.

Rounding can pass a failure. The check compares exact statement counts, not the displayed percentage. A module shown as 90.00 percent can still be one statement short, and it fails.

Part two: module floors, and why they moved twice in a day​

The floors went in at what the modules could pass that day: 85 for Local Relay, Hosted Relay and the tooling module, 90 for the CLI client, the dashboard backend and the shared gateway library. The frontend apps got their own thresholds from Vitest's V8 coverage: 85 percent of web statements, 90 of desktop statements, 90 percent of lines.

Four hours later both relay modules went to 90. That was not a change of heart. It was the cheapest moment to raise the bar. The first pull request put the floors into CI, checked on both database drivers. Each of the next two raised one module's floor and carried the tests that met it. The Hosted Relay one added 3,167 lines across 34 files. The Local Relay one added 4,696 across 46.

Raise a floor in the same change that meets it. A floor raised on its own breaks every open branch. A floor raised with its tests passes on day one and blocks only the regressions.

A module floor runs on every push to main, over the full suite. It is the backstop. It is also a poor gate for a single pull request, for a reason that is pure arithmetic.

Part three: gate the lines a change touches​

Take a module with 10,000 production statements at 96 percent coverage. A pull request adds 100 statements and no tests. The module is now at 9,600 of 10,100, which is 95.05 percent. The 90 percent floor passes. The new code is entirely untested, and the floor cannot see it, because it is averaging the new code with months of old code.

So on October 16 pull requests stopped running the module floor and started gating patch coverage: the share of changed production lines that a test ran.

The branch on the right is the one people forget. A changed line counts only if some profile holds a statement on it. A brand new production file that no test package instruments has no statements in any profile, so naively it has zero coverable lines and passes at 100 percent. The tool catches this case: a change that adds function code to a production file that no report instruments fails with PATCH_NOT_INSTRUMENTED.

The gate started at 90 percent on October 16 and went to 95 on October 19. Google's testing blog gives a useful frame for those numbers: it calls 90 percent "exemplary" for a whole project, says project-wide goals above 90 are probably not worth it, and suggests that per-commit goals of 99 percent are reasonable with 90 as a lower threshold (Code Coverage Best Practices). That is the shape we landed on independently: a strict bar on the diff, a looser bar on the whole.

The output is written for the agent that has to fix it. Every module's job prints one line, Patch coverage: P% (covered/total changed lines), and below the target it fails with PATCH_COVERAGE_BELOW_TARGET and a table of uncovered lines per file in the job summary. An agent reading a failed run does not need to run coverage locally. It reads the list, adds a case per line, and pushes.

Part four: the direction rule​

The last part is a rule, not a gate. On October 17 we wrote it into the instructions every agent in the repository reads:

AGENTS.md · coverage goal
  • The CI floor is 90 percent. The repository's goal is 95: production statements in every Go module, lines in each frontend app.
  • Cover at least 95 percent of the production lines you add or change, even while the gate requires 90.
  • Never lower a package's or module's coverage under the same suite selection.
  • When you touch a package below 95 percent, add meaningful cases for its uncovered paths in a separate commit.
  • Convert any Go behaviour test you edit that is not yet table-driven.

"Under the same suite selection" is doing real work there. Coverage depends on which tests ran. Comparing a full run against a selected run, or a SQLite run against a PostgreSQL run, produces a drop that is not a drop. The rule compares like with like.

The separate-commit rule keeps reviews readable. A feature change and forty new cases for neighbouring code are two different things to review.

Why the rule asks for tables​

The table-driven rule looks like style. It is really about where the cases live. A table-driven Go test is one TestXxx per behaviour, with a list of named cases: the input or starting state, the action, the expected outcome. Each case runs in its own t.Run.

When the gate says line 212 of a handler is uncovered, and line 212 is the branch that refuses an expired token, the fix in a table is one row: "expired token is refused". In a file of loose test functions the fix is a new function with its own setup, and the next agent writes a slightly different setup again. Tables make the missing case visible as a missing row.

The conversion was not small. One pull request on October 17 reworked behaviour tests across 171 files, adding 41,236 lines and removing 20,873. The same conventions also say what tables are not. A table is not a speed improvement, since each case still owns its fixture. And consolidating tests must keep every assertion and failure scenario, because "equal line coverage alone does not prove redundancy."

The write-tests skill​

On October 19, alongside the 95 percent gate, we added a write-tests skill that every agent loads when it changes production code. It turns the ratchet into a procedure:

.claude/skills/write-tests/SKILL.md · summary
  1. Before writing code, list the cases for each behaviour you add or change: the ordinary path, every error and refusal, boundaries such as empty, zero, maximum and over-limit, and each configuration branch.
  2. Go: one table-driven test per behaviour contract, generated mocks only, injected clocks and synctest instead of sleeps. Frontend: Vitest and Testing Library tests of what a person sees and does.
  3. Run only the tests you wrote, locally. CI runs the suites and measures coverage.
  4. Read each module's patch coverage line in CI. For every uncovered line, add a row to the table that owns that behaviour.
  5. Never exclude production code, weaken an assertion, skip a test or add a test without assertions to raise the number.
  6. A line that cannot run in a test, such as a process exit after a fatal start-up error, is named in the pull request with its reason.

Rule five is the important one. Every way of gaming a coverage number is listed there by name, because an agent under a failing gate will try the cheapest move that turns it green.

A worked example​

Here is a typical change under the full ratchet, described as each gate sees it. The change and its numbers are illustrative, not one specific pull request. An agent adds a retry with a cap to a connector's send path, in a package that sits at 93 percent.

StepWhat the agent doesWhich part of the ratchet applies
PlanLists cases: success first try, success after one retry, cap reached, non-retryable error, context cancelled during backoffwrite-tests, step 1
First pushTests the first three cases. 31 of 36 changed lines covered, 86 percentChanged-lines gate fails and lists 5 lines
Read the failureThe five lines are the non-retryable branch and the cancel branchGate output names them
Second pushAdds two rows to the existing table36 of 36, 100 percent
Package below 95Adds cases for two old uncovered error paths, in a separate commitDirection rule
Edited an old non-table testConverts it to a table while thereTable rule
Merge to mainFull suite runs, module floor checkedModule floor, the backstop

Notice what composes. The changed-lines gate makes the new code honest. The direction rule pays down a little of the old code each time someone passes through. The floor catches the case where both were skipped. No single part does all three jobs.

How to measure it​

If you build this, these are the numbers worth having on a dashboard, and the ones we print in CI today:

  • Production statement coverage per module, with mocks and test support excluded and listed. Track it on main only, from the full run.
  • Changed-line coverage per pull request, per module, with the uncovered lines listed in the job summary, not only the percentage.
  • The excluded list. Review it. If it grows, ask why. A directory renamed to testsupport should not be a way out.
  • Floor headroom. How far each module sits above its floor. A module far above is the next one to raise.

What coverage does not tell you​

We want to be precise about the limit here, because the ratchet makes it easier to forget.

Coverage tells you a line ran during a test. It does not tell you the test checked anything about what that line did. The Google post above says it in one sentence: coverage "does not guarantee that the covered lines or branches have been tested correctly, it just guarantees that they have been executed by a test." Its suggested complement is mutation testing, which changes the code and checks whether any test notices.

With agents writing the tests, the gap has specific shapes:

How coverage misleadsWhat it looks likeThe check that catches it
Executed, not assertedA case calls the handler and asserts only that no panic happenedReview for an assertion on the outcome; mutation testing
The mock agrees with itselfA test sets a mock to return X, then asserts the code returned XAssert on what the product does with X, not on X
No statement to countA changed constant, type or struct tag has no statement, so the gate sees nothingTests of the behaviour the constant controls
Fake diverges from realityA fake provider accepts input the real one refusesContract tests against recorded real responses
Integration only works in partsEach package covered, the end-to-end path never runAn end-to-end suite that CI actually executes

The honest position is that the ratchet removes one failure, untested code, and leaves the others where they were. Over the next weeks we will watch the signals that coverage cannot give us: how many merged changes are fixes, what the fixes say about why the bug got through, and whether bugs land in code the gate reported as covered. If those stay flat while coverage climbs, the number is measuring effort, not safety.

What it costs​

  • Review load. One week produced tens of thousands of lines of test code. Nobody reads 41,000 lines line by line. Tests become code that needs its own quality bar, and that bar is harder to automate than a percentage.
  • CI time. Coverage instrumentation runs on every pull request and on every main push, across integration suites with real databases. It is not free on shared runners.
  • Friction on small fixes. A two-line fix in a package at 88 percent now carries a separate commit of cases. That is the point, and it is still a cost someone pays at an inconvenient moment.
  • Lines that cannot run. Fatal start-up exits and similar paths have to be named in the pull request. That is a judgement call reviewers now make.

The ratchet, as a framework​

PartThe question it answersOur setting
MeasureWhat counts as code we are responsible for?Production statements; mocks and test support listed separately; exact counts; per-module minimums
Module floorIs any module drifting?90 percent on main for the Go modules except tooling at 85; frontend thresholds per app
Changed-lines gateIs this change tested?95 percent of changed production lines per module; uncovered lines listed; uninstrumented new code fails
Direction ruleDoes every change leave things better?Never lower a package; add cases when touching one below 95; convert edited tests to tables

If you are adopting it, the order matters:

  1. Fix the denominator first. A floor on raw coverage rewards mocks.
  2. Set floors at what each module passes today, so the change merges green.
  3. Raise each floor only in a pull request that carries the tests meeting it.
  4. Add the changed-lines gate, with output an agent can act on line by line.
  5. Write the direction rule and the anti-gaming list where agents read them.
  6. Decide, before the numbers look good, which signal will tell you whether coverage is buying correctness.

A ratchet makes sure the number only goes one way. It cannot tell you the number is the right one.


Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.