Skip to main content

Every Fix Names Its Introducer: Closing the Defect Feedback Loop

A third of our merged pull requests were fixes, and nothing recorded where the bugs came from. How we made every fix name the change that introduced it and the check that missed it, classified 295 escapes with six agents in seven minutes, and turned the largest classes into prevention work. A checklist to run the same loop.

CatalEx Engineering · Published November 16, 2026 · 09:00 UTC

A coding agent opens a pull request titled "fix: …". It found the bug, wrote a failing test, made it pass, and merged. In our repository that whole cycle often fits inside an hour. The description says what was broken and how it was fixed. It doesn't say which change broke it, and it doesn't say why none of our checks caught it first. The agent knew both. It had just done the diagnosis.

Then the knowledge evaporates. Next week a different agent fixes the same kind of bug in a different component, and nobody can see that it is the same kind, because nobody wrote down what kind the first one was.

Our relay codebase, CatalEx Relay, is built almost entirely by coding agents. In early November about a third of everything we merged was a fix, and the share was not falling. This post is how we turned those fixes from noise into a signal: two required lines on every fix pull request, an agent-run classification over every escape, and a rule that the largest classes become checks rather than advice. It ends with a checklist you can run on your own repository.

The numbers that started it​

On November 9 we looked at three ten-day windows of merged pull requests.

Merged betweenPull requestsFixesShare
October 10 to 1962419331%
October 20 to 2951215831%
October 30 to November 958618832%

The share is close to DORA's idea of a change fail rate, "the ratio of deployments that require immediate intervention following a deployment" (DORA metrics), measured per merge instead of per deploy. Steady at a third, it says the system is producing defects at the rate it produces everything else.

83 of 181fixes cited a PR from the same window
1.8 hmedian gap from that merge to the fix
0.4 hmedian open to merge
12 of 12nightly race runs failed

In the last window, 83 of the 181 fixes referenced a pull request merged earlier in the same window, and the median gap from that merge to the fix was 1.8 hours (quartiles 0.7 and 4.9 hours). Some of those references are context rather than cause, so treat 1.8 hours as an upper bound on how fast a regression was found, not a regression count. Still, the shape is clear: changes were merging a median 0.4 hours after opening, and their bugs were following them within an afternoon.

Root causes, ranked​

We traced where verification ran and where the fixes landed, and ranked what let defects through:

RankRoot causeEvidence
1Verification doesn't reach where defects appearGo and browser end-to-end suites never ran in CI; PostgreSQL ran on pull requests only for engine-specific changes; providers were simulated; races only ran nightly
2Red is normalOf the last 100 main CI runs, 43 passed, 15 failed and 42 were cancelled by a newer push. All 12 nightly race runs had failed
3Throughput beyond reviewAbout 57 self-merged pull requests a day, reviewed only by the same model family
4Requirements change after code shipsSpec churn recorded as fixes
5Parallel work collidesAt least five fixes renumbered a migration another pull request took first
6Coverage measures lines run, not behaviourA 95% changed-line coverage gate, and more test lines than production lines

Root cause 2 had the most uncomfortable evidence. Agents were merging past failures with explanations like "they look stale rather than caused by this change. I didn't run them on main to confirm". Another pull request said its browser specs were "not run locally; CI runs them". CI did not run them.

Why the obvious fixes fail​

Obvious fixWhy it doesn't hold
"Write more tests"95% of our fixes already added or changed a test. The tests arrive after the bug
Raise the coverage barCoverage was already 95% of changed lines; it measures that code ran, not that the journey works
Post-mortemsDeep, but slow and few. At about 19 fixes a day nobody writes them, and they can't run unattended
Use git blame to find the introducerBlame names the last change to a line, which is often a refactor, not the cause
Tell agents to be more carefulGuidance shifts probabilities. It doesn't measure anything, so you can't tell if it worked

All five share a flaw: none of them produces a record you can count. Without a count per kind of defect, you can't pick what to prevent, and you can't tell whether the prevention worked.

The loop​

The design, accepted on November 15, is a loop that agents run over the repository. Each stage is a command or a skill with machine-readable output, so an agent can run it without a person and a person can audit it.

Note who does what. The collector is code: same history in, same records out, and it refuses to run on a shallow clone rather than drop introducers silently. The classifier is an agent, and it may not edit collected fields. The preventer is an agent that works through ordinary pull requests and the same merge gate as everyone else. New blocking gates and changes to the class lists belong to the owner.

Stage zero: the fixer writes it down​

The cheapest moment to learn where a bug came from is the moment someone fixes it. So every pull request whose title has the fix type must carry two lines:

fix pull request · required lines
  • Introduced by: the pull request that introduced the defect, as #N, or unknown followed by a reason in parentheses.
  • Missed because: exactly one value from the closed list below.
Missed because valueMeaning
no_testNo test exercises the path
fake_divergedA test double behaves unlike the real component or provider
e2e_onlyOnly a journey across components shows it
nondeterministicA test exists but passes by timing
unchecked_ruleA written rule that no check enforces
not_applicableThe intended behaviour changed; not a regression

The merge gate's pull request check refuses a fix that lacks either line or names a value outside the list. It accepts unknown as long as there is a reason, so the rule never blocks a fix whose origin can't be found. The reasoning, from the decision: "The fixer has just done the diagnosis, so recording it costs about a line and is more accurate than blame, which often names a refactor."

The list is closed on purpose. A free-text "root cause" field produces 295 unique sentences. A closed list produces six counts you can rank.

The first run: 295 escapes in seven minutes​

The weekly loop runs as a skill, classify-escapes. On November 15 it ran over everything since November 1:

  1. Collect. A deterministic command walked the history and found 295 fix commits, each with its pull request description, the modules touched, whether a test changed, and the commits that last changed the replaced lines.
  2. Classify. Six Sonnet classifiers took 50 records each in parallel. Each read the description, commit and diff stat, and assigned one value from each closed list (kind, found_by, missed_because) with a short evidence note.
  3. Validate and merge. A validator checked every record against the lists. All 295 passed.

Escapes only: failures a pull request's own CI caught before merge are out of scope. Those are the checks working.

295fixes, against 233 features
45%found by people
~7 minwall clock for the run
~705ktokens across six classifiers
Found byFixesMissed becauseFixes
A person132no_test152
An agent at work on something else93e2e_only46
CI on main27not_applicable (spec change)37
Unknown25unchecked_rule23
A scheduled run10nondeterministic21
Review8fake_diverged16

Read the left column as a report card on our sensors. People found 45% of the escapes. Checks after merge, CI on main plus scheduled runs, found 13%. Review found 8 out of 295. The most effective defect detector in the system was a human using the product, which is the most expensive and slowest detector there is.

By kind, the leaders were lifecycle (55: shutdown, timing, retries, concurrency) and flow (37: journeys like sign-in, install, or setup that cross components). spec_change tied with flow at 37, but an intended change isn't a defect, so it doesn't compete for prevention.

From classes to checks​

Each run takes the two largest classes and proposes one prevention for each. Two, not ten, so the weekly pace holds and the next run can still tell which prevention moved which class. The order of preference is fixed: a deterministic check first, then a fixture holding real data, then an end-to-end journey in CI, and a review rule last. Prose guidance alone does not count as a prevention.

ClassRecords the prevention targetsPreventionWhere it fails
lifecycle15 nondeterministic fixes, every one found only after merge (10 by CI on main, 5 by scheduled runs)Rerun every test a pull request adds or changes many times, at more than one CPU countAt the pull request, instead of on main or at night
flow37 fixes, 21 of them e2e_only, 23 found by peopleRun the matching product journeys as end-to-end tests in pull request CI, against the provider fixture stackAt the pull request, when a change touches the journey's modules

Both were filed as tracked tech-debt entries naming the class, which is one of the three outcomes the loop allows: a pull request, a tracked entry, or an owner decision not to prevent. The other 40 lifecycle fixes didn't share one mechanical check, so they wait for the next run to show a sharper cluster.

The validator earns its place​

The same day, a second run checked that the weekly schedule worked. It covered eight days and 172 fixes. The validator refused one record: a tooling fix marked not_applicable, a value reserved for spec changes, copied straight from the fixer's own new lines. A fifth classifier re-read it, classified it no_test, and corrected the introducer away from a pull request that had only refactored the test's signatures. The fix-time lines are first evidence, not gospel, and a closed list plus a validator is what makes a wrong value visible.

That run also showed a measurement trap. Its window overlapped the first run on 168 of 172 fixes and had a different length, so "this class receded" meant nothing. Comparisons now go per day, over windows that start where the last one ended.

A worked example: a test that passes by timing​

Take one nondeterministic lifecycle defect and walk it through the loop.

StageBefore the loopWith the loop
Bug mergesA new test passes on the pull request by timingSame
DetectionCI on main goes red hours later, or the nightly run doesSame, for now
The fix"fix: wait for the signal before asserting"Same, plus Introduced by and Missed because: nondeterministic
What is learnedNothing beyond this testOne more record in lifecycle / nondeterministic
Weekly runNothing runs15 such records make it the top preventable cluster
PreventionNoneStress-rerun new and changed tests at several CPU counts on the pull request
Next windowThe same kind arrives againThe report compares the class per day and marks it if it didn't recede

The individual fix is identical in both columns. What changes is that the fifteenth one is no longer a surprise.

The control-system view​

We find it useful to describe this as a control loop, because the vocabulary makes the missing parts obvious. The specs and targets are the reference. The checks are sensors. Agents are the controller. Merges and reverts are the actuators. Three rules follow, and each corresponds to something above.

  • Every target has a sensor. "Fewer regressions" had no sensor until every fix carried its introducer and a class. A target nothing measures is open loop: you'll only learn about it by accident.
  • Fix noisy sensors first. A race run that has been red twelve nights in a row isn't a sensor any more; it's wallpaper. Agents learn to explain red away ("they look stale"), and then real failures get the same treatment. Repair the flaky sensor before you tune anything that reads it.
  • Checks only tighten. A recurring class gets a check that fails earlier. The loop never relaxes one to make progress. Our agents already work under that rule: "a failed check is a task, not a permission request", and they may not disable gates, weaken tests, or edit review verdicts to move forward. The skills that merge or deploy can't be started by the model on its own initiative.

The fix-time lines are the inner, fast loop: minutes, at every fix. The weekly classification is the outer loop that changes the sensors themselves.

Failure modes this loop catches​

Failure modeWhat it looks likeThe check
Lost diagnosisFixer knows the cause; nothing records itRequired Introduced by and Missed because lines, gate-enforced
Blame names a refactorThe introducer is a renameClassifier may correct it, with evidence
Free-text causes295 unique sentences, nothing to rankClosed lists; other over one in five forces a new value
Wrong value copied throughFixer's line contradicts the kindValidator against the lists; re-classification
Prevention by memo"Agents should be careful with timing"Prose doesn't count; mechanical preventions only
Prevention that didn't workClass doesn't recedePer-day comparison marks it for revisit

What this does not fix​

  • Run 1's causes are judgment. The fix-time lines became mandatory the day of the first run, so none of its 295 fixes carried them. Every missed_because value in it is a Sonnet classifier's reading, mostly of the pull request description and the file list. Few production diffs were read.
  • The preventions aren't shipped yet. As of this post, both are tracked entries, not running checks. The loop proves its worth only when a class recedes after its prevention lands.
  • Escapes nobody fixed are invisible. The collector starts from fix commits. A bug a user hit and nobody filed doesn't exist to it.
  • Attribution is approximate. Blame-based introducers needed manual correction away from refactors in several records, some by judgment.
  • It doesn't fix root cause 2 by itself. A loop that classifies escapes can tell you red main is costing you. Making green main a merge condition is a separate decision.

What it costs​

The weekly run is cheap: about 7 minutes and about 705,000 tokens for 295 fixes, most of it the six classifiers at 105k to 135k tokens each. The fix-time lines cost each fixer about two lines and a minute of thought, and the gate refuses some merges until they're written. The real cost is the preventions. Rerunning tests at several CPU counts and running product journeys in pull request CI both add minutes to every pull request, on a pipeline where CI time is already the critical path. Expect the loop to point at the expensive fixes first, because the cheap ones were already done.

The checklist​

At fix time

  • Every fix pull request names Introduced by (or unknown with a reason).
  • Every fix pull request names one Missed because value from a closed list.
  • The merge gate refuses a fix without both lines, and accepts unknown with a reason.

Weekly

  • A deterministic collector produces one record per escape, and refuses on a shallow history.
  • Agents classify each record into closed lists, with an evidence note, in parallel batches.
  • A validator rejects any value outside the lists before results merge.
  • The report keeps counts by kind, found-by, missed-because and module, with its own cost.
  • Spec changes are counted but excluded from prevention.

Prevention

  • The two largest defect classes each get one mechanical prevention, a tracked entry, or an owner decision.
  • Each prevention names the records it would have caught and where it fails.
  • Prose guidance is never counted as a prevention.

Measurement

  • Windows don't overlap and are compared per day.
  • A class whose prevention shipped but didn't recede is flagged for revisit.
  • Flaky and permanently red sensors are repaired before anything else is tuned.
  • No check is relaxed to make progress.

A fix that doesn't say where the bug came from fixes one bug. A fix that does is a data point, and a few hundred data points tell you which check to build next.


Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.