Every Fix Names Its Introducer: Closing the Defect Feedback Loop
A third of our merged pull requests were fixes, and nothing recorded where the bugs came from. How we made every fix name the change that introduced it and the check that missed it, classified 295 escapes with six agents in seven minutes, and turned the largest classes into prevention work. A checklist to run the same loop.
A coding agent opens a pull request titled "fix: …". It found the bug, wrote a failing test, made it pass, and merged. In our repository that whole cycle often fits inside an hour. The description says what was broken and how it was fixed. It doesn't say which change broke it, and it doesn't say why none of our checks caught it first. The agent knew both. It had just done the diagnosis.
Then the knowledge evaporates. Next week a different agent fixes the same kind of bug in a different component, and nobody can see that it is the same kind, because nobody wrote down what kind the first one was.
Our relay codebase, CatalEx Relay, is built almost entirely by coding agents. In early November about a third of everything we merged was a fix, and the share was not falling. This post is how we turned those fixes from noise into a signal: two required lines on every fix pull request, an agent-run classification over every escape, and a rule that the largest classes become checks rather than advice. It ends with a checklist you can run on your own repository.
The numbers that started it
On November 9 we looked at three ten-day windows of merged pull requests.
| Merged between | Pull requests | Fixes | Share |
|---|---|---|---|
| October 10 to 19 | 624 | 193 | 31% |
| October 20 to 29 | 512 | 158 | 31% |
| October 30 to November 9 | 586 | 188 | 32% |
The share is close to DORA's idea of a change fail rate, "the ratio of deployments that require immediate intervention following a deployment" (DORA metrics), measured per merge instead of per deploy. Steady at a third, it says the system is producing defects at the rate it produces everything else.
In the last window, 83 of the 181 fixes referenced a pull request merged earlier in the same window, and the median gap from that merge to the fix was 1.8 hours (quartiles 0.7 and 4.9 hours). Some of those references are context rather than cause, so treat 1.8 hours as an upper bound on how fast a regression was found, not a regression count. Still, the shape is clear: changes were merging a median 0.4 hours after opening, and their bugs were following them within an afternoon.
Root causes, ranked
We traced where verification ran and where the fixes landed, and ranked what let defects through:
| Rank | Root cause | Evidence |
|---|---|---|
| 1 | Verification doesn't reach where defects appear | Go and browser end-to-end suites never ran in CI; PostgreSQL ran on pull requests only for engine-specific changes; providers were simulated; races only ran nightly |
| 2 | Red is normal | Of the last 100 main CI runs, 43 passed, 15 failed and 42 were cancelled by a newer push. All 12 nightly race runs had failed |
| 3 | Throughput beyond review | About 57 self-merged pull requests a day, reviewed only by the same model family |
| 4 | Requirements change after code ships | Spec churn recorded as fixes |
| 5 | Parallel work collides | At least five fixes renumbered a migration another pull request took first |
| 6 | Coverage measures lines run, not behaviour | A 95% changed-line coverage gate, and more test lines than production lines |
Root cause 2 had the most uncomfortable evidence. Agents were merging past
failures with explanations like "they look stale rather than caused by this
change. I didn't run them on main to confirm". Another pull request said its
browser specs were "not run locally; CI runs them". CI did not run them.
Why the obvious fixes fail
| Obvious fix | Why it doesn't hold |
|---|---|
| "Write more tests" | 95% of our fixes already added or changed a test. The tests arrive after the bug |
| Raise the coverage bar | Coverage was already 95% of changed lines; it measures that code ran, not that the journey works |
| Post-mortems | Deep, but slow and few. At about 19 fixes a day nobody writes them, and they can't run unattended |
Use git blame to find the introducer | Blame names the last change to a line, which is often a refactor, not the cause |
| Tell agents to be more careful | Guidance shifts probabilities. It doesn't measure anything, so you can't tell if it worked |
All five share a flaw: none of them produces a record you can count. Without a count per kind of defect, you can't pick what to prevent, and you can't tell whether the prevention worked.
The loop
The design, accepted on November 15, is a loop that agents run over the repository. Each stage is a command or a skill with machine-readable output, so an agent can run it without a person and a person can audit it.
Note who does what. The collector is code: same history in, same records out, and it refuses to run on a shallow clone rather than drop introducers silently. The classifier is an agent, and it may not edit collected fields. The preventer is an agent that works through ordinary pull requests and the same merge gate as everyone else. New blocking gates and changes to the class lists belong to the owner.
Stage zero: the fixer writes it down
The cheapest moment to learn where a bug came from is the moment someone
fixes it. So every pull request whose title has the fix type must carry two
lines:
- Introduced by: the pull request that introduced the defect, as
#N, orunknownfollowed by a reason in parentheses. - Missed because: exactly one value from the closed list below.
Missed because value | Meaning |
|---|---|
no_test | No test exercises the path |
fake_diverged | A test double behaves unlike the real component or provider |
e2e_only | Only a journey across components shows it |
nondeterministic | A test exists but passes by timing |
unchecked_rule | A written rule that no check enforces |
not_applicable | The intended behaviour changed; not a regression |
The merge gate's pull request check refuses a fix that lacks either line or
names a value outside the list. It accepts unknown as long as there is a
reason, so the rule never blocks a fix whose origin can't be found. The
reasoning, from the decision: "The fixer has just done the diagnosis, so
recording it costs about a line and is more accurate than blame, which often
names a refactor."
The list is closed on purpose. A free-text "root cause" field produces 295 unique sentences. A closed list produces six counts you can rank.
The first run: 295 escapes in seven minutes
The weekly loop runs as a skill, classify-escapes. On November 15 it ran
over everything since November 1:
- Collect. A deterministic command walked the history and found 295 fix commits, each with its pull request description, the modules touched, whether a test changed, and the commits that last changed the replaced lines.
- Classify. Six Sonnet classifiers took 50 records each in parallel.
Each read the description, commit and diff stat, and assigned one value
from each closed list (
kind,found_by,missed_because) with a short evidence note. - Validate and merge. A validator checked every record against the lists. All 295 passed.
Escapes only: failures a pull request's own CI caught before merge are out of scope. Those are the checks working.
| Found by | Fixes | Missed because | Fixes | |
|---|---|---|---|---|
| A person | 132 | no_test | 152 | |
| An agent at work on something else | 93 | e2e_only | 46 | |
CI on main | 27 | not_applicable (spec change) | 37 | |
| Unknown | 25 | unchecked_rule | 23 | |
| A scheduled run | 10 | nondeterministic | 21 | |
| Review | 8 | fake_diverged | 16 |
Read the left column as a report card on our sensors. People found 45% of
the escapes. Checks after merge, CI on main plus scheduled runs, found 13%.
Review found 8 out of 295. The most effective defect detector in the
system was a human using the product, which is the most expensive and
slowest detector there is.
By kind, the leaders were lifecycle (55: shutdown, timing, retries,
concurrency) and flow (37: journeys like sign-in, install, or setup that
cross components). spec_change tied with flow at 37, but an intended
change isn't a defect, so it doesn't compete for prevention.
From classes to checks
Each run takes the two largest classes and proposes one prevention for each. Two, not ten, so the weekly pace holds and the next run can still tell which prevention moved which class. The order of preference is fixed: a deterministic check first, then a fixture holding real data, then an end-to-end journey in CI, and a review rule last. Prose guidance alone does not count as a prevention.
| Class | Records the prevention targets | Prevention | Where it fails |
|---|---|---|---|
lifecycle | 15 nondeterministic fixes, every one found only after merge (10 by CI on main, 5 by scheduled runs) | Rerun every test a pull request adds or changes many times, at more than one CPU count | At the pull request, instead of on main or at night |
flow | 37 fixes, 21 of them e2e_only, 23 found by people | Run the matching product journeys as end-to-end tests in pull request CI, against the provider fixture stack | At the pull request, when a change touches the journey's modules |
Both were filed as tracked tech-debt entries naming the class, which is one
of the three outcomes the loop allows: a pull request, a tracked entry, or an
owner decision not to prevent. The other 40 lifecycle fixes didn't share
one mechanical check, so they wait for the next run to show a sharper
cluster.
The validator earns its place
The same day, a second run checked that the weekly schedule worked. It
covered eight days and 172 fixes. The validator refused one record: a tooling
fix marked not_applicable, a value reserved for spec changes, copied
straight from the fixer's own new lines. A fifth classifier re-read it,
classified it no_test, and corrected the introducer away from a pull request
that had only refactored the test's signatures. The fix-time lines are first
evidence, not gospel, and a closed list plus a validator is what makes a
wrong value visible.
That run also showed a measurement trap. Its window overlapped the first run on 168 of 172 fixes and had a different length, so "this class receded" meant nothing. Comparisons now go per day, over windows that start where the last one ended.
A worked example: a test that passes by timing
Take one nondeterministic lifecycle defect and walk it through the loop.
| Stage | Before the loop | With the loop |
|---|---|---|
| Bug merges | A new test passes on the pull request by timing | Same |
| Detection | CI on main goes red hours later, or the nightly run does | Same, for now |
| The fix | "fix: wait for the signal before asserting" | Same, plus Introduced by and Missed because: nondeterministic |
| What is learned | Nothing beyond this test | One more record in lifecycle / nondeterministic |
| Weekly run | Nothing runs | 15 such records make it the top preventable cluster |
| Prevention | None | Stress-rerun new and changed tests at several CPU counts on the pull request |
| Next window | The same kind arrives again | The report compares the class per day and marks it if it didn't recede |
The individual fix is identical in both columns. What changes is that the fifteenth one is no longer a surprise.
The control-system view
We find it useful to describe this as a control loop, because the vocabulary makes the missing parts obvious. The specs and targets are the reference. The checks are sensors. Agents are the controller. Merges and reverts are the actuators. Three rules follow, and each corresponds to something above.
- Every target has a sensor. "Fewer regressions" had no sensor until every fix carried its introducer and a class. A target nothing measures is open loop: you'll only learn about it by accident.
- Fix noisy sensors first. A race run that has been red twelve nights in a row isn't a sensor any more; it's wallpaper. Agents learn to explain red away ("they look stale"), and then real failures get the same treatment. Repair the flaky sensor before you tune anything that reads it.
- Checks only tighten. A recurring class gets a check that fails earlier. The loop never relaxes one to make progress. Our agents already work under that rule: "a failed check is a task, not a permission request", and they may not disable gates, weaken tests, or edit review verdicts to move forward. The skills that merge or deploy can't be started by the model on its own initiative.
The fix-time lines are the inner, fast loop: minutes, at every fix. The weekly classification is the outer loop that changes the sensors themselves.
Failure modes this loop catches
| Failure mode | What it looks like | The check |
|---|---|---|
| Lost diagnosis | Fixer knows the cause; nothing records it | Required Introduced by and Missed because lines, gate-enforced |
| Blame names a refactor | The introducer is a rename | Classifier may correct it, with evidence |
| Free-text causes | 295 unique sentences, nothing to rank | Closed lists; other over one in five forces a new value |
| Wrong value copied through | Fixer's line contradicts the kind | Validator against the lists; re-classification |
| Prevention by memo | "Agents should be careful with timing" | Prose doesn't count; mechanical preventions only |
| Prevention that didn't work | Class doesn't recede | Per-day comparison marks it for revisit |
What this does not fix
- Run 1's causes are judgment. The fix-time lines became mandatory the
day of the first run, so none of its 295 fixes carried them. Every
missed_becausevalue in it is a Sonnet classifier's reading, mostly of the pull request description and the file list. Few production diffs were read. - The preventions aren't shipped yet. As of this post, both are tracked entries, not running checks. The loop proves its worth only when a class recedes after its prevention lands.
- Escapes nobody fixed are invisible. The collector starts from fix commits. A bug a user hit and nobody filed doesn't exist to it.
- Attribution is approximate. Blame-based introducers needed manual correction away from refactors in several records, some by judgment.
- It doesn't fix root cause 2 by itself. A loop that classifies escapes
can tell you red
mainis costing you. Making greenmaina merge condition is a separate decision.
What it costs
The weekly run is cheap: about 7 minutes and about 705,000 tokens for 295 fixes, most of it the six classifiers at 105k to 135k tokens each. The fix-time lines cost each fixer about two lines and a minute of thought, and the gate refuses some merges until they're written. The real cost is the preventions. Rerunning tests at several CPU counts and running product journeys in pull request CI both add minutes to every pull request, on a pipeline where CI time is already the critical path. Expect the loop to point at the expensive fixes first, because the cheap ones were already done.
The checklist
At fix time
- Every
fixpull request namesIntroduced by(orunknownwith a reason). - Every
fixpull request names oneMissed becausevalue from a closed list. - The merge gate refuses a fix without both lines, and accepts
unknownwith a reason.
Weekly
- A deterministic collector produces one record per escape, and refuses on a shallow history.
- Agents classify each record into closed lists, with an evidence note, in parallel batches.
- A validator rejects any value outside the lists before results merge.
- The report keeps counts by kind, found-by, missed-because and module, with its own cost.
- Spec changes are counted but excluded from prevention.
Prevention
- The two largest defect classes each get one mechanical prevention, a tracked entry, or an owner decision.
- Each prevention names the records it would have caught and where it fails.
- Prose guidance is never counted as a prevention.
Measurement
- Windows don't overlap and are compared per day.
- A class whose prevention shipped but didn't recede is flagged for revisit.
- Flaky and permanently red sensors are repaired before anything else is tuned.
- No check is relaxed to make progress.
A fix that doesn't say where the bug came from fixes one bug. A fix that does is a data point, and a few hundred data points tell you which check to build next.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.