Seven Rounds: Find Every Defect Before You Fix Any
Between October 6 and 8 an agent drove a real browser through our relay's full product journey seven times, against a throwaway deployment of main with live agent turns. It fixed nothing during a pass. Defects per round went 14, 16, 17, 6, 6, 3, 4. Here is the method and the rules that made it converge.
On October 6 an agent was partway through a checklist in a
headless Chromium window. It sent a message to a chat in our relay's web
console, waited 0.7 seconds, and killed the controller process with
kill -9. Then it brought the deployment back up and watched. The browser
re-sent the same command, which was correctly deduplicated. The session
stayed in state unknown, with the input queued, for more than nine minutes
and across two restarts. Start new session was refused with
RECOVERY_REQUIRED. Even dev down was refused.
The agent did not try to fix it. It wrote the defect down with its exact repro, marked it high, and moved to the next item on the list. By the end of the pass it had fourteen.
That pass was the first of seven. Over roughly 39 hours, from the first
report to the last, the same agent-driven journey ran against a fresh
deployment of main seven times. Between rounds, the defects were fixed in
small groups of pull requests, each proven live before merging. Each new
round re-checked everything, not only what had been fixed. The defect counts
per round were 14, 16, 17, 6, 6, 3 and 4. From round five onward the verdict
read: "The product flow works end to end."
This post is about the method: why we separate finding from fixing, how a round is set up, what made the counts go down instead of sideways, and the handful of rules that kept the process honest.
Why fixing as you go fails
The instinct, when you find a bug during testing, is to fix it right away. You understand it now, the context is loaded, and the fix looks small. With an agent doing the testing the instinct is stronger, because the same agent that found it can write the patch in a minute.
It goes wrong in four ways, and they compound.
- The system under test changes mid-pass. After the first fix, the remaining results describe a build that never existed: half the checklist ran on the old code, half on the new. A pass that ends green tells you nothing about any single commit.
- You stop at the first blocker. If the first defect is a crash, fixing it eats the session. Everything past it in the journey goes unseen, and the next defect (often a worse one) waits for another day.
- Fixes follow discovery order, not importance. The first thing you hit gets fixed first. A medium bug in setup gets attention ahead of a high one in recovery, because setup comes first in the journey.
- Nobody can group the work. Five defects in the same subsystem, found one at a time and fixed one at a time, become five patches that each touch the same code. Found together, they become one coherent change.
A pass is a measurement. You do not adjust the instrument while you are reading it. Every round report we wrote says the same sentence near the top: "Nothing was fixed in this pass; each defect below has its own repro."
How a round is set up
Every round followed the same shape, so that rounds could be compared.
A throwaway deployment of main. A fresh relay dev init on the current
head of main, with an embedded SQLite store, a mock peer and a two-minute
idle timeout so that idle-close behaviour shows up within a round. The commit
was recorded at the top of each report. Nothing from the previous round
carried over.
A real browser. Chromium, driven by Playwright, using the console as a person would: creating chats in folders, attaching an image, a CSV and a WAV file, answering typed forms, allowing and denying permissions, reacting, quoting, stopping a turn, reloading. A recorder attached over the DevTools protocol wrote every WebSocket frame, page error and console error to a log, so every verdict had frames behind it.
Live agent turns. The relay's job is to carry conversations with coding agents, so the agents on the other end were real: Codex on a small model at low effort, and Claude Code on Haiku. The cheapest models were a deliberate choice. The product was under test, not the models' intelligence, and cheap turns meant the journey could run end to end many times.
Written assumptions. Each report opened with what it assumed and why. Two mattered most. A model-quality problem (for example, an imperfect description of an image) is not a relay defect. And anything that needed an external account (a real Slack workspace or WhatsApp number) was marked BLOCKED, never quietly skipped or faked.
A fixed vocabulary. Every item ended as PASS, FAIL, PARTIAL or BLOCKED, with its evidence in the same row.
What a defect looks like
A defect in a round report is written so that someone who never saw the run can fix it. That means the fixer can be a different agent, in a different session, a day later. Here is the first one from round one, condensed:
| Field | Content |
|---|---|
| Title | A bad quote or reaction handle ends the call as TOOL_UNAVAILABLE, not a correctable refusal |
| Feature | The chat tools that let an agent quote a message or react to it |
| Repro 1 | Codex quotes an AI message using a message ID from history, without first fetching the current handles |
| Repro 2 | Claude Code reacts and quotes using the session ID as the handle |
| Expected | A schema error naming the bad field, with a fix the agent can follow, or accept the history message ID as a target |
| Actual | "This tool cannot serve this call now. Do not retry it; continue without it." Codex lost its quote. Claude Code dropped its reaction and chart and told the user the tools were unavailable |
| Cause | The resolver returned a generic unavailable error for a bad handle |
The pattern is the same for all 66: feature, numbered repro steps, expected, actual, and the cause where the run made it visible. A defect without a repro is a rumour, and rumours do not get fixed in grouped pull requests.
The seven rounds
| Round | Defects | Examples from the report | Fixed between rounds by |
|---|---|---|---|
| 1 | 14 | A bad quote handle kills the tool call (D1); a crash during a native load leaves the session stuck in unknown (D14); a revoked attachment still offers Download; the inline audio player shows no duration | Fixes for all 14 merged before round 2 |
| 2 | 16 | The gateway crashes with SIGBUS in SQLite's write-ahead-log index at start (7 of 19 up runs); restoring archived chats restores empty chats; reloading a chat over 100 messages sends 55 identical history requests in 4 ms | Three groups: console and harness, session lifecycle, store and recovery |
| 3 | 17 | After a second crash with work pending, every bridge attach fails with a disk I/O error; the history flood is down from 55 requests to 4, but the target is 2; a reaction shows as the raw reaction prompt | Three groups: console, bridge and approvals, crash recovery |
| 4 | 6 | A crash during a native load leaves the chat unable to run anything until a reset (H1, the one blocker); the history reload now sends exactly 2 requests | Three groups, including chats unstuck after a question crash |
| 5 | 6 | A resent input after a crash gets a failed receipt although it ran; a Claude Code background command outlives the idle close | Two groups plus a gateway fix |
| 6 | 3 | Some Bash permission requests are denied without being shown; restored chats wait minutes before they can be used | Two fixes plus a test synchronisation fix |
| 7 | 4 | A message over the queue limit is stuck as "Submitting" and lost on reload | Stopped; remaining gaps filed as one issue |
Two things in this table are worth noticing.
The count went up before it went down. Round two found more than round one, and round three more than round two. That is not the fixes making things worse. Each fix group unblocked parts of the journey that earlier rounds could not reach: once crash recovery stopped hanging, the rounds could exercise what happens after recovery, and found problems there. A rising count in the early rounds means the test is getting deeper.
Severity fell faster than the count. By round four the only blocker was H1, and by round five there was none. The last three rounds found real defects, but none that stopped a person completing the journey.
Re-checking everything
Every round after the first opened with a table of issue verdicts. For each issue the previous fix groups claimed to close, the round re-ran its acceptance checks live and recorded PASS or FAIL with evidence. Then it ran the whole journey again, plus spot checks for earlier regressions.
The history flood shows why this matters. Round two found that reloading a 113-message chat sent 55 identical history requests. The fix landed. Round three re-checked it and found 4 requests in 1.5 seconds: much better, still wrong. It went back into a fix group as a new defect. Round four found exactly 2 requests and rendered 106 messages in 1.4 seconds. If round three had only checked "does the page load," the fix would have been marked done at 4.
A fix is not done when its pull request merges. It is done when the next
full round says so. Each fix pull request was already proven live on its
own branch before merge. The round is the second, independent check that it
still holds on main, next to every other change that merged the same day.
The crash loop
Round four found one defect that blocked the journey: a crash during a native load left that chat unable to run anything until a full reset. It was the same shape as D14 in round one and a journal conflict in round three: earlier fixes had closed those repros, and the crash survived in a slightly different sequence.
A single re-run after a fix would have proved almost nothing about a timing defect. So round five turned the repro into a loop and ran it ten times on fresh chats:
The controller's log confirmed that each kill landed right after the native load of that chat started, so every iteration hit the window that mattered. All ten recovered. The crash shapes from rounds two to four were also re-run and recovered. Rounds six and seven kept the same loop as a standing check.
The rules, as we ran them
None of these is clever. Each one exists because its opposite is the easy thing to do at hour thirty.
| Rule | What it means in practice | Why |
|---|---|---|
| Find everything, fix nothing, during a pass | The pass ends with a report, not a patch | A pass measures one commit; changing the code mid-pass voids it |
| One fix cycle per full run | Defects from a run are fixed in a few grouped pull requests, each proven live, then merged; then the next full run | Grouped fixes touch each subsystem once; the next run checks them together |
| Never retry until it passes | A second attempt is a new run, and both are reported | Round two's SIGBUS crash failed 7 of 19 up runs. Retrying until green would have shown a pass and hidden a high-severity bug |
| Never change a scenario while running it | If a step is wrong, finish the run, then fix the scenario for the next one | Otherwise the verdict describes a test nobody wrote down |
| One stack and one browser at a time | A lock prevents two runs sharing ports, state directories or a browser; at most two sub-agents | Two runs on one machine produce defects that belong to neither |
| Cheap models for QA | QA work goes to cheaper models, and the live agent turns ran on Haiku and a low-effort Codex model | The scenario carries the intelligence; volume and repetition matter more |
| Every verdict has evidence | Frames, logs, command output or a screenshot next to each PASS or FAIL | An agent's "it worked" is a claim until something shows it |
| Mark what you could not test | BLOCKED, with the reason and what it would take | Silent skips read as passes |
The retry rule deserves a second look, because it is the one most often broken. A flaky failure is information about the system: something depends on timing, ordering or state that the test does not control. An agent told to make the suite green will retry until it is green, and the bug it was looking at disappears from the record. The rule makes the second attempt a separate observation, so seven failures in nineteen tries is the result, not an inconvenience on the way to a pass.
Knowing when to stop
The defect count did not reach zero. Round six found 3, round seven found 4. A loop like this can run forever, because a deeper test always finds something, and each round costs real time.
After round seven the repo owner stopped the fix loop. The report's last section collects everything still open into a single list, ready to be filed as one issue. Looking back, the stopping condition was three things at once: the full journey had passed end to end for three consecutive rounds, no open defect blocked a person completing it, and the regression spot checks were all passing. Round seven's own checks of the round six fixes give a sense of where things ended up: a 212-message chat reopened in a median of 0.31 seconds over 7 runs, and after restoring 29 archived chats, all 29 were usable 2 seconds after Restore.
How to measure a verification loop
| Metric | What it tells you | What we saw |
|---|---|---|
| Defects per round | Whether the loop is converging | 14, 16, 17, 6, 6, 3, 4 |
| Blocking defects per round | Whether a person can finish the journey | One blocker in round four, none from round five |
| Closed-issue re-check pass rate | Whether fixes hold on main | The history flood failed its first re-check, passed the second |
| Regressions found by spot checks | Whether fixes broke earlier work | Recorded per round in its own table |
| Loop pass rate for timing defects | Whether a race is gone or just rare | 10 of 10 crash loops in round five |
| BLOCKED items | What the loop cannot see | Real Slack and WhatsApp paths, every round |
If defects per round is flat across three or four rounds, the loop is not converging. Look at whether the fix groups are too large, or whether fixes are being merged without being proven live first.
What this does not fix
- Real providers stayed out of reach. Every round marked the real Slack and WhatsApp paths BLOCKED, because no test accounts existed. The remote path was exercised with a simulated native side and a real hosted gateway, which is not the same thing.
- One run per behaviour is a small sample. Most journey items ran once per harness per round. The crash loop is the exception that shows the rule: timing defects need loops, and most items did not get one.
- A loop of ten is not a proof. Ten clean crash loops make a race much less likely. They do not show it is gone.
- One machine, one operating system. All seven rounds ran on one macOS host. Linux service management was not covered.
- The journey is the scope. The rounds check the paths the checklist walks. A defect off that path is invisible to all seven rounds.
- The reports are written by the agent that ran them. That is why every verdict carries its evidence. A reader can check the frames; they should not have to trust the summary.
What it cost
Seven full journeys in about 39 hours, each one run alone on the machine, because two at once would have contaminated each other. Several fix pull requests between rounds, each proven live on its own branch before merging, which is slower than merging on green unit tests. A lot of written reporting: each round's record ran to several hundred lines. We did not record the token cost of the rounds, so we cannot give one. And serialising everything means the loop's speed is set by the slowest step, usually a full deployment and journey, not by how many agents you can start.
What it bought was a product that went from a session stuck in unknown
after a crash to a full journey that passed end to end three rounds running,
in three days, with every claim along the way backed by a frame log.
The checklist
Before a round:
- Deploy a fresh, throwaway environment from the current head of
main, and record the commit. - Write down assumptions: which models, what counts as a defect, what is BLOCKED and why.
- Attach a recorder that captures frames, errors and logs for every verdict.
- Take the lock: one stack, one browser, at most two sub-agents.
During a round:
- Fix nothing. Record every defect with feature, repro steps, expected, actual, and cause if visible.
- Never retry until it passes. A second attempt is a new run; report both.
- Never change the scenario mid-run. Note the problem and finish.
- Mark every item PASS, FAIL, PARTIAL or BLOCKED, with evidence in the same row.
- Turn any timing defect into a loop, and report the loop's pass count.
Between rounds:
- Group defects by subsystem. One pull request per group.
- Prove each fix live on its branch before merging.
- Merge all groups, then start the next full round from
main.
At the start of the next round:
- Re-run the acceptance checks of every issue the fixes claimed to close.
- Re-run the whole journey, not only the fixed parts.
- Spot-check earlier regressions.
To stop:
- The journey passes end to end for several rounds in a row, no open defect blocks it, and spot checks pass.
- Collect everything still open into one list and file it.
Find everything first. The bugs you have not seen yet are the ones that decide what the right fix for the first one was.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.