Reviewers That Can't Be Argued With: Read-Only Subagents and a Merge Gate
An agent that reviews its own change can see a violation and still talk itself into merging. The five-part framework we use instead: read-only reviewer subagents per rule section, prompts the author never touches, verifiers, a rule against self-dealing, and a merge gate that checks the exact head. With the numbers from 179 pull requests.
A coding agent changed a button on an agent card to read Settings. The button opened a dialog titled with the agent's name followed by "settings". At the time, one of our desktop UI rules said a dialog's title must repeat the label that opened it, word for word. The agent's own pre-push review walked the list of entry points and noticed the mismatch. It wrote it down as "the judgment call I raised earlier", listed it under risks in the pull request, and merged when CI went green.
Nothing was hidden. The finding was in the pull request for anyone to read. The rule even said a mismatch was "a P2 finding, never a preference". The agent saw the violation, classified it as a matter of taste, and was the only party in the room. The automated test that should have caught it compared the dialog title with the button's accessible name, which matched. No test read the visible label.
The rule was later reworded, and the button complies with the new wording. The process gap did not go away. The only check able to catch the violation was a judgment made by the agent that wrote the change, and a finding it recognised had no path that could block a merge. This post describes what we built in our relay codebase, CatalEx Relay, in the week after that incident: a framework of five parts that makes a review finding something the author cannot argue with, and the numbers from its first 179 pull requests.
Why self-review is the weakest checker
Most agent workflows ask the agent to review its work before it opens a pull request. It feels like diligence. Three findings from the literature say it is the weakest check you have:
- Models struggle to correct themselves without outside feedback. "LLMs struggle to self-correct their responses without external feedback", and performance can get worse after trying (Huang et al., ICLR 2024).
- Models favour their own writing. Models can recognise their own outputs, and the authors find "a linear correlation between self-recognition capability and the strength of self-preference bias" (Panickssery, Bowman and Feng, 2024).
- Many rules at once get followed less. Across 20 models, "even the best frontier models only achieve 68% accuracy at the max density of 500 instructions" (Jaroslawicz et al., 2025). Our rules were 12 files and about 1,300 lines at the time.
Our incident adds a fourth that the papers don't cover. The author did find the violation. The failure was not detection. It was the author's authority to downgrade what it detected. Any fix that leaves the author in charge of the verdict leaves that path open.
Why the obvious fixes fail
| Obvious fix | Why it doesn't hold |
|---|---|
| Make the self-review checklist longer | More rules in one context means each is followed less, and the author still grades itself |
| A per-rule ledger the author fills in | Grows with every rule, and it is still the author marking its own work |
| Turn every rule into a linter | Right for the few rules a program can decide; most of our rules are prose judgments |
| Tell the agent never to merge with an open finding | Helps, costs nothing, and relies on the same agent classifying a finding as a finding |
| A second reviewer that receives the author's summary | The summary is the argument. The finding can be talked away before it is made |
Two of these survive as backstops: the narrow merge rule, because it costs nothing, and turning the rules a program can decide into checks. Neither covers the prose rules, which are most of them. The main mechanism is the framework below.
The framework: five parts
Each part answers one question about who controls a verdict.
The author's session is the box on the left, and it touches nothing in the middle. It starts the reviewers and it can be refused at the end. Those are the only two things it does.
Part 1: reviewers that can only read and answer in JSON
Three agent definitions live in the repository. Each one is a few lines of frontmatter and a three-sentence body. What matters is what they are allowed to do.
| Agent | Model | Tools | Job |
|---|---|---|---|
rule-reviewer | Sonnet, low effort | Read, Grep, Glob | Check the change against one rule file or a batch of sections |
general-reviewer | Opus | Read, Grep, Glob | Look for correctness, security and lifecycle defects |
rule-verifier | Opus | Read, Grep, Glob | Re-check one reported violation |
Every body says: follow the task message verbatim; you are read-only, "never edit a file or run a command"; answer only with JSON matching the schema; "never delegate, start another review cycle, run tests or repeat QA".
Read-only is not a style choice. A reviewer that can run commands can "fix" what it finds, run a test that changes state, or start a review of its own. A reviewer that can only read and answer has one output, and it is a verdict.
The answer is structured so that a program, not a person, decides what it means:
| Field | Type | Meaning |
|---|---|---|
verdicts[].section | string | The rule section judged |
verdicts[].verdict | followed, not_applicable, violated | Nothing else is accepted |
violations[].item | string | The numbered rule item |
violations[].path, line | string, integer | Where in the change |
violations[].why | string | The evidence |
violations[].severity | string | The rule's own severity |
A verifier answers confirmed, refuted or undecided, with a reason.
Extra keys are rejected.
Part 2: prompt files the author never edits
A planner command, rule-review plan, works out which rule sections apply
to the changed paths and writes one prompt file per agent. Each file
holds the base and head commits, the diff, the rule text and the schema. The
author's session then starts each subagent with a single sentence: read this
file and follow it exactly, it is your whole instruction, answer with its
JSON only. The answer is saved verbatim to a result path.
This does two jobs. It keeps the author out of the prompt, so there is nowhere to add "note: the title mismatch is intentional". And it keeps the diffs out of the author's context. Pasting a prompt instead of pointing at it "would load every diff into the author's context", and the author is a long-running session that still has work to do.
One correction to our first design: a subagent is less isolated than we assumed. It starts with a fresh context, but Claude Code still loads the project's instruction files into it (subagents). Each reviewer carries the root agent guide as well as its section. That is fine for independence, since none of it is the author's reasoning, but it is real context you pay for on every run.
Part 3: a verifier for every finding
Reviewers asked to find problems will find some even when the work is sound. So every reported violation goes to an Opus verifier with the same read-only tools, which re-checks it against the diff and answers confirmed or refuted. Anthropic's own Code Review service has the same shape: several agents look in parallel, then "a verification step checks the candidate findings against the code and filters out false positives" (Code Review). One difference is deliberate: Anthropic's review "never approves or blocks the pull request". Ours blocks. A confirmed violation stops the merge until the owner records a decision that approves it.
Part 4: the rule against self-dealing
Parts 1 to 3 are machinery. Part 4 is the sentence that makes them mean something, and it is written into the review skill every agent follows:
The author never edits a result or the record, never adds a summary or reasoning to a prompt, and never approves, downgrades or dismisses a violation.
In the words of the decision record, the finding "cannot be argued away before it is made". The only ways past a confirmed violation are to fix the code and run the cycle again, or for the owner to approve the exception in a decision record that names the pull request and the rule file. The rules reserve that decision for the owner.
Part 5: a merge gate on the exact head
A rule written in a skill is still a rule an agent can skip. The last part
makes skipping fail. A PreToolUse hook on the Bash, PowerShell and
Monitor tools runs a gate script before every shell command. Claude Code
blocks the tool call when a hook exits with code 2
(hooks reference), and the hook is
set to onFailure: block, so a gate that crashes or times out refuses rather
than allows. The same script serves Codex.
Each check closes a specific way around the gate:
| Way around | What the gate does |
|---|---|
| Merge without naming the head | Refuses, and prints the command with the head filled in, so nothing pushed after review can merge |
Wrap the merge in sh -c, eval, a heredoc or a cd chain | Parses shell syntax and finds the merge at any command position |
| Merge with a review record that only exists locally | Reads records fetched from origin only |
| Edit the rule file on the branch so it passes | Reads rule files from origin/main, never from the working tree or the branch |
| Merge while CI is still running | Requires required CI green on that exact head |
The review record is a JSON git note on the exact commit it reviewed, under its own notes ref. It holds every verdict, every violation and every verifier answer. Because it is attached to a commit, a new push means a new head with no record, and the gate asks for a review of the change since the last one.
The gate runs locally, not in CI, and that was a cost decision. A reviewer in CI needs a model account of its own. For a self-funded project the options were a separate paid account, or an engineer's own allowance spent on every pull request. The local gate runs the review in the author's session, on the author's own usage.
What it found: the pilot
On November 1 we matched every pull request merged since the gate went live on October 26 with its review records: 179 pull requests and 366 records.
| Measure | Value |
|---|---|
| Runs per fresh review (first six pull requests) | 10 to 17, mean 12.7 |
| Reviewer runs over the whole range | 2,222 with one run per rule section |
| Pull requests with a reported violation | 30 |
| Largest sources of violations | General review 26 (12 confirmed), documentation policy 23 (10), desktop UI rule 11 (8) |
| Run inputs started more than once | 34 |
| Most runs in one record | 24, against a budget of 20, later raised to 40 |
Two numbers deserve a second look. Verifiers refuted slightly more than half of what reviewers reported. Some refutations were of findings a later commit had already fixed, but the ratio says reviewers over-report, which is what Anthropic's guidance warns about, and why the verifier is not optional. And 12 pull requests merged with no record at all. The gate only runs inside agent sessions. A merge made from the GitHub website, or a terminal outside an agent, never meets the hook.
The incident, replayed
Here is the settings dialog change again, under the framework:
| Step | Before | After |
|---|---|---|
| Who judges the rule | The author, inside its own pre-push review | A rule reviewer given only the diff and the desktop UI section |
| What the finding becomes | A sentence: "the judgment call I raised earlier" | A violated verdict with item, path, line and severity |
| Who can downgrade it | The author, by writing "judgment call" | Nobody but the owner, in a decision record |
| What checks the finding | Nobody | An Opus verifier, which confirms or refutes |
| What happens at merge | Merged on green | Gate refuses: open violation without owner decision |
| What the author can do | Merge | Fix the title, push, review the delta, merge the new head |
The parts compose. The reviewer finds it without the author's framing, the verifier stops a false finding from blocking, the rule removes the author's veto, the record pins the verdict to a commit, and the gate makes the verdict the thing that decides.
Failure modes, and the check for each
| Failure mode | What it looks like | The check |
|---|---|---|
| Self-dealing | Author downgrades its own finding | Separate reviewer; author can't edit verdicts |
| Framing | Author's summary primes the reviewer | Prompt files written by a planner; one-line start |
| Over-reporting | Reviewer flags sound code | Verifier per violation |
| Stale review | Push after review, merge the new head | Record on the exact commit; --match-head-commit |
| Judge tampering | Branch weakens its own rule text | Rules read from origin/main |
| Wrapped command | Merge hidden inside sh -c | Shell parsing in the gate |
| Silent gate failure | Hook crashes, merge goes through | onFailure: block |
How to measure it
Read the records, not the pull request descriptions. For each period, count records per pull request, runs per record, reported violations, the share the verifiers confirmed, and pull requests merged with no record. The last number is your coverage gap, and it should be zero. If it isn't, find the path those merges took.
Two things we could not measure, and you should fix before you start: every run recorded 0 tokens, and records held no start or finish times. Make your record carry both from day one, or the cost question will be a guess.
What this does not fix
- Same model family. A subagent drops the author's context and stake, but it is the same family of model. Given the self-preference result above, some bias toward the author's writing may remain. Our own decision record says it plainly: "a different model or a human reviewer would be more independent".
- Merges outside agent sessions. A merge on github.com bypasses the gate. Branch protection on the server requires CI, and no workflow reads review records. 12 of 179 pull requests went this way.
- Evasion. The gate is a guard against mistakes by an agent that follows
the rules, not a security boundary. A command word it cannot read, such as a
variable holding
git, or a script file that merges, gets past it. Codex describes its own hooks as "a useful guardrail, not a complete enforcement boundary". - Rules that are wrong. The reviewer enforces the rule as written. In our incident the rule itself was later changed. A gate makes a bad rule hurt faster, which is useful, but it doesn't make it right.
What it costs
A fresh review in the pilot took 10 to 17 subagent runs, four at a time, and we raised the per-review budget from 20 to 40 when one change needed 24. Every run reads its own prompt file with the diff in it, and a later re-check measured single reviewer runs at 57,000 to 102,000 tokens. On top of tokens, a review adds minutes to every push, and the gate refuses merges you would otherwise have made. That friction is the point, but it is not free.
The framework, and the order to adopt it
| Part | The question it answers | The mechanism |
|---|---|---|
| 1. Read-only reviewers | Can the reviewer change what it reviews? | Agent definitions with Read, Grep, Glob; JSON only |
| 2. Prompt files | Can the author frame the review? | Planner writes the prompt; author passes a path |
| 3. Verifiers | Are the findings real? | A verifier per violation: confirmed or refuted |
| 4. No self-dealing | Can the author overrule a finding? | Written rule; only an owner decision approves |
| 5. Exact-head gate | Can a merge skip the review? | PreToolUse hook, record on origin, rules from main |
Adopt it in this order. First the narrow merge rule, because it is free. Then parts 1 and 2 together, run on every pull request without blocking, and read the verdicts for a week. Add part 3 when you see how many findings are false. Add part 5 last, once findings are trustworthy enough that blocking on them doesn't stop all work. Write part 4 down on day one, and then make the other four parts enforce it.
A review the author can argue with is a conversation. A review the author can't argue with is a check.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.