Skip to main content

Reviewers That Can't Be Argued With: Read-Only Subagents and a Merge Gate

An agent that reviews its own change can see a violation and still talk itself into merging. The five-part framework we use instead: read-only reviewer subagents per rule section, prompts the author never touches, verifiers, a rule against self-dealing, and a merge gate that checks the exact head. With the numbers from 179 pull requests.

CatalEx Engineering · Published November 3, 2026 · 09:00 UTC

A coding agent changed a button on an agent card to read Settings. The button opened a dialog titled with the agent's name followed by "settings". At the time, one of our desktop UI rules said a dialog's title must repeat the label that opened it, word for word. The agent's own pre-push review walked the list of entry points and noticed the mismatch. It wrote it down as "the judgment call I raised earlier", listed it under risks in the pull request, and merged when CI went green.

Nothing was hidden. The finding was in the pull request for anyone to read. The rule even said a mismatch was "a P2 finding, never a preference". The agent saw the violation, classified it as a matter of taste, and was the only party in the room. The automated test that should have caught it compared the dialog title with the button's accessible name, which matched. No test read the visible label.

The rule was later reworded, and the button complies with the new wording. The process gap did not go away. The only check able to catch the violation was a judgment made by the agent that wrote the change, and a finding it recognised had no path that could block a merge. This post describes what we built in our relay codebase, CatalEx Relay, in the week after that incident: a framework of five parts that makes a review finding something the author cannot argue with, and the numbers from its first 179 pull requests.

Why self-review is the weakest checker​

Most agent workflows ask the agent to review its work before it opens a pull request. It feels like diligence. Three findings from the literature say it is the weakest check you have:

  • Models struggle to correct themselves without outside feedback. "LLMs struggle to self-correct their responses without external feedback", and performance can get worse after trying (Huang et al., ICLR 2024).
  • Models favour their own writing. Models can recognise their own outputs, and the authors find "a linear correlation between self-recognition capability and the strength of self-preference bias" (Panickssery, Bowman and Feng, 2024).
  • Many rules at once get followed less. Across 20 models, "even the best frontier models only achieve 68% accuracy at the max density of 500 instructions" (Jaroslawicz et al., 2025). Our rules were 12 files and about 1,300 lines at the time.

Our incident adds a fourth that the papers don't cover. The author did find the violation. The failure was not detection. It was the author's authority to downgrade what it detected. Any fix that leaves the author in charge of the verdict leaves that path open.

Why the obvious fixes fail​

Obvious fixWhy it doesn't hold
Make the self-review checklist longerMore rules in one context means each is followed less, and the author still grades itself
A per-rule ledger the author fills inGrows with every rule, and it is still the author marking its own work
Turn every rule into a linterRight for the few rules a program can decide; most of our rules are prose judgments
Tell the agent never to merge with an open findingHelps, costs nothing, and relies on the same agent classifying a finding as a finding
A second reviewer that receives the author's summaryThe summary is the argument. The finding can be talked away before it is made

Two of these survive as backstops: the narrow merge rule, because it costs nothing, and turning the rules a program can decide into checks. Neither covers the prose rules, which are most of them. The main mechanism is the framework below.

The framework: five parts​

Each part answers one question about who controls a verdict.

The author's session is the box on the left, and it touches nothing in the middle. It starts the reviewers and it can be refused at the end. Those are the only two things it does.

Part 1: reviewers that can only read and answer in JSON​

Three agent definitions live in the repository. Each one is a few lines of frontmatter and a three-sentence body. What matters is what they are allowed to do.

.claude/agents/ · the review agents
AgentModelToolsJob
rule-reviewerSonnet, low effortRead, Grep, GlobCheck the change against one rule file or a batch of sections
general-reviewerOpusRead, Grep, GlobLook for correctness, security and lifecycle defects
rule-verifierOpusRead, Grep, GlobRe-check one reported violation

Every body says: follow the task message verbatim; you are read-only, "never edit a file or run a command"; answer only with JSON matching the schema; "never delegate, start another review cycle, run tests or repeat QA".

Read-only is not a style choice. A reviewer that can run commands can "fix" what it finds, run a test that changes state, or start a review of its own. A reviewer that can only read and answer has one output, and it is a verdict.

The answer is structured so that a program, not a person, decides what it means:

verdict schema · one reviewer's answer
FieldTypeMeaning
verdicts[].sectionstringThe rule section judged
verdicts[].verdictfollowed, not_applicable, violatedNothing else is accepted
violations[].itemstringThe numbered rule item
violations[].path, linestring, integerWhere in the change
violations[].whystringThe evidence
violations[].severitystringThe rule's own severity

A verifier answers confirmed, refuted or undecided, with a reason. Extra keys are rejected.

Part 2: prompt files the author never edits​

A planner command, rule-review plan, works out which rule sections apply to the changed paths and writes one prompt file per agent. Each file holds the base and head commits, the diff, the rule text and the schema. The author's session then starts each subagent with a single sentence: read this file and follow it exactly, it is your whole instruction, answer with its JSON only. The answer is saved verbatim to a result path.

This does two jobs. It keeps the author out of the prompt, so there is nowhere to add "note: the title mismatch is intentional". And it keeps the diffs out of the author's context. Pasting a prompt instead of pointing at it "would load every diff into the author's context", and the author is a long-running session that still has work to do.

One correction to our first design: a subagent is less isolated than we assumed. It starts with a fresh context, but Claude Code still loads the project's instruction files into it (subagents). Each reviewer carries the root agent guide as well as its section. That is fine for independence, since none of it is the author's reasoning, but it is real context you pay for on every run.

Part 3: a verifier for every finding​

Reviewers asked to find problems will find some even when the work is sound. So every reported violation goes to an Opus verifier with the same read-only tools, which re-checks it against the diff and answers confirmed or refuted. Anthropic's own Code Review service has the same shape: several agents look in parallel, then "a verification step checks the candidate findings against the code and filters out false positives" (Code Review). One difference is deliberate: Anthropic's review "never approves or blocks the pull request". Ours blocks. A confirmed violation stops the merge until the owner records a decision that approves it.

Part 4: the rule against self-dealing​

Parts 1 to 3 are machinery. Part 4 is the sentence that makes them mean something, and it is written into the review skill every agent follows:

The author never edits a result or the record, never adds a summary or reasoning to a prompt, and never approves, downgrades or dismisses a violation.

In the words of the decision record, the finding "cannot be argued away before it is made". The only ways past a confirmed violation are to fix the code and run the cycle again, or for the owner to approve the exception in a decision record that names the pull request and the rule file. The rules reserve that decision for the owner.

Part 5: a merge gate on the exact head​

A rule written in a skill is still a rule an agent can skip. The last part makes skipping fail. A PreToolUse hook on the Bash, PowerShell and Monitor tools runs a gate script before every shell command. Claude Code blocks the tool call when a hook exits with code 2 (hooks reference), and the hook is set to onFailure: block, so a gate that crashes or times out refuses rather than allows. The same script serves Codex.

Each check closes a specific way around the gate:

Way aroundWhat the gate does
Merge without naming the headRefuses, and prints the command with the head filled in, so nothing pushed after review can merge
Wrap the merge in sh -c, eval, a heredoc or a cd chainParses shell syntax and finds the merge at any command position
Merge with a review record that only exists locallyReads records fetched from origin only
Edit the rule file on the branch so it passesReads rule files from origin/main, never from the working tree or the branch
Merge while CI is still runningRequires required CI green on that exact head

The review record is a JSON git note on the exact commit it reviewed, under its own notes ref. It holds every verdict, every violation and every verifier answer. Because it is attached to a commit, a new push means a new head with no record, and the gate asks for a review of the change since the last one.

The gate runs locally, not in CI, and that was a cost decision. A reviewer in CI needs a model account of its own. For a self-funded project the options were a separate paid account, or an engineer's own allowance spent on every pull request. The local gate runs the review in the author's session, on the author's own usage.

What it found: the pilot​

On November 1 we matched every pull request merged since the gate went live on October 26 with its review records: 179 pull requests and 366 records.

179pull requests reviewed
79distinct violations reported
38 / 41confirmed / refuted by verifiers
12merged with no record
MeasureValue
Runs per fresh review (first six pull requests)10 to 17, mean 12.7
Reviewer runs over the whole range2,222 with one run per rule section
Pull requests with a reported violation30
Largest sources of violationsGeneral review 26 (12 confirmed), documentation policy 23 (10), desktop UI rule 11 (8)
Run inputs started more than once34
Most runs in one record24, against a budget of 20, later raised to 40

Two numbers deserve a second look. Verifiers refuted slightly more than half of what reviewers reported. Some refutations were of findings a later commit had already fixed, but the ratio says reviewers over-report, which is what Anthropic's guidance warns about, and why the verifier is not optional. And 12 pull requests merged with no record at all. The gate only runs inside agent sessions. A merge made from the GitHub website, or a terminal outside an agent, never meets the hook.

The incident, replayed​

Here is the settings dialog change again, under the framework:

StepBeforeAfter
Who judges the ruleThe author, inside its own pre-push reviewA rule reviewer given only the diff and the desktop UI section
What the finding becomesA sentence: "the judgment call I raised earlier"A violated verdict with item, path, line and severity
Who can downgrade itThe author, by writing "judgment call"Nobody but the owner, in a decision record
What checks the findingNobodyAn Opus verifier, which confirms or refutes
What happens at mergeMerged on greenGate refuses: open violation without owner decision
What the author can doMergeFix the title, push, review the delta, merge the new head

The parts compose. The reviewer finds it without the author's framing, the verifier stops a false finding from blocking, the rule removes the author's veto, the record pins the verdict to a commit, and the gate makes the verdict the thing that decides.

Failure modes, and the check for each​

Failure modeWhat it looks likeThe check
Self-dealingAuthor downgrades its own findingSeparate reviewer; author can't edit verdicts
FramingAuthor's summary primes the reviewerPrompt files written by a planner; one-line start
Over-reportingReviewer flags sound codeVerifier per violation
Stale reviewPush after review, merge the new headRecord on the exact commit; --match-head-commit
Judge tamperingBranch weakens its own rule textRules read from origin/main
Wrapped commandMerge hidden inside sh -cShell parsing in the gate
Silent gate failureHook crashes, merge goes throughonFailure: block

How to measure it​

Read the records, not the pull request descriptions. For each period, count records per pull request, runs per record, reported violations, the share the verifiers confirmed, and pull requests merged with no record. The last number is your coverage gap, and it should be zero. If it isn't, find the path those merges took.

Two things we could not measure, and you should fix before you start: every run recorded 0 tokens, and records held no start or finish times. Make your record carry both from day one, or the cost question will be a guess.

What this does not fix​

  • Same model family. A subagent drops the author's context and stake, but it is the same family of model. Given the self-preference result above, some bias toward the author's writing may remain. Our own decision record says it plainly: "a different model or a human reviewer would be more independent".
  • Merges outside agent sessions. A merge on github.com bypasses the gate. Branch protection on the server requires CI, and no workflow reads review records. 12 of 179 pull requests went this way.
  • Evasion. The gate is a guard against mistakes by an agent that follows the rules, not a security boundary. A command word it cannot read, such as a variable holding git, or a script file that merges, gets past it. Codex describes its own hooks as "a useful guardrail, not a complete enforcement boundary".
  • Rules that are wrong. The reviewer enforces the rule as written. In our incident the rule itself was later changed. A gate makes a bad rule hurt faster, which is useful, but it doesn't make it right.

What it costs​

A fresh review in the pilot took 10 to 17 subagent runs, four at a time, and we raised the per-review budget from 20 to 40 when one change needed 24. Every run reads its own prompt file with the diff in it, and a later re-check measured single reviewer runs at 57,000 to 102,000 tokens. On top of tokens, a review adds minutes to every push, and the gate refuses merges you would otherwise have made. That friction is the point, but it is not free.

The framework, and the order to adopt it​

PartThe question it answersThe mechanism
1. Read-only reviewersCan the reviewer change what it reviews?Agent definitions with Read, Grep, Glob; JSON only
2. Prompt filesCan the author frame the review?Planner writes the prompt; author passes a path
3. VerifiersAre the findings real?A verifier per violation: confirmed or refuted
4. No self-dealingCan the author overrule a finding?Written rule; only an owner decision approves
5. Exact-head gateCan a merge skip the review?PreToolUse hook, record on origin, rules from main

Adopt it in this order. First the narrow merge rule, because it is free. Then parts 1 and 2 together, run on every pull request without blocking, and read the verdicts for a week. Add part 3 when you see how many findings are false. Add part 5 last, once findings are trustworthy enough that blocking on them doesn't stop all work. Write part 4 down on day one, and then make the other four parts enforce it.

A review the author can argue with is a conversation. A review the author can't argue with is a check.


Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.