Hill-Climbing the Reviewer: 1,200 Lines, 200 Lines, 600 Lines
How much rule text should one AI reviewer read per run? We tried three answers in four days. Batching hid real violations, small batches cost five times more and mostly said not applicable, and the setting that stuck came with a second change: moving mechanical rules out of review and into code.
An 858-line app change went through our AI rule review and came back clean.
One reviewer run had checked it against 24 rule sections at once, and every
verdict said followed. So did the next 34 pull requests reviewed the same
way. Not one reported a violation.
That silence was the problem. Before we started batching rule sections into one run, about one pull request in five had a reported violation. After, none did. Either our agents had become much better at following the rules overnight, or the reviewer had stopped seeing them.
In our relay codebase, CatalEx Relay, every pull request is checked by
read-only reviewer subagents against the repository's rule files, a verifier
re-checks each finding, and a merge gate refuses to merge until the review
covers the exact head. This post is about one number in that system's
configuration, rule_batch_lines, and how we hill-climbed it from 1,200 to
200 to 600 between November 1 and November 3. It is a small knob. Getting it
wrong either hides real violations or spends five times the tokens on
reviewers that mostly answer "not applicable".
The knob, and what it trades
When a change is pushed, a planner lists the rule sections that apply to the
changed paths and packs them into reviewer runs. rule_batch_lines is the
most rule text one run may hold. Each run gets the diff plus its batch of
sections, and answers one verdict per section.
The trade is easy to state:
| Bigger batches | Smaller batches |
|---|---|
| Fewer runs, and the diff is read fewer times | More runs, each re-reading the whole diff |
| Each run holds many rules at once | Each run holds a few rules and can attend to them |
| Cheaper per pull request | More expensive per pull request |
| Higher risk a rule is skimmed | More verdicts of not_applicable |
The research on instruction density says which way the risk runs. Across 20 models, "even the best frontier models only achieve 68% accuracy at the max density of 500 instructions", and models favour earlier instructions (Jaroslawicz et al., 2025). A reviewer holding 24 sections is in that regime. What the research cannot tell you is where your own threshold is. You have to measure it.
Step 1: notice the silence
The review began with one run per rule section. A fresh review of a typical pull request took 10 to 17 runs. To cut cost, the configuration moved to batches of 1,200 lines, which in practice put every applicable section into a single run. Here is what the records showed when we compared the periods on November 1:
| One run per section | Batched at 1,200 lines | |
|---|---|---|
| Pull requests | 144 | 35 |
| Runs per review record | 11.1 | 2.0 |
| Verifier runs | 124 | 0 |
| Pull requests with a reported violation | 30 | 0 |
Two runs per record is one general reviewer and one rule reviewer. Zero verifier runs means there was never anything to verify. A reviewer that never finds anything is indistinguishable from a reviewer that never looks. The pull requests differed between the two periods, so this table shows a change, not its cause. It was enough to justify a direct test.
Step 2: re-check, don't argue
We took the five largest batched pull requests, each 650 to 2,000 added lines, and reviewed them again with one run per rule file, split at 200 lines as before batching. Same Sonnet reviewer, same prompts, same diff at the recorded base and head, and an Opus verifier on every finding.
Both misses were confirmed by the verifier, and both were already on main:
| Pull request | Rule | What the batched run marked followed |
|---|---|---|
| A CLI change | CLI diagnostics: failure messages name no file paths | Four failure messages included the user's file path |
| A sessions change | Fail fast on programmer errors | A missing injected environment reader silently fell back to the real one |
Neither is a style quibble. The first leaks a local path into output a person reads. The second hides a wiring bug behind a working default, which is exactly what the fail-fast rule exists to prevent. The other three pull requests had no violation under either setting.
Step 3: lower it, and price it
The owner approved lowering rule_batch_lines to 200 the same day, as a
configuration-only pull request. The pull request priced it up front: a large
change goes from about 2 runs per review to about 10. The most runs any
record had used was 24, still under the per-review budget of 40.
This is the right first move after a miss. You go to the setting that is known to find the violations, and then measure what it costs, rather than guessing at a midpoint before you have data.
Step 4: measure 72 hours of it
By November 3 we had 153 pull requests reviewed at 200 lines, and enough records to compare the two kinds of reviewer side by side:
| Reviewer | Model | Runs | Confirmed findings | Refuted |
|---|---|---|---|---|
| Rule reviewer | Sonnet | 1,866 | about 56, mostly test style and process | about 57 |
| General reviewer | Opus | 269 | 23, mostly real bugs | 24 |
Three things stand out.
- Volume without yield. The rule reviewer made seven times as many runs as the general reviewer for about twice the confirmed findings, and those findings were mostly about test style and process. About ten runs went to every Go change and every pushed fix.
- 51% of rule verdicts were
not_applicable. Half of every reviewer's attention went to sections that had nothing to say about the change. That is a packing problem as much as a size problem: large rule files mixed Go, test, database and frontend rules, so a change in one area dragged in sections written for another. - Mechanical sections did worst. "No magic numbers" produced 1 confirmed finding and 6 refuted. Product naming produced 2 and 7.
That last point has a well-known standard behind it. Google's static analysis platform requires new checks to "produce less than 10% effective false positives", where an effective false positive is an issue developers took no action on (Software Engineering at Google, ch. 20). A section where six of seven findings are refuted is nowhere near that bar. An LLM is the wrong tool for a rule a regular expression can decide, and it is also the most expensive one.
Step 5: settle at 600, and move the mechanical rules out
The setting that stuck was 600 lines, approved on November 3: the midpoint between the 1,200 that missed and the 200 that found. On the rule files as they stood, a typical Go change went from 7 rule reviewers to 3.
The number alone would have been a compromise. What made it work was the second half of the change, landed the same day: the mechanical rules left the LLM review entirely and became deterministic checks that run locally and in CI on what a change adds.
| Check | Tool | Flags in added code |
|---|---|---|
table-test | Semgrep | A Go test without t.Run inside a range loop |
magic-number | Semgrep | An integer literal used as a timeout, limit, retry count or similar, outside a const |
enum-case | Semgrep | A typed code constant not in UPPER_SNAKE_CASE, or a state constant not lowercase |
product-name | Semgrep | Any spelling of the product name other than the two allowed ones, in user-facing strings |
guide-size | pr_checks.py | An agent guide over 400 lines that grows |
ui-evidence | pr_checks.py | UI files changed and fewer than two screenshot links in the pull request |
Each rule enforced by a check then left the rule files, keeping one line that
names the check. Large rule files were split by path scope (Go, tests,
database, frontend), so fewer sections reach a reviewer only to be marked
not_applicable. Both changes shrink the prompt every reviewer reads, which
makes the 600-line batches hold more of what matters.
A prompt fix found along the way
By November 4, reviewers were returning JSON with the wrong keys: things
like rule_item and reason where the schema required item and why. The
verdicts were unusable, because the cycle validates every answer against the
schema before it records anything.
The cause was in the prompt. It described a violation in words, "the rule item" and "its reason", which primed exactly those key names. The schema that named the real keys came at the end, after hundreds of kilobytes of diff. The fix did two things at once: it stated the schema and its keys before the scope and the diff, and it changed the prose to use the schema's own names. A test now reads the required keys from the schema file and fails if any prompt omits one before the diff or uses the wrong names.
We changed two variables at once, and that matters. Anthropic's own long-context guidance says the opposite of half our fix: put long documents at the top and the query at the end, which "can improve response quality by up to 30 percent" (prompting best practices). Studies of long contexts find models use information at the start and end best, and the middle worst (Liu et al., Lost in the Middle). Our prompt had the schema at the end, so position alone may not have been the fault. The mismatched names probably were. We don't know which change did the work, and the honest version of this lesson is: make the prose and the schema use the same words, and pin the structure with a test.
The prompt limit kept climbing
The reviewer prompt has a size bound, prompt_limit_bytes, so a runaway diff
can't produce a run nobody can afford. Every raise is a configuration change
in its own reviewed pull request, inside a validated range. Here is how often
that happened:
| Date | Limit |
|---|---|
| Start | 400 KB |
| October 29 | 600 KB, then 700 KB |
| November 1 | 900 KB |
| November 2 | 1 MB, 1.15 MB, 2.2 MB, 2.9 MB |
Seven raises in five days. Each one was locally reasonable: a verifier on one branch needed a 484,162-byte instruction, so the first raise went to 600 KB rather than split that pull request. But a bound you raise every time you hit it is not a bound. And a bigger prompt is not a better review, for the same reasons a bigger batch isn't.
So the November 3 change added a rule to the review skill: split first. When a review exceeds the limit, split the review range or the change. Raise the limit only when no split can fit, and then by the smallest amount, in a separate reviewed configuration pull request.
Where the tokens actually went
It is easy to tune a reviewer and miss that it is a small part of the bill. Session logs from one laptop between October 31 and November 3 came to about $975 of tokens at list prices, and 61% of that was cache reads of contexts between 250K and 950K tokens. Long-running orchestrator sessions polled CI and tailed subagent output; implementers read whole files and screenshots. The reviewers mattered, and so did everything around them. The same plan that set 600 lines also added token-economy guidance aimed at those sessions, not only at the reviewers.
Failure modes of a tuned reviewer
| Failure mode | Symptom | The check |
|---|---|---|
| Overloaded reviewer | Zero findings across many pull requests | Re-review a sample at a smaller batch |
| Underloaded reviewer | High run count, most verdicts not_applicable | Track the not_applicable share; split rule files by path |
| Wrong tool for the rule | Refuted findings dominate a section | Per-section confirmed to refuted ratio; move to a linter |
| Prompt and schema disagree | Invalid keys, unusable verdicts | Test that the prompt names schema keys before the diff |
| Limit creep | Repeated raises of a size bound | Split-first rule; raises only as separate reviewed changes |
| Cost hidden elsewhere | Reviewer tuned, bill unchanged | Measure whole-session token use, not only reviewer runs |
How to measure your own
Keep a review record per head, and compute these per week:
- Runs per pull request, split by reviewer kind.
- Confirmed findings per 100 runs, per reviewer kind and per rule section.
- Refuted share per section. Anything far over 10% is a candidate for a deterministic check, or for deletion.
not_applicableshare. Above half means your packing is wrong before your batch size is.- Re-check miss rate. Every so often, re-review a few of the largest
pull requests at a smaller batch and count confirmed violations the
normal run marked
followed. This is the only number that tells you what the reviewer is not seeing.
What this does not fix
- The sample is small. The decision to lower the batch rests on five re-checked pull requests and two misses. The working tree the re-check read was newer than the reviewed heads, though both findings were checked against the reviewed diff.
- 600 is a midpoint, not an optimum. It was chosen between a value known to miss and a value known to cost too much. Larger runs may still miss some violations 200-line runs would find. The plan is to re-measure findings per pull request after the mechanical rules moved out.
- Tokens weren't recorded per run. Reviewer records logged zero tokens, so the cost side of this comparison comes from run counts and a one-laptop session log, not per-review billing.
- Same model family. None of this tuning makes a Sonnet reviewer independent of an Opus or Sonnet author. It makes it more efficient.
What it costs
Every turn of this loop spent money to learn something. The re-check alone
was 58 runs at up to 102,000 tokens each. The 200-line period ran 1,866 rule
reviewer runs in 72 hours. The deterministic checks cost engineering time to
write and maintain, and each one needs an escape hatch (a nosemgrep comment
with a reason) for values an external contract fixes. And the split-first
rule makes some changes slower to land. All of that is cheaper than a
reviewer that approves everything, but none of it is free.
The playbook
| When you see | Do this |
|---|---|
| A reviewer that has stopped finding anything | Re-review the largest recent pull requests at a smaller batch before trusting the silence |
| A re-check that finds confirmed misses | Drop to the batch size that found them, and price it in the same change |
Most verdicts are not_applicable | Split rule files by path scope before touching the batch size |
| One section is mostly refuted | Turn it into a linter or a structural check, then delete it from the rules |
| A general reviewer finds real bugs and a rule reviewer finds style | Spend on the general reviewer; shrink what the rule reviewer reads |
| Invalid or misnamed JSON answers | Use the schema's own words in the prompt and pin the order with a test |
| A size limit raised more than once a week | Split first; raise only in a separate reviewed change |
| A reviewer bill that won't move | Measure the whole session; the orchestrator may be the cost |
Tune the reviewer like any other model in production: with a sample you re-check, a number you trust, and the humility to hand the easy rules to code.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.