Skip to main content

Hill-Climbing the Reviewer: 1,200 Lines, 200 Lines, 600 Lines

How much rule text should one AI reviewer read per run? We tried three answers in four days. Batching hid real violations, small batches cost five times more and mostly said not applicable, and the setting that stuck came with a second change: moving mechanical rules out of review and into code.

CatalEx Engineering · Published November 6, 2026 · 09:00 UTC

An 858-line app change went through our AI rule review and came back clean. One reviewer run had checked it against 24 rule sections at once, and every verdict said followed. So did the next 34 pull requests reviewed the same way. Not one reported a violation.

That silence was the problem. Before we started batching rule sections into one run, about one pull request in five had a reported violation. After, none did. Either our agents had become much better at following the rules overnight, or the reviewer had stopped seeing them.

In our relay codebase, CatalEx Relay, every pull request is checked by read-only reviewer subagents against the repository's rule files, a verifier re-checks each finding, and a merge gate refuses to merge until the review covers the exact head. This post is about one number in that system's configuration, rule_batch_lines, and how we hill-climbed it from 1,200 to 200 to 600 between November 1 and November 3. It is a small knob. Getting it wrong either hides real violations or spends five times the tokens on reviewers that mostly answer "not applicable".

The knob, and what it trades​

When a change is pushed, a planner lists the rule sections that apply to the changed paths and packs them into reviewer runs. rule_batch_lines is the most rule text one run may hold. Each run gets the diff plus its batch of sections, and answers one verdict per section.

The trade is easy to state:

Bigger batchesSmaller batches
Fewer runs, and the diff is read fewer timesMore runs, each re-reading the whole diff
Each run holds many rules at onceEach run holds a few rules and can attend to them
Cheaper per pull requestMore expensive per pull request
Higher risk a rule is skimmedMore verdicts of not_applicable

The research on instruction density says which way the risk runs. Across 20 models, "even the best frontier models only achieve 68% accuracy at the max density of 500 instructions", and models favour earlier instructions (Jaroslawicz et al., 2025). A reviewer holding 24 sections is in that regime. What the research cannot tell you is where your own threshold is. You have to measure it.

Step 1: notice the silence​

The review began with one run per rule section. A fresh review of a typical pull request took 10 to 17 runs. To cut cost, the configuration moved to batches of 1,200 lines, which in practice put every applicable section into a single run. Here is what the records showed when we compared the periods on November 1:

One run per sectionBatched at 1,200 lines
Pull requests14435
Runs per review record11.12.0
Verifier runs1240
Pull requests with a reported violation300

Two runs per record is one general reviewer and one rule reviewer. Zero verifier runs means there was never anything to verify. A reviewer that never finds anything is indistinguishable from a reviewer that never looks. The pull requests differed between the two periods, so this table shows a change, not its cause. It was enough to justify a direct test.

Step 2: re-check, don't argue​

We took the five largest batched pull requests, each 650 to 2,000 added lines, and reviewed them again with one run per rule file, split at 200 lines as before batching. Same Sonnet reviewer, same prompts, same diff at the recorded base and head, and an Opus verifier on every finding.

2 of 5pull requests had a missed violation
58re-check runs, against 5 batched
57k to 102ktokens per re-check run

Both misses were confirmed by the verifier, and both were already on main:

Pull requestRuleWhat the batched run marked followed
A CLI changeCLI diagnostics: failure messages name no file pathsFour failure messages included the user's file path
A sessions changeFail fast on programmer errorsA missing injected environment reader silently fell back to the real one

Neither is a style quibble. The first leaks a local path into output a person reads. The second hides a wiring bug behind a working default, which is exactly what the fail-fast rule exists to prevent. The other three pull requests had no violation under either setting.

Step 3: lower it, and price it​

The owner approved lowering rule_batch_lines to 200 the same day, as a configuration-only pull request. The pull request priced it up front: a large change goes from about 2 runs per review to about 10. The most runs any record had used was 24, still under the per-review budget of 40.

This is the right first move after a miss. You go to the setting that is known to find the violations, and then measure what it costs, rather than guessing at a midpoint before you have data.

Step 4: measure 72 hours of it​

By November 3 we had 153 pull requests reviewed at 200 lines, and enough records to compare the two kinds of reviewer side by side:

ReviewerModelRunsConfirmed findingsRefuted
Rule reviewerSonnet1,866about 56, mostly test style and processabout 57
General reviewerOpus26923, mostly real bugs24

Three things stand out.

  • Volume without yield. The rule reviewer made seven times as many runs as the general reviewer for about twice the confirmed findings, and those findings were mostly about test style and process. About ten runs went to every Go change and every pushed fix.
  • 51% of rule verdicts were not_applicable. Half of every reviewer's attention went to sections that had nothing to say about the change. That is a packing problem as much as a size problem: large rule files mixed Go, test, database and frontend rules, so a change in one area dragged in sections written for another.
  • Mechanical sections did worst. "No magic numbers" produced 1 confirmed finding and 6 refuted. Product naming produced 2 and 7.

That last point has a well-known standard behind it. Google's static analysis platform requires new checks to "produce less than 10% effective false positives", where an effective false positive is an issue developers took no action on (Software Engineering at Google, ch. 20). A section where six of seven findings are refuted is nowhere near that bar. An LLM is the wrong tool for a rule a regular expression can decide, and it is also the most expensive one.

Step 5: settle at 600, and move the mechanical rules out​

The setting that stuck was 600 lines, approved on November 3: the midpoint between the 1,200 that missed and the 200 that found. On the rule files as they stood, a typical Go change went from 7 rule reviewers to 3.

The number alone would have been a compromise. What made it work was the second half of the change, landed the same day: the mechanical rules left the LLM review entirely and became deterministic checks that run locally and in CI on what a change adds.

tools/rule-check · deterministic checks that replaced review sections
CheckToolFlags in added code
table-testSemgrepA Go test without t.Run inside a range loop
magic-numberSemgrepAn integer literal used as a timeout, limit, retry count or similar, outside a const
enum-caseSemgrepA typed code constant not in UPPER_SNAKE_CASE, or a state constant not lowercase
product-nameSemgrepAny spelling of the product name other than the two allowed ones, in user-facing strings
guide-sizepr_checks.pyAn agent guide over 400 lines that grows
ui-evidencepr_checks.pyUI files changed and fewer than two screenshot links in the pull request

Each rule enforced by a check then left the rule files, keeping one line that names the check. Large rule files were split by path scope (Go, tests, database, frontend), so fewer sections reach a reviewer only to be marked not_applicable. Both changes shrink the prompt every reviewer reads, which makes the 600-line batches hold more of what matters.

A prompt fix found along the way​

By November 4, reviewers were returning JSON with the wrong keys: things like rule_item and reason where the schema required item and why. The verdicts were unusable, because the cycle validates every answer against the schema before it records anything.

The cause was in the prompt. It described a violation in words, "the rule item" and "its reason", which primed exactly those key names. The schema that named the real keys came at the end, after hundreds of kilobytes of diff. The fix did two things at once: it stated the schema and its keys before the scope and the diff, and it changed the prose to use the schema's own names. A test now reads the required keys from the schema file and fails if any prompt omits one before the diff or uses the wrong names.

We changed two variables at once, and that matters. Anthropic's own long-context guidance says the opposite of half our fix: put long documents at the top and the query at the end, which "can improve response quality by up to 30 percent" (prompting best practices). Studies of long contexts find models use information at the start and end best, and the middle worst (Liu et al., Lost in the Middle). Our prompt had the schema at the end, so position alone may not have been the fault. The mismatched names probably were. We don't know which change did the work, and the honest version of this lesson is: make the prose and the schema use the same words, and pin the structure with a test.

The prompt limit kept climbing​

The reviewer prompt has a size bound, prompt_limit_bytes, so a runaway diff can't produce a run nobody can afford. Every raise is a configuration change in its own reviewed pull request, inside a validated range. Here is how often that happened:

DateLimit
Start400 KB
October 29600 KB, then 700 KB
November 1900 KB
November 21 MB, 1.15 MB, 2.2 MB, 2.9 MB

Seven raises in five days. Each one was locally reasonable: a verifier on one branch needed a 484,162-byte instruction, so the first raise went to 600 KB rather than split that pull request. But a bound you raise every time you hit it is not a bound. And a bigger prompt is not a better review, for the same reasons a bigger batch isn't.

So the November 3 change added a rule to the review skill: split first. When a review exceeds the limit, split the review range or the change. Raise the limit only when no split can fit, and then by the smallest amount, in a separate reviewed configuration pull request.

Where the tokens actually went​

It is easy to tune a reviewer and miss that it is a small part of the bill. Session logs from one laptop between October 31 and November 3 came to about $975 of tokens at list prices, and 61% of that was cache reads of contexts between 250K and 950K tokens. Long-running orchestrator sessions polled CI and tailed subagent output; implementers read whole files and screenshots. The reviewers mattered, and so did everything around them. The same plan that set 600 lines also added token-economy guidance aimed at those sessions, not only at the reviewers.

Failure modes of a tuned reviewer​

Failure modeSymptomThe check
Overloaded reviewerZero findings across many pull requestsRe-review a sample at a smaller batch
Underloaded reviewerHigh run count, most verdicts not_applicableTrack the not_applicable share; split rule files by path
Wrong tool for the ruleRefuted findings dominate a sectionPer-section confirmed to refuted ratio; move to a linter
Prompt and schema disagreeInvalid keys, unusable verdictsTest that the prompt names schema keys before the diff
Limit creepRepeated raises of a size boundSplit-first rule; raises only as separate reviewed changes
Cost hidden elsewhereReviewer tuned, bill unchangedMeasure whole-session token use, not only reviewer runs

How to measure your own​

Keep a review record per head, and compute these per week:

  1. Runs per pull request, split by reviewer kind.
  2. Confirmed findings per 100 runs, per reviewer kind and per rule section.
  3. Refuted share per section. Anything far over 10% is a candidate for a deterministic check, or for deletion.
  4. not_applicable share. Above half means your packing is wrong before your batch size is.
  5. Re-check miss rate. Every so often, re-review a few of the largest pull requests at a smaller batch and count confirmed violations the normal run marked followed. This is the only number that tells you what the reviewer is not seeing.

What this does not fix​

  • The sample is small. The decision to lower the batch rests on five re-checked pull requests and two misses. The working tree the re-check read was newer than the reviewed heads, though both findings were checked against the reviewed diff.
  • 600 is a midpoint, not an optimum. It was chosen between a value known to miss and a value known to cost too much. Larger runs may still miss some violations 200-line runs would find. The plan is to re-measure findings per pull request after the mechanical rules moved out.
  • Tokens weren't recorded per run. Reviewer records logged zero tokens, so the cost side of this comparison comes from run counts and a one-laptop session log, not per-review billing.
  • Same model family. None of this tuning makes a Sonnet reviewer independent of an Opus or Sonnet author. It makes it more efficient.

What it costs​

Every turn of this loop spent money to learn something. The re-check alone was 58 runs at up to 102,000 tokens each. The 200-line period ran 1,866 rule reviewer runs in 72 hours. The deterministic checks cost engineering time to write and maintain, and each one needs an escape hatch (a nosemgrep comment with a reason) for values an external contract fixes. And the split-first rule makes some changes slower to land. All of that is cheaper than a reviewer that approves everything, but none of it is free.

The playbook​

When you seeDo this
A reviewer that has stopped finding anythingRe-review the largest recent pull requests at a smaller batch before trusting the silence
A re-check that finds confirmed missesDrop to the batch size that found them, and price it in the same change
Most verdicts are not_applicableSplit rule files by path scope before touching the batch size
One section is mostly refutedTurn it into a linter or a structural check, then delete it from the rules
A general reviewer finds real bugs and a rule reviewer finds styleSpend on the general reviewer; shrink what the rule reviewer reads
Invalid or misnamed JSON answersUse the schema's own words in the prompt and pin the order with a test
A size limit raised more than once a weekSplit first; raise only in a separate reviewed change
A reviewer bill that won't moveMeasure the whole session; the orchestrator may be the cost

Tune the reviewer like any other model in production: with a sample you re-check, a number you trust, and the humility to hand the easy rules to code.


Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.