If Code Can Decide It, Don't Ask a Model
Some engineering rules belong in a linter, a Semgrep rule, a schema test or a drift check. Others need a reviewer who can read intent. This playbook sorts rules by type, shows how to migrate a rule from prose to a check, and explains why the allowlist for old violations should only ever shrink.
A team writes its rules down in a file their coding agents read: no magic numbers, no imports from another module's internals, every migration gets a down step, generated clients are never edited by hand. Then they add an AI reviewer and give it the same file. For a week it looks great. It catches things, it explains them well, and the pull requests feel safer.
In the second week the reviewer flags a constant in a test fixture as a magic number, and the author spends ten minutes arguing it out of the finding. Two days later a 1,400-line pull request imports a private package from the billing module, and the review says every rule was followed. The team asked a model to do a job that a twenty-line script does perfectly, and a job that only a careful reader can do, in the same breath.
This post is a playbook for splitting those jobs. The test is simple to state: if you can write down a function from the repository's state to pass or fail, with no judgement inside it, that rule belongs in code. Everything else (what the change is for, whether it is correct, whether the design will hold) belongs with a reviewer who can read. Most rule files mix the two, and both halves get worse for it.
Two kinds of judge
It helps to be precise about what each kind of check is, mechanically.
A deterministic check is a program. It reads files, or a syntax tree, or a schema, and returns a verdict. Given the same input it returns the same answer every time. It runs in milliseconds to seconds, costs nothing per run, and can sit in a pre-commit hook, a CI job and an agent's tool hook at once. When it is wrong, you can see exactly why, because the rule is the code. Its blind spots are knowable: a pattern that does not match is a pattern you did not write.
An LLM reviewer is a reader. It takes the diff, some context and a list of rules, and produces findings. It can judge things no program can: whether the change does what the pull request says, whether an abstraction will survive the next feature, whether an error message would help a person at 2 a.m. But it is probabilistic. Two runs on the same diff can disagree. It costs tokens on every run, takes seconds to minutes, and its attention thins out as the diff and the rule list grow. When it is wrong, there is no line of code to fix.
The middle branch matters more than it looks. Many rules have a mechanical
core wrapped in a judgement call. "No magic numbers" is mostly mechanical
(a numeric literal outside a named constant) with a judgement edge (is 2
in len(parts) == 2 magic?). Split it: the check enforces the core with a
short list of allowed literals, and the reviewer never sees the rule at all.
Here is how the two compare on the dimensions that decide where a rule should live:
| Dimension | Deterministic check | LLM reviewer |
|---|---|---|
| Same input, same verdict | Always | Often, not always |
| Cost per run | Effectively zero | Tokens on every run, growing with diff size |
| Latency | Milliseconds to seconds | Seconds to minutes |
| Where it can run | Editor, agent hook, pre-commit, CI | Usually once per change |
| False positives | Fixed by editing the rule, once | Argued case by case, forever |
| False negatives | Known: whatever the pattern misses | Unknown: depends on attention that run |
| What it can judge | Structure, names, paths, schemas, equality | Intent, correctness, design, clarity |
| Effect of a longer rule list | None | Each rule gets less attention |
Why "put it in the reviewer prompt" fails
The obvious move, when a rule keeps getting broken, is to write it down more forcefully and hand it to the AI reviewer. It fails for three reasons that compound.
Exhaustive scanning is the thing models are worst at. A mechanical rule asks "is there any occurrence of X anywhere in this diff?" A program answers that by looking at every token. A model answers it by reading, and reading skims. The bigger the diff, the more likely the one violating line sits in a part of the context that got less attention. The reviewer will often say "followed" with full confidence, because it saw no violation in what it looked at.
Mechanical rules produce the noisiest findings. A model asked to find magic numbers will find numbers. Some are magic, many are not, and every false positive costs an author's time and a reviewer's credibility. Google's static analysis team learned this a decade ago at scale. Their criteria for a new check in Tricorder include producing fewer than 10% "effective false positives", where an issue counts as an effective false positive if the developer took no positive action after seeing it. They also note that developers ignore results they do not understand. A reviewer that is wrong a third of the time on a rule teaches everyone to skip that rule's findings, including the true ones.
Findings can be argued away. A deterministic check does not negotiate. An LLM finding sits in a conversation, and the author (often another agent) can reply with a plausible reason why this case is fine. Sometimes it is. Often it is the author's own framing being read back. A rule that can be talked out of its verdict is not enforced; it is suggested.
And every mechanical rule in the list takes attention from the judgement rules. A long rule list makes the reviewer worse at its real job.
The same logic applies to agent instructions. Claude Code's own docs make
the point about hooks: they give you deterministic control, so certain
actions always happen "rather than relying on the LLM to choose to run
them". A rule in CLAUDE.md is
a request. A rule in a hook or a CI job is a fact about the repository.
The rule types, and where each one lives
Most rule files contain the same handful of rule types. The playbook below sorts them. The left column is the kind of rule; the right columns say who should decide it and with what.
| Rule type | Example | Decided by | Typical mechanism |
|---|---|---|---|
| Formatting and style | Import grouping, quote style, line length | Code | Formatter plus linter (gofmt, Prettier, ruff) |
| Banned call or pattern | No time.Sleep in tests; no console.log in shipped code | Code | Linter rule or Semgrep pattern |
| Module and layer boundaries | The web app never imports server internals | Code | Import rules (depguard, dependency-cruiser, ArchUnit) or a grep over paths |
| Schemas and contracts | Every API response matches the published schema | Code | Schema validation in tests; contract tests |
| Generated files | Mocks, clients and docs match their source | Code | Regenerate, then fail on any diff |
| Configured limits | No numeric literals for timeouts or sizes | Code, with a narrow allowlist | Semgrep or a custom lint rule |
| Test hygiene | No wall-clock upper bounds on elapsed time | Code | A custom lint over test files |
| Migrations | Every up migration has a down; numbers never collide | Code | A script over the migrations folder |
| Naming that carries meaning | A function name says what it does | Review | Human or LLM reviewer |
| Intent | The change does what the pull request claims, and nothing else | Review | LLM reviewer with the PR text and diff |
| Correctness in context | The retry cannot double-charge; the lock order is safe | Review, plus tests | Reviewer finds it, a test pins it |
| Design | The abstraction will survive the next two features | Review | Human, with LLM help |
| Error messages and UX copy | A failure tells the reader what to do next | Review | LLM reviewer against a short guide |
| Test meaningfulness | The test would fail if the feature broke | Review, plus mutation testing where cheap | Reviewer reads the assertion |
The top half of the table is long and the bottom half short, and most reviewer prompts have them inverted. The review rows are where being wrong is expensive and subtle, which is where you want a strong model's full attention.
Make boundaries greppable
Some rules are hard to check only because of how the code is laid out. "The
console must not reach into the storage layer" is a judgement call if storage
code is scattered across the tree. It is a one-line check if storage lives
under one directory and the rule is "no file under console/ imports a path
under store/internal/."
A boundary you can grep for is a boundary an agent cannot accidentally cross, because the check fails on the first commit that tries. When review keeps flagging the same architectural violation, ask whether moving files would turn it into a path rule.
Drift checks for generated files
Generated files are the purest case for a check. The rule is "this file equals what the generator produces from its source." The check is: run the generator in CI, then fail if the working tree changed. It catches the three ways generated files go wrong: someone hand-edited the output, someone changed the source and forgot to regenerate, or the generator version moved. An LLM reviewer cannot do this at all without running the generator, at which point it is a deterministic check with extra steps. The same shape works for anything derived: an API reference, a lockfile, an inventory of test files.
Migrating a rule from prose to a check
Moving a rule is a small project with a fixed sequence. Skipping steps is how teams end up with checks everyone disables.
- Pick the rule. Start with rules that are broken often and that you can decide without judgement. Your reviewer's history tells you which: rules with many findings, many of them disputed, are the best candidates.
- Write the predicate. Use the cheapest tool that can express it: a grep for paths, a linter or Semgrep pattern (which matches the syntax tree, not text) for code shapes, a test for schemas.
- Run it on everything. Not on the diff: on the whole repository. The hit count is your first real measurement of the rule.
- Triage the hits. Read a sample. False positives mean the predicate is too broad, so go back to step 2. Do not ship a check you would not trust on its first day.
- Baseline the true hits. Existing violations go into an allowlist that may only shrink. More on that below.
- Wire it in everywhere. The agent's post-edit hook, pre-commit, and CI. CI cannot be skipped; the hook saves the most time.
- Delete the prose. Remove the rule from the reviewer's prompt, or it keeps producing noisy findings about something already enforced.
- Measure. Track hits, allowlist size and any violation that reached the main branch anyway (an escape means the predicate has a gap).
The failure message is the instruction
When an agent is the one reading the failure, the message is effectively a prompt. A message that says "rule 14 violated" sends the agent searching. A message that names the file, the line, the rule and the fix gets it right on the next edit. Write every check's output the way you would write a good compiler error.
| Part | Example |
|---|---|
| Where | console/chat.go:88 |
| What | Imports store/internal/rows, which is private to the store module |
| Rule | Module boundaries: other modules use the store's public package |
| Fix | Import store and call store.ListRows instead |
| Escape hatch | None for this rule; ask the store owners to export what you need |
If a check has no legitimate exception, say so, or an agent will look for one.
Ratchets: allowlists that may only shrink
Almost no rule can be switched on cleanly in an existing codebase. There are always hundreds of old violations. The two common responses are both bad. Fixing everything first means the check never ships. Turning it on as a warning means it is ignored.
A ratchet is the third option. You record the current violations, per file and per rule, in a checked-in file. The check passes if a file has no more violations than its recorded count, and fails if it has more. When someone fixes a violation, the count must come down in the same change. The count can go down. It cannot go up.
Notion's engineering team published a clear account of their ratcheting system for ESLint in March 2025: counts per file and rule, automatic lowering when developers fix issues, CI blocking increases, and approval required for any increase. They moved the file from JSON to TSV because JSON caused merge conflicts, and open-sourced the system as eslint-seatbelt. The idea generalises to any check.
A few rules keep a ratchet honest:
| Ratchet rule | Why |
|---|---|
| Key entries by file and rule, never by line number | Line numbers move on every edit and make the file churn |
| Fail when the count drops but the file was not lowered | Otherwise the slack can be reused by a new violation later |
| Any increase is a separate, reviewed change with a reason | Growth should be a decision someone signs, not a side effect |
| An agent may lower entries but never raise them | Agents under pressure to get green will take any exit you leave |
| On merge conflict, take the smaller count | Two branches that each fixed something should both keep the gain |
| Report the total on a dashboard or in a weekly note | A ratchet nobody watches stalls at its starting count |
The fourth row deserves emphasis for teams running agents. An agent that can edit the allowlist can make any check pass. Put the allowlist under the same protection as the check itself: a change that raises it should fail CI unless it carries a reviewed justification.
What stays with the reviewer
Once the mechanical rules leave, the reviewer's job gets smaller and sharper. That is the point. Give it fewer things and it does them better.
- Intent. Does the diff do what the pull request says, and only that? An agent that was asked to fix a bug and also "tidied" three files has changed behaviour nobody asked about.
- Correctness in context. Can this retry double-charge? Is the new lock taken in the same order as the old one? What happens when the list is empty? These need someone who reads the surrounding code.
- Design. Will this abstraction hold for the next two features, or is it shaped exactly like today's one caller?
- Clarity for the next reader. Names, comments and error messages that a tired person can act on.
- Test adequacy. Would this test fail if the feature broke? Coverage tools say a line ran. Only a reader can say the assertion means something.
Keep diffs small, and hand the reviewer the deterministic results as facts ("lint passed") rather than as tasks to re-derive.
Models are excellent check authors. The best use of an LLM in this playbook is often upstream: ask it to read your rule file, propose which rules are decidable, and draft the Semgrep rule or lint plugin for each. The draft is checkable (run it, count hits, read a sample), so a wrong draft costs minutes. The model writes the check once; the check then runs a million times for free.
A worked example
Take a typical rule file of twelve rules that a team hands both to its coding agents and to an AI reviewer. Here is where each one ends up after a pass through this playbook.
| Rule as written | Where it moved | Why |
|---|---|---|
| Format with the project formatter | Formatter in a post-edit hook and CI | Fully mechanical |
No imports from another module's internal/ | Import rule in the linter, plus a path grep | Decidable once files live under clear directories |
No sleep calls in tests | Semgrep pattern on test files | Syntax match; a few legitimate cases go in an allowlist |
| No numeric literals for timeouts or sizes | Custom lint with a short list of allowed literals; old hits ratcheted | Mechanical core; the judgement edge became the allowlist |
| Generated clients are never hand-edited | Regenerate in CI and fail on diff | Derived file |
| API responses match the published schema | Schema validation in the API test suite | Decidable by a validator |
| Every migration has a down step | A short script over the migrations folder | Filename and content check |
| Error messages say what to do next | Stays in review | Needs a reader |
| The change does only what the PR describes | Stays in review | Intent |
| Retries must be idempotent | Review finds it, a test pins it | Correctness in context |
| Prefer small, composable functions | Stays in review, as a design note | Judgement; a line limit would be gamed |
| Tests assert behaviour, not implementation | Stays in review | Test meaning |
Seven of twelve rules moved to code. The reviewer prompt went from twelve rules to five, and the five left are the ones where a reader adds value. The fixes compose: greppable boundaries made rule 2 checkable; the ratchet made rule 4 shippable on day one; the shorter list gives the reviewer more attention for rules 8 to 12.
The penultimate row is worth a second look. "Prefer small functions" could be made mechanical with a line limit. Do not. A hard limit on a judgement rule gets satisfied by splitting a function into two that make less sense. Only mechanise rules whose letter is their spirit.
How to measure it
You cannot manage the split without numbers. Track these per rule:
| Metric | What it tells you | Action when it is bad |
|---|---|---|
| Reviewer findings per rule, confirmed versus dismissed | Which review rules are noisy | A noisy mechanical rule is a migration candidate |
| Effective false positive rate per check | Whether people trust the check | Above about 10%, fix the predicate before anything else |
| Allowlist size over time | Whether the ratchet is moving | Flat for a month means nobody owns it |
| Escapes: violations found on main after merge | Gaps in the predicate or in review | Each escape becomes a test case for the check |
| Reviewer tokens per change | What review costs | Rising cost with flat findings means the rule list is too long |
The first metric is the engine of the whole playbook. If your reviewer emits structured findings (rule, file, verdict), you can count, per rule, how often its findings are confirmed. Rules with low confirmation and a mechanical shape are the next ones to move.
What this does not fix
Deterministic checks are not a free lunch, and some failure modes get worse.
- Checks catch only what you encoded. A check is silent about everything outside its pattern. A team that moves its rules to code and stops reading diffs has traded one blind spot for another.
- Agents satisfy the letter. A model under pressure to pass will rename a
variable, move a literal into a constant called
TWO, or split a file to dodge a size limit. The check passes and the rule's purpose is lost. This is why judgement rules stay in review. - Text patterns are brittle. A grep for an import path misses an alias or a re-export. Prefer syntax-aware tools for anything beyond paths.
- Checks rot. Frameworks change, directories move, and a check that matched yesterday's layout passes today because it matches nothing. Give every check a test with a known violation, so a check that silently stopped matching fails its own test.
- Ratchets can freeze. An allowlist that never shrinks is a permanent exemption with extra steps. It needs an owner and a visible trend.
What it costs
Every check is code you maintain, with an owner, a test, a clear failure message, and updates when the codebase moves. Sixty custom checks are a small product. Early migrations take a day or two each, mostly in triage and baselining. And the ratchet file gets small, frequent edits, which means occasional merge conflicts.
Weigh that against the alternative: a reviewer that re-litigates the same mechanical rules on every pull request, at token cost, with a false positive rate high enough to teach everyone to ignore it.
The playbook
Keyed to the kind of rule in front of you:
| If the rule is about... | Decide it with | Ship it by | Watch for |
|---|---|---|---|
| Formatting or style | Formatter and linter | Post-edit hook plus CI | Nothing; this is solved |
| A banned call or pattern | Semgrep or a lint rule | Run on the repo, triage, ratchet existing hits | Aliases and wrappers the pattern misses |
| A module or layer boundary | Import rules or a path grep | Move files until the boundary is greppable | Re-exports that launder the import |
| A schema or contract | Validation in tests | Generate fixtures from the schema | Fields optional in the schema but required in practice |
| A generated or derived file | Regenerate and diff in CI | Pin the generator version | Generators that are not deterministic |
| A limit or magic value | Custom lint with a narrow allowlist | Ratchet old hits | Constants named after their values |
| Mixed: a mechanical core plus judgement | Split it | Core to a check, the rest to review | Mechanising the judgement part |
| Intent, correctness, design, clarity | A reviewer: model, human, or both | A short rule list and small diffs | Long prompts and same-model rubber-stamping |
Start with the reviewer's own history: the rules it flags most and gets confirmed least are almost always the ones a program should be enforcing. Move one a week. Code decides what code can decide, and the reader gets its attention back for the questions only a reader can answer.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.