Skip to main content

If Code Can Decide It, Don't Ask a Model

Some engineering rules belong in a linter, a Semgrep rule, a schema test or a drift check. Others need a reviewer who can read intent. This playbook sorts rules by type, shows how to migrate a rule from prose to a check, and explains why the allowlist for old violations should only ever shrink.

CatalEx Engineering · Published September 29, 2026 · 09:00 UTC

A team writes its rules down in a file their coding agents read: no magic numbers, no imports from another module's internals, every migration gets a down step, generated clients are never edited by hand. Then they add an AI reviewer and give it the same file. For a week it looks great. It catches things, it explains them well, and the pull requests feel safer.

In the second week the reviewer flags a constant in a test fixture as a magic number, and the author spends ten minutes arguing it out of the finding. Two days later a 1,400-line pull request imports a private package from the billing module, and the review says every rule was followed. The team asked a model to do a job that a twenty-line script does perfectly, and a job that only a careful reader can do, in the same breath.

This post is a playbook for splitting those jobs. The test is simple to state: if you can write down a function from the repository's state to pass or fail, with no judgement inside it, that rule belongs in code. Everything else (what the change is for, whether it is correct, whether the design will hold) belongs with a reviewer who can read. Most rule files mix the two, and both halves get worse for it.

Two kinds of judge​

It helps to be precise about what each kind of check is, mechanically.

A deterministic check is a program. It reads files, or a syntax tree, or a schema, and returns a verdict. Given the same input it returns the same answer every time. It runs in milliseconds to seconds, costs nothing per run, and can sit in a pre-commit hook, a CI job and an agent's tool hook at once. When it is wrong, you can see exactly why, because the rule is the code. Its blind spots are knowable: a pattern that does not match is a pattern you did not write.

An LLM reviewer is a reader. It takes the diff, some context and a list of rules, and produces findings. It can judge things no program can: whether the change does what the pull request says, whether an abstraction will survive the next feature, whether an error message would help a person at 2 a.m. But it is probabilistic. Two runs on the same diff can disagree. It costs tokens on every run, takes seconds to minutes, and its attention thins out as the diff and the rule list grow. When it is wrong, there is no line of code to fix.

The middle branch matters more than it looks. Many rules have a mechanical core wrapped in a judgement call. "No magic numbers" is mostly mechanical (a numeric literal outside a named constant) with a judgement edge (is 2 in len(parts) == 2 magic?). Split it: the check enforces the core with a short list of allowed literals, and the reviewer never sees the rule at all.

Here is how the two compare on the dimensions that decide where a rule should live:

DimensionDeterministic checkLLM reviewer
Same input, same verdictAlwaysOften, not always
Cost per runEffectively zeroTokens on every run, growing with diff size
LatencyMilliseconds to secondsSeconds to minutes
Where it can runEditor, agent hook, pre-commit, CIUsually once per change
False positivesFixed by editing the rule, onceArgued case by case, forever
False negativesKnown: whatever the pattern missesUnknown: depends on attention that run
What it can judgeStructure, names, paths, schemas, equalityIntent, correctness, design, clarity
Effect of a longer rule listNoneEach rule gets less attention

Why "put it in the reviewer prompt" fails​

The obvious move, when a rule keeps getting broken, is to write it down more forcefully and hand it to the AI reviewer. It fails for three reasons that compound.

Exhaustive scanning is the thing models are worst at. A mechanical rule asks "is there any occurrence of X anywhere in this diff?" A program answers that by looking at every token. A model answers it by reading, and reading skims. The bigger the diff, the more likely the one violating line sits in a part of the context that got less attention. The reviewer will often say "followed" with full confidence, because it saw no violation in what it looked at.

Mechanical rules produce the noisiest findings. A model asked to find magic numbers will find numbers. Some are magic, many are not, and every false positive costs an author's time and a reviewer's credibility. Google's static analysis team learned this a decade ago at scale. Their criteria for a new check in Tricorder include producing fewer than 10% "effective false positives", where an issue counts as an effective false positive if the developer took no positive action after seeing it. They also note that developers ignore results they do not understand. A reviewer that is wrong a third of the time on a rule teaches everyone to skip that rule's findings, including the true ones.

Findings can be argued away. A deterministic check does not negotiate. An LLM finding sits in a conversation, and the author (often another agent) can reply with a plausible reason why this case is fine. Sometimes it is. Often it is the author's own framing being read back. A rule that can be talked out of its verdict is not enforced; it is suggested.

And every mechanical rule in the list takes attention from the judgement rules. A long rule list makes the reviewer worse at its real job.

The same logic applies to agent instructions. Claude Code's own docs make the point about hooks: they give you deterministic control, so certain actions always happen "rather than relying on the LLM to choose to run them". A rule in CLAUDE.md is a request. A rule in a hook or a CI job is a fact about the repository.

The rule types, and where each one lives​

Most rule files contain the same handful of rule types. The playbook below sorts them. The left column is the kind of rule; the right columns say who should decide it and with what.

Rule typeExampleDecided byTypical mechanism
Formatting and styleImport grouping, quote style, line lengthCodeFormatter plus linter (gofmt, Prettier, ruff)
Banned call or patternNo time.Sleep in tests; no console.log in shipped codeCodeLinter rule or Semgrep pattern
Module and layer boundariesThe web app never imports server internalsCodeImport rules (depguard, dependency-cruiser, ArchUnit) or a grep over paths
Schemas and contractsEvery API response matches the published schemaCodeSchema validation in tests; contract tests
Generated filesMocks, clients and docs match their sourceCodeRegenerate, then fail on any diff
Configured limitsNo numeric literals for timeouts or sizesCode, with a narrow allowlistSemgrep or a custom lint rule
Test hygieneNo wall-clock upper bounds on elapsed timeCodeA custom lint over test files
MigrationsEvery up migration has a down; numbers never collideCodeA script over the migrations folder
Naming that carries meaningA function name says what it doesReviewHuman or LLM reviewer
IntentThe change does what the pull request claims, and nothing elseReviewLLM reviewer with the PR text and diff
Correctness in contextThe retry cannot double-charge; the lock order is safeReview, plus testsReviewer finds it, a test pins it
DesignThe abstraction will survive the next two featuresReviewHuman, with LLM help
Error messages and UX copyA failure tells the reader what to do nextReviewLLM reviewer against a short guide
Test meaningfulnessThe test would fail if the feature brokeReview, plus mutation testing where cheapReviewer reads the assertion

The top half of the table is long and the bottom half short, and most reviewer prompts have them inverted. The review rows are where being wrong is expensive and subtle, which is where you want a strong model's full attention.

Make boundaries greppable​

Some rules are hard to check only because of how the code is laid out. "The console must not reach into the storage layer" is a judgement call if storage code is scattered across the tree. It is a one-line check if storage lives under one directory and the rule is "no file under console/ imports a path under store/internal/."

A boundary you can grep for is a boundary an agent cannot accidentally cross, because the check fails on the first commit that tries. When review keeps flagging the same architectural violation, ask whether moving files would turn it into a path rule.

Drift checks for generated files​

Generated files are the purest case for a check. The rule is "this file equals what the generator produces from its source." The check is: run the generator in CI, then fail if the working tree changed. It catches the three ways generated files go wrong: someone hand-edited the output, someone changed the source and forgot to regenerate, or the generator version moved. An LLM reviewer cannot do this at all without running the generator, at which point it is a deterministic check with extra steps. The same shape works for anything derived: an API reference, a lockfile, an inventory of test files.

Migrating a rule from prose to a check​

Moving a rule is a small project with a fixed sequence. Skipping steps is how teams end up with checks everyone disables.

  1. Pick the rule. Start with rules that are broken often and that you can decide without judgement. Your reviewer's history tells you which: rules with many findings, many of them disputed, are the best candidates.
  2. Write the predicate. Use the cheapest tool that can express it: a grep for paths, a linter or Semgrep pattern (which matches the syntax tree, not text) for code shapes, a test for schemas.
  3. Run it on everything. Not on the diff: on the whole repository. The hit count is your first real measurement of the rule.
  4. Triage the hits. Read a sample. False positives mean the predicate is too broad, so go back to step 2. Do not ship a check you would not trust on its first day.
  5. Baseline the true hits. Existing violations go into an allowlist that may only shrink. More on that below.
  6. Wire it in everywhere. The agent's post-edit hook, pre-commit, and CI. CI cannot be skipped; the hook saves the most time.
  7. Delete the prose. Remove the rule from the reviewer's prompt, or it keeps producing noisy findings about something already enforced.
  8. Measure. Track hits, allowlist size and any violation that reached the main branch anyway (an escape means the predicate has a gap).

The failure message is the instruction​

When an agent is the one reading the failure, the message is effectively a prompt. A message that says "rule 14 violated" sends the agent searching. A message that names the file, the line, the rule and the fix gets it right on the next edit. Write every check's output the way you would write a good compiler error.

Anatomy of a useful check failure
PartExample
Whereconsole/chat.go:88
WhatImports store/internal/rows, which is private to the store module
RuleModule boundaries: other modules use the store's public package
FixImport store and call store.ListRows instead
Escape hatchNone for this rule; ask the store owners to export what you need

If a check has no legitimate exception, say so, or an agent will look for one.

Ratchets: allowlists that may only shrink​

Almost no rule can be switched on cleanly in an existing codebase. There are always hundreds of old violations. The two common responses are both bad. Fixing everything first means the check never ships. Turning it on as a warning means it is ignored.

A ratchet is the third option. You record the current violations, per file and per rule, in a checked-in file. The check passes if a file has no more violations than its recorded count, and fails if it has more. When someone fixes a violation, the count must come down in the same change. The count can go down. It cannot go up.

Notion's engineering team published a clear account of their ratcheting system for ESLint in March 2025: counts per file and rule, automatic lowering when developers fix issues, CI blocking increases, and approval required for any increase. They moved the file from JSON to TSV because JSON caused merge conflicts, and open-sourced the system as eslint-seatbelt. The idea generalises to any check.

A few rules keep a ratchet honest:

Ratchet ruleWhy
Key entries by file and rule, never by line numberLine numbers move on every edit and make the file churn
Fail when the count drops but the file was not loweredOtherwise the slack can be reused by a new violation later
Any increase is a separate, reviewed change with a reasonGrowth should be a decision someone signs, not a side effect
An agent may lower entries but never raise themAgents under pressure to get green will take any exit you leave
On merge conflict, take the smaller countTwo branches that each fixed something should both keep the gain
Report the total on a dashboard or in a weekly noteA ratchet nobody watches stalls at its starting count

The fourth row deserves emphasis for teams running agents. An agent that can edit the allowlist can make any check pass. Put the allowlist under the same protection as the check itself: a change that raises it should fail CI unless it carries a reviewed justification.

What stays with the reviewer​

Once the mechanical rules leave, the reviewer's job gets smaller and sharper. That is the point. Give it fewer things and it does them better.

  • Intent. Does the diff do what the pull request says, and only that? An agent that was asked to fix a bug and also "tidied" three files has changed behaviour nobody asked about.
  • Correctness in context. Can this retry double-charge? Is the new lock taken in the same order as the old one? What happens when the list is empty? These need someone who reads the surrounding code.
  • Design. Will this abstraction hold for the next two features, or is it shaped exactly like today's one caller?
  • Clarity for the next reader. Names, comments and error messages that a tired person can act on.
  • Test adequacy. Would this test fail if the feature broke? Coverage tools say a line ran. Only a reader can say the assertion means something.

Keep diffs small, and hand the reviewer the deterministic results as facts ("lint passed") rather than as tasks to re-derive.

Models are excellent check authors. The best use of an LLM in this playbook is often upstream: ask it to read your rule file, propose which rules are decidable, and draft the Semgrep rule or lint plugin for each. The draft is checkable (run it, count hits, read a sample), so a wrong draft costs minutes. The model writes the check once; the check then runs a million times for free.

A worked example​

Take a typical rule file of twelve rules that a team hands both to its coding agents and to an AI reviewer. Here is where each one ends up after a pass through this playbook.

Rule as writtenWhere it movedWhy
Format with the project formatterFormatter in a post-edit hook and CIFully mechanical
No imports from another module's internal/Import rule in the linter, plus a path grepDecidable once files live under clear directories
No sleep calls in testsSemgrep pattern on test filesSyntax match; a few legitimate cases go in an allowlist
No numeric literals for timeouts or sizesCustom lint with a short list of allowed literals; old hits ratchetedMechanical core; the judgement edge became the allowlist
Generated clients are never hand-editedRegenerate in CI and fail on diffDerived file
API responses match the published schemaSchema validation in the API test suiteDecidable by a validator
Every migration has a down stepA short script over the migrations folderFilename and content check
Error messages say what to do nextStays in reviewNeeds a reader
The change does only what the PR describesStays in reviewIntent
Retries must be idempotentReview finds it, a test pins itCorrectness in context
Prefer small, composable functionsStays in review, as a design noteJudgement; a line limit would be gamed
Tests assert behaviour, not implementationStays in reviewTest meaning

Seven of twelve rules moved to code. The reviewer prompt went from twelve rules to five, and the five left are the ones where a reader adds value. The fixes compose: greppable boundaries made rule 2 checkable; the ratchet made rule 4 shippable on day one; the shorter list gives the reviewer more attention for rules 8 to 12.

The penultimate row is worth a second look. "Prefer small functions" could be made mechanical with a line limit. Do not. A hard limit on a judgement rule gets satisfied by splitting a function into two that make less sense. Only mechanise rules whose letter is their spirit.

How to measure it​

You cannot manage the split without numbers. Track these per rule:

MetricWhat it tells youAction when it is bad
Reviewer findings per rule, confirmed versus dismissedWhich review rules are noisyA noisy mechanical rule is a migration candidate
Effective false positive rate per checkWhether people trust the checkAbove about 10%, fix the predicate before anything else
Allowlist size over timeWhether the ratchet is movingFlat for a month means nobody owns it
Escapes: violations found on main after mergeGaps in the predicate or in reviewEach escape becomes a test case for the check
Reviewer tokens per changeWhat review costsRising cost with flat findings means the rule list is too long

The first metric is the engine of the whole playbook. If your reviewer emits structured findings (rule, file, verdict), you can count, per rule, how often its findings are confirmed. Rules with low confirmation and a mechanical shape are the next ones to move.

What this does not fix​

Deterministic checks are not a free lunch, and some failure modes get worse.

  • Checks catch only what you encoded. A check is silent about everything outside its pattern. A team that moves its rules to code and stops reading diffs has traded one blind spot for another.
  • Agents satisfy the letter. A model under pressure to pass will rename a variable, move a literal into a constant called TWO, or split a file to dodge a size limit. The check passes and the rule's purpose is lost. This is why judgement rules stay in review.
  • Text patterns are brittle. A grep for an import path misses an alias or a re-export. Prefer syntax-aware tools for anything beyond paths.
  • Checks rot. Frameworks change, directories move, and a check that matched yesterday's layout passes today because it matches nothing. Give every check a test with a known violation, so a check that silently stopped matching fails its own test.
  • Ratchets can freeze. An allowlist that never shrinks is a permanent exemption with extra steps. It needs an owner and a visible trend.

What it costs​

Every check is code you maintain, with an owner, a test, a clear failure message, and updates when the codebase moves. Sixty custom checks are a small product. Early migrations take a day or two each, mostly in triage and baselining. And the ratchet file gets small, frequent edits, which means occasional merge conflicts.

Weigh that against the alternative: a reviewer that re-litigates the same mechanical rules on every pull request, at token cost, with a false positive rate high enough to teach everyone to ignore it.

The playbook​

Keyed to the kind of rule in front of you:

If the rule is about...Decide it withShip it byWatch for
Formatting or styleFormatter and linterPost-edit hook plus CINothing; this is solved
A banned call or patternSemgrep or a lint ruleRun on the repo, triage, ratchet existing hitsAliases and wrappers the pattern misses
A module or layer boundaryImport rules or a path grepMove files until the boundary is greppableRe-exports that launder the import
A schema or contractValidation in testsGenerate fixtures from the schemaFields optional in the schema but required in practice
A generated or derived fileRegenerate and diff in CIPin the generator versionGenerators that are not deterministic
A limit or magic valueCustom lint with a narrow allowlistRatchet old hitsConstants named after their values
Mixed: a mechanical core plus judgementSplit itCore to a check, the rest to reviewMechanising the judgement part
Intent, correctness, design, clarityA reviewer: model, human, or bothA short rule list and small diffsLong prompts and same-model rubber-stamping

Start with the reviewer's own history: the rules it flags most and gets confirmed least are almost always the ones a program should be enforcing. Move one a week. Code decides what code can decide, and the reader gets its attention back for the questions only a reader can answer.


Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.