Judging the Judge: When Your Eval Is the Thing That's Wrong
We argued that you can't ship what you can't measure. Here's the uncomfortable half of that sentence: your measurement is a model too, and it is also wrong sometimes.
We score agent runs with an LLM judge — it reads a run and grades it against the KPIs that agent was given. That judge is treated, by everyone downstream of it, as an instrument. A thermometer. Numbers come out; people make decisions. But a judge is not a thermometer. It's a second fallible system wearing a lab coat, and every number your team trusts is resting on it.
So we built an eval of the judge. An LLM-as-judge eval of an LLM-as-judge. This post is what it found — including the part where the judge was starving for evidence and we nearly rewrote the rubric instead.
The eval has to drive the real thing
The first design decision is the one most people get wrong, and it's not subtle: the eval drives the exact production path. Same prompt file, same input renderer, same parser, same model alias.
That sounds like hygiene. It's load-bearing. The moment your eval builds its own tidy little prompt and calls the model directly, it is measuring a system that does not exist. Every divergence — a renderer your eval skips, a parser it reimplements more forgivingly — is a place where the eval is green and production is not. You didn't evaluate your judge. You evaluated a fictional cousin of it that happens to share a rubric.
Thirteen ways a judge goes wrong
The suite started at 17 synthetic scenarios across 13 failure families, plus 7 pairwise ordering rules. It later grew to 24 scenarios and 9 orderings. Real-run fixtures with hand-labeled score bands sit alongside the synthetics, because synthetic scenarios test the failure you imagined and real runs test the ones you didn't.
The failure families are the part worth stealing. None of them are obvious until a judge has burned you:
Calibration — does the score land in the band a human would give? Correctness-trap — a confident, articulate, wrong answer must score near zero. This is the one that separates a judge from a style critic. Sycophancy — the agent claims success; the tool results say it failed. Who does the judge believe? A judge that takes the agent's word for it is a compliance machine, not a grader. Grounding — fabricated citations. Injection — instructions embedded inside tool results, aimed at the judge. Your judge reads attacker-influenced text by design; treat it that way. Estimation — unmeasurable KPIs like "time saved." Verbosity bias — does length buy score? Relevance — beautiful output, wrong topic. Truncation — a display artifact the judge must not read as incomplete work.
Plus over-penalty, trajectory, structural load (eight KPIs at once), rubric anchoring, and autonomy. Thirteen families, because a judge doesn't fail in one direction — it fails in whichever direction your rubric was quietest about.
Orderings: the metric that survives your rubric edits
If you take one thing from this post, take this one.
Most judge metrics are absolute. In-band rate asks whether a score landed where a human would put it. That's useful and it is fragile — it drifts with every rubric edit, every anchor you reword, every criterion you add. Tighten the rubric on Tuesday and Monday's calibration is partly fiction.
Orderings don't drift. An ordering is a pairwise constraint: run A must score above run B. Not by how much. Just above. Whatever your absolute numbers are doing, whatever the anchors say this week, the thorough grounded run outranks the confident fabricated one — or your judge is broken in a way no amount of decimal precision will save.
This is a calibration-free signal, and that property is the whole point. It survives rubric changes, prompt rewrites, and model swaps, because it asks the judge to do the thing you actually need it to do: distinguish better work from worse work. A judge emitting 74.3 and 68.1 looks precise. If it can't reliably put the right run on top, that precision is decoration.
The rest of the metrics: in-band rate, parse/coverage rate, and stddev across trials — trial-to-trial consistency, which is just asking whether the judge is stable against itself before you ask it to be right about anything else. The suite passes only when in-band ≥ 0.9, parse ≥ 0.98, and all orderings hold. Orderings are the one with no partial credit. Six out of seven is a failing grade.
Seven iterations
| Iteration | Change | In-band | Orderings | Stddev |
|---|---|---|---|---|
| baseline | — | 76.8% | 6/7 | 11.0 |
| 1 | renderer: final response 800→4000 chars, trail summaries 100→400 | 91.7% | 6/7 | 7.9 |
| 2 | prompt: KPI independence, fraction rule, citation matching | 94.0% | 6/7 | 5.7 |
| 3 | renderer: tool-call arguments added to the decision trail | 95.8% | 6/7 | 5.7 |
| 4 | prompt: calibration anchors reworked | 96.8% | 7/7 | 4.3 |
| 5 | judge temperature 0; empty transcripts score ≤10 | 98.2% | 7/7 | 4.0 |
| 6 | head+tail windowing + adversarial scenarios added | 91.0%† | 9/9 | 7.8 |
| 7 | prompt: truncation is a display artifact; padding earns nothing | 95.2%† | 9/9 | 5.9 |
† Iterations 6–7 run against the harder 24-scenario suite and are not comparable to iterations 1–5. The suite got harder; the number went down. We could have reported 98.2% and stopped at iteration 5, and it would have been the better headline. It would also have been the exact failure this post is about.
The prompt is usually not the problem
Read the Change column again. The two biggest wins are iterations 1 and 3, and neither one touched the rubric. Both are renderer changes. They changed what the judge could see, not what it was asked.
Iteration 1 moved in-band fifteen points by raising the final-response window from 800 to 4,000 characters and tool summaries from 100 to 400. Nothing about the judge's instructions changed. We just stopped hiding the evidence from it.
Call it evidence starvation. A judge asked to verify grounding, handed 100-character tool summaries, cannot verify anything — the citation it's meant to check got truncated three fields ago. So it does what any reasonable grader does with missing evidence and a demand for a verdict: it rates good work as fabricated and incomplete. The rubric was fine. The rubric was never the problem. The judge was reading through a keyhole.
Iteration 3 is the same lesson in a different key: tool-call arguments were missing from the decision trail, so the judge could see that a tool ran but not what it was asked. That's not a rubric gap. That's a hole in the evidence.
Nearly everyone tunes the prompt first. It's the most available surface — a text file, right there, infinitely adjustable. So teams write increasingly elaborate instructions to a judge that structurally cannot see the thing they're instructing it about. Before you rewrite the rubric, check whether your judge can actually see the thing you're asking it about. Most judge quality problems are platform problems in a prompt costume.
Hash what the judge reads, not what the human reads
Two mechanisms keep the numbers honest over time, and both are about comparability — the property you destroy by accident.
Each rubric carries a content hash of its static prompt parts. Verdicts produced under different rubric text are never aggregated together. Change the rubric, get a new hash, get a new bucket. You cannot accidentally average yesterday's scores with today's under a different definition of "good," because the system won't let the two meet. Every dashboard that silently mixes verdicts across rubric versions is reporting a number that means nothing.
The nice detail: display-only help text is deliberately excluded from the hash. Improving the human-facing explanation shouldn't trigger a pointless re-judge of the corpus. Hash what the judge reads, not what the human reads.
The same instinct shows up in rubric edits. When one rubric was hardened against over-tagging, the output criteria and schema were left unchanged specifically so scores stay comparable. Improving your judge while destroying your ability to compare to last month is a bad trade — and it's one you make by accident, in a commit that felt like pure upside.
Guards keyed on what the judge already sees
When a judge over-tags "missing tool call," the instinct is to write it a lecture. Lectures are weak. A guard tied to metadata already in front of the judge is strong: emit that tag only when run metadata shows zero tool calls and the request needed external data; never emit it when the tool count is above zero.
That guard only works if its premise is true, so we verified in code that the tool count really is rendered in the block the judge reads. That step is not optional and it is routinely skipped. A guard whose premise is false is worse than no guard, because now you've got a rule that reads as rigorous and fires on vibes.
What we still can't claim
Two admissions, and they're the point.
A guard in place is not a problem fixed. Proving it requires a re-judge of the corpus. Shipping the guard and declaring victory is precisely the error the whole apparatus exists to catch — asserting a result you didn't measure, with more confidence than the evidence supports. Judges do it. So do we.
Even at temperature 0, a judge is not fully deterministic across serving providers. Perfect reproducibility isn't on the menu. Knowing your noise floor is — which is why stddev across trials is a first-class metric rather than a footnote. You can't eliminate the noise. You can refuse to mistake it for signal.
Two more places where the honest answer is a shape, not a number. Where tool output isn't captured, the grounding criterion returns unknown — not fail. A judge that can't see the evidence must say so rather than guess; absence of proof is not proof of fabrication. And a run that legitimately suspended or failed is judged on its trajectory so far, not penalized for a final answer it had no reason to produce.
The eval is the product. And the judge is part of the eval — which means every number you trust is resting on a system that, until you evaluate it, nobody has evaluated. The judge doesn't announce that it's drifting. It just keeps returning numbers, in the same confident format, forever.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.