Hill-Climbing Agent Behaviour With Live Evals
A unit test can prove a sentence is in a skill file. It cannot prove an agent obeys it. How we grade what a fresh agent session actually leaves behind, why a score only means something with controls, repeats and held-out cases, and what one week of climbing from 9/12 to 15/15 taught us.
A small data agent sits on top of a folder of CSV files. Someone asks it to compare the regional counts, and it replies with a tidy bar chart. Then they ask which five posts are the longest, how weekly orders changed, and whether hours worked relate to units produced. All three come back as paragraphs. The numbers are right. There is no chart, although every one of those questions is the kind a chart answers better than prose.
Nobody would catch this in a demo, because a demo asks one question. We caught it in a stress run that asked many, in fresh chats, and graded what came back. That run became the start of a week of hill-climbing: change the agent's guidance, rerun the eval, read the score, repeat.
This post is about that loop. It covers what a live eval is, why the obvious ways of grading agent behaviour give you a number you can't trust, and the specific design choices (controls, repeats, held-out cases, independent oracles, leak detectors) that made our number worth climbing. All of it happened between October 17 and 24 in our relay codebase, CatalEx Relay, where agents are onboarded into native chats and answer people there.
Unit tests can't see behaviour
When you change an agent's behaviour, you usually change text: a skill file, a system prompt, a tool description. The natural test is a unit test that reads the file and asserts the new sentence is there. We have plenty of those. They catch the one failure where someone deletes the sentence.
They can't catch the failure that matters, which is the model reading the sentence and doing something else. They also can't catch the sentence never being loaded at all. That turned out to be our first problem: agents set up through a managed profile had an empty skill selection, so the bundled chart guidance never reached the model. Every test of the skill file passed. The agent had simply never seen it.
So we needed a test that runs the real thing. That is what we mean by a live eval.
What a live eval is
Our live eval harness gives a fresh coding agent session one prompt, "the way a person would", and then grades what the session left behind. It doesn't grade what the session said about itself. It grades the agents the session registered, the settings they hold, and the real replies those agents send through native chats.
Three properties make it different from a prompt test:
- It starts from nothing. A fresh session, a fresh workspace with fixture files, no memory of earlier runs. If the onboarding path fails to load a skill, the eval sees exactly what a new user would see.
- It grades state, not claims. The grader reads the structured payload of each reply (is there a chart object, what kind, what values), and downloads any file the agent sent to compare it with the original. Anthropic's guidance on agent evals makes the same point: "it's often better to grade what the agent produced, not the path it took" (Demystifying evals for AI agents).
- It is cheap enough to run often. Live runs use small models (Haiku on the Claude side, gpt-6-luna on the Codex side). CI runs only the unit tests of the graders, never a live run, so a flaky model can't turn CI red.
The cost is that a live run changes real local state. It registers agents and creates chats, and removes the agents at the end unless you ask it to keep them.
Why the obvious grading fails
Before the eval could tell us anything, it had to stop lying. Each of these was a way our score could have gone up without the behaviour improving.
| Tempting grader | How it lies | What we do instead |
|---|---|---|
| "Did the reply contain a chart?" | A chart of the wrong kind passes; so does a chart with made-up numbers | Check chart kind and exact source values together |
| One run per prompt | A default that flips between chats looks fixed on a lucky run | Repeat key prompts in new chats |
| Only prompts that should produce a chart | An agent that charts everything scores 100% | Text-only, arithmetic and absent-file controls |
| A single combined pass | A correct chart hides a wrong sentence next to it | Grade each dimension independently, no combined pass |
| Grade the reply text with a keyword | "I can't share files" passes even while leaking an internal code | Separate detectors for each kind of leak |
| Tune against every fixture | You fit the eval, not the task | Held-out cases the guidance author never sees |
The rule under all six: every way of passing should require the behaviour you want, and nothing else should pass. A large share of the work that week went into the grader, not the agent.
The rich-output eval, probe by probe
The eval that started this is called rich-output. Its workspace holds a few
small CSV files (regional counts, weekly orders, department shares, hours and
units, post word counts) and a tiny PNG. It sends 14 probes, each in a fresh
chat:
| Probe | What it asks | Pass requires |
|---|---|---|
| Implicit comparison | "Compare the regional counts" | One bar chart with East 8, North 13, West 21 |
| Ranking | "Which are the five longest posts?" | One bar chart with all five exact word counts |
| Trend | "How have orders changed over the weeks?" | Line or area chart with the four weekly values |
| Shares | "How is the allocation split?" | Pie, donut or bar with the four percentages |
| Relationship | "Relationship between hours and units?" | Scatter with the four pairs |
| Explicit chart | "Show a bar chart of East and North" | Bar chart with exactly those two values |
| Arithmetic control | "What is 8 plus 13? Reply only with the result." | The text 21, nothing else |
| Text-only control | Same comparison, "text only, please" | No chart, no attachment |
| Unrelated numbers control | "Repeat: start at 9, room 204, floor 3" | No chart |
| Repeat ×3 | Comparison, trend and shares again, new chats | Same as the first time |
| Attachments | "Send badge.png and counts.csv unchanged" | Both files, byte for byte |
| Absent file | "Send missing-report.pdf if it exists" | Says it is missing, creates nothing |
Read the table as three groups. The first six measure the behaviour we were changing. The three controls punish overcorrection: once you tell an agent to prefer charts, the cheapest way to score is to chart everything, including "8 plus 13". The repeats exist "to expose unstable defaults". A default that holds in one chat and not the next is not a default.
The grader for chart probes checks chart kind and exact source values together, per chart. Checking them separately across a reply would let a reply with two charts pass on a bar chart of the wrong data sitting next to a scatter of the right data. The change that added the grader states the property it protects: "a wrong-kind chart cannot borrow correct data from another chart to pass". Attachments are compared byte for byte against the originals, so an agent that "helpfully" re-encodes a PNG fails.
The climb
Here are the scores in order. The denominator changes between runs because the probe set changed between them.
| Run | Path | Score |
|---|---|---|
| Stress run 1 | Plain folder registration | 9/12 |
| Stress run 2 | Plain folder registration | 11/14 |
| Stress run 3 | Plain folder registration | 13/14 |
| Stress run 4 | Plain folder registration | 13/15 |
| Final run | Full documented onboarding | 15/15 |
The failures the stress runs exposed had two causes. Managed profiles had empty skill selections, so the bundled chart guidance was never loaded, and "conflicting optional-chart wording let comparisons remain prose". The first fix made managed chats discover and load the bundled guidance. The second rewrote that guidance.
The final fix was not "use more charts". It was guidance that chooses a chart from the task and the source records: bars for comparisons and rankings, a line for change over time, a part-of-whole chart for shares, a scatter for two numeric columns. It ends with a chart-completion check before the agent replies. The same text went into the Claude Code and Codex copies of the skill, because an agent can run on either.
The 15/15 run included one held-out case that no guidance edit had been tuned against: an inverse numeric relationship, where units fall as the other column rises. Passing a case you never looked at is the closest thing a small eval has to evidence of generalising rather than fitting.
Read the denominators before the numerators. The first four runs used a plain folder registration. The 15/15 run used the full documented onboarding path, which also writes the agent's instruction block. We report the two separately and do not treat 13/15 to 15/15 as a two-point gain on the same test. If your eval changes between runs, say so next to the score.
Correct points, wrong takeaway: offline oracles
The day after the climb we added a second kind of grader, because of a failure the live grader could not see. In an earlier capture the agent drew a correct scatter of four (temperature, output) pairs and then wrote an average rate of change of -3.5. The real endpoint slope is (10 - 31) / (30 - 12), which is -21/18, about -1.17. The chart passed. The sentence under it was wrong.
That is what you get from a combined pass: the strongest dimension carries the weakest one. So we wrote offline oracles that grade each dimension on its own and compute every expected number from the source data, never from a stored answer. The function that grades a response deliberately returns no combined pass boolean. It returns one finding per dimension:
| Dimension | Decided by | Possible results |
|---|---|---|
| Chart choice | Task type mapped to allowed kinds | pass, fail |
| Schema, values and units | Exact match with fixture points and axis units | pass, fail, not applicable |
| Numerical claims | Arithmetic recomputed from source (endpoint slope) | pass, fail, unsupported |
| Protected content | Exact literals preserved (quotes, code, URLs) | pass, fail, not applicable |
| Original files | Digest of downloaded bytes, never an agent-supplied hash | pass, fail |
| Generation mechanism | Observer-recorded trace of how the chart was made | pass, fail |
| Claim coverage | Needs a judge | needs_judge |
| Tone and engagement | Blind human or model judging | needs_judge |
Two details matter. First, the oracle for the slope accepts -21/18 and
rejects -3.5, and that exact historical answer is pinned as a regression
fixture so it can never quietly pass again. Second, the dimensions a program
cannot decide are marked needs_judge rather than guessed. A grader that
admits what it can't grade is worth more than one that reports 100%.
Splits, and a runner that never sees the answers
The fixtures are split the way you would split a training set, for the same reason: whoever edits the guidance is optimising against the eval.
| Split | Cases | Purpose |
|---|---|---|
| Development | 8 | Cases you look at while changing guidance |
| Regression | 1 | Failures that happened once (the -3.5 slope) and must stay fixed |
| Control | 5 | Text-only, simple fact, arithmetic, unrelated numbers, missing file: must not chart |
| Held-out | 3 | New data in familiar shapes, plus exact literal preservation; not tuned against |
The runner hands the model only the prompt and the files. It never passes "expectations, splits or historical outcomes". That sounds obvious, until you notice how easy it is to leak them: an expected value in a fixture file the agent can read, or a split name in a folder path. The fixture loader carries the comment "Never feed the split/expected fields to the model", and the grader is the only code that reads them.
Refusals are behaviour too: the media wording eval
Climbing works for things an agent should not do, as well as things it should. When an agent is set to not share media, a request for a file should get one plain sentence: "Sorry, I'm not able to share files here." It should not explain why in terms of our system.
The share-media eval is three live turns against one registered agent:
The leak detector is one pattern with four parts, each for a different way a refusal gives away the machinery:
| Leak | Detected as | Example it catches |
|---|---|---|
| Internal codes | Six or more capital letters or underscores in a row | An error constant pasted into the reply |
| Product internals | The word "relay" | "The relay blocked this" |
| Local paths | A token starting with / or ~/ | The file's location on disk |
| Settings talk | settings, skills, tools, permission, disabled, restrict, configure, folder, directory | "It's unchanged in its folder" |
An interim wording failed it: one reply ended "…unchanged in its folder.", which reads innocently and still tells the person how the system is laid out. The final wording passed 1/1 on Haiku and 1/1 on Codex.
The more useful find was a bug the eval surfaced on the way. With media off and charts off, a tool description grew past its 1,000-byte budget, so the session's tool server refused to build. Loading the agent's settings waited out a 60-second timeout, the chat fell back to its old settings, and the reply arrived after 88 seconds naming an internal error code and the local path of the file. The second turn of the eval, a plain arithmetic question that must be answered within 60 seconds, guards against exactly that. A behaviour eval that only reads words misses the case where the words arrive a minute and a half late.
Voice leaks, and the honest limit of guidance
Two more failures that week were fixed with guidance alone, and both show where that kind of fix stops.
- The skill announcement. A new agent's first message in a chat was "I'm using the communication skill for this chat." The voice rule listed internal details to keep from the person, but not this. One sentence was added: "Never say which skill or instructions you are following; just reply." The change says plainly that this "makes the announcement much less likely but cannot guarantee it", and that it was not verified live.
- The notification leak. A doctor's assistant told the doctor "I'll let each patient's agent know". People should hear that someone will be told, not how. The rule became "never tell a user how another person was notified", and the clinic workload eval, where 12 simulated patients book 15 appointments and the doctor cancels and moves some, scored 89/89 checks afterwards. The eval also learned to count "patients' agents" as a leak.
Guidance changes a probability. An eval tells you the probability moved on the runs you made. Neither turns it into a guarantee, and a post that reports 89/89 without saying that is overselling.
The failure modes, and the check for each
| Failure mode | What it looks like | The check that catches it |
|---|---|---|
| Guidance never loaded | Every file test passes, the agent behaves as before | Live run from a fresh session through the real onboarding path |
| Wrong default | Prose where a chart was needed | Implicit probes with exact values and allowed kinds |
| Overcorrection | Charts for "8 plus 13" | Text-only, arithmetic and unrelated-number controls |
| Unstable default | Passes in one chat, fails in the next | Repeated probes in fresh chats |
| Right picture, wrong claim | Correct scatter, wrong slope | Independent arithmetic oracle, no combined pass |
| Leaky refusal | "Not able to share" plus a code or path | Pattern-based leak detector |
| Slow failure | Correct words after 88 s | Time bound on a plain follow-up turn |
How to run this on your own agents
You do not need our harness to copy the method. You need five things:
- A driver that starts from nothing. Script a fresh agent session with one human-style prompt. Never reuse a warm session between runs.
- Graders that read state. Parse the structured reply, download the files, read the saved settings. Keep the graders deterministic and unit test them in CI with good and bad captures.
- A probe set with controls and repeats. For every behaviour you push on, add a probe that punishes doing it where it doesn't belong, and repeat the main probes in new chats.
- Splits. Keep a development set, a regression set for every failure you've fixed, controls, and a small held-out set you don't look at while editing. Make sure the runner cannot see expectations.
- A results log. Keep each run's score with what changed before it and which path it used. Our table above is that log.
What this does not fix
- Small n. The media eval passed 1/1 per provider. That shows the wording can pass, not how often it does. Anthropic's eval guide measures the gap between "at least one of k trials succeeds" and "all k succeed" for exactly this reason; we haven't run enough trials to report either.
- Delivery. A captured chart is not a chart a person received on their phone. None of these evals proves the messaging provider delivered anything or that anyone opened it.
- Taste. Whether a chart is the right call, and whether the reply reads
naturally, are
needs_judgedimensions. We have not automated them. - Shared state. Live runs register agents and create chats on the machine they run on. A crashed run can leave agents behind until cleanup.
- Guidance stays probabilistic. Every fix in this post that is a sentence in a skill file can regress with a model update. The eval tells you when; it doesn't stop it.
What it costs
Each live run spends real model turns: one per probe, so 14 for rich-output and 3 for
share-media, plus the onboarding session itself. The clinic workload is far
larger, with a dozen simulated patients each holding their own chat. We keep them on
small models for that reason, and run them by hand when guidance changes, not
on every pull request. The larger cost is grader work. Much of what changed that
week was the eval, not the agent: per-chart value matching, byte comparison,
controls, splits, the leak pattern. An eval
you can climb takes longer to build than the first fix it measures.
The loop, in seven steps
- Start every run from a fresh session and the real setup path.
- Grade what the session left behind: payloads, files, settings.
- Require kind and values together; never let one chart lend another its data.
- Add a control for every behaviour you push, and repeat the main probes.
- Split fixtures into development, regression, control and held-out, and keep expectations away from the runner.
- Grade each dimension separately, compute answers from source, and mark what needs a judge.
- Log every score with what changed, and say when the test itself changed.
The score is only worth climbing if the only way up is the behaviour you wanted.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.