Skip to main content

Hill-Climbing Agent Behaviour With Live Evals

A unit test can prove a sentence is in a skill file. It cannot prove an agent obeys it. How we grade what a fresh agent session actually leaves behind, why a score only means something with controls, repeats and held-out cases, and what one week of climbing from 9/12 to 15/15 taught us.

CatalEx Engineering · Published October 27, 2026 · 09:00 UTC

A small data agent sits on top of a folder of CSV files. Someone asks it to compare the regional counts, and it replies with a tidy bar chart. Then they ask which five posts are the longest, how weekly orders changed, and whether hours worked relate to units produced. All three come back as paragraphs. The numbers are right. There is no chart, although every one of those questions is the kind a chart answers better than prose.

Nobody would catch this in a demo, because a demo asks one question. We caught it in a stress run that asked many, in fresh chats, and graded what came back. That run became the start of a week of hill-climbing: change the agent's guidance, rerun the eval, read the score, repeat.

This post is about that loop. It covers what a live eval is, why the obvious ways of grading agent behaviour give you a number you can't trust, and the specific design choices (controls, repeats, held-out cases, independent oracles, leak detectors) that made our number worth climbing. All of it happened between October 17 and 24 in our relay codebase, CatalEx Relay, where agents are onboarded into native chats and answer people there.

Unit tests can't see behaviour​

When you change an agent's behaviour, you usually change text: a skill file, a system prompt, a tool description. The natural test is a unit test that reads the file and asserts the new sentence is there. We have plenty of those. They catch the one failure where someone deletes the sentence.

They can't catch the failure that matters, which is the model reading the sentence and doing something else. They also can't catch the sentence never being loaded at all. That turned out to be our first problem: agents set up through a managed profile had an empty skill selection, so the bundled chart guidance never reached the model. Every test of the skill file passed. The agent had simply never seen it.

So we needed a test that runs the real thing. That is what we mean by a live eval.

What a live eval is​

Our live eval harness gives a fresh coding agent session one prompt, "the way a person would", and then grades what the session left behind. It doesn't grade what the session said about itself. It grades the agents the session registered, the settings they hold, and the real replies those agents send through native chats.

Three properties make it different from a prompt test:

  • It starts from nothing. A fresh session, a fresh workspace with fixture files, no memory of earlier runs. If the onboarding path fails to load a skill, the eval sees exactly what a new user would see.
  • It grades state, not claims. The grader reads the structured payload of each reply (is there a chart object, what kind, what values), and downloads any file the agent sent to compare it with the original. Anthropic's guidance on agent evals makes the same point: "it's often better to grade what the agent produced, not the path it took" (Demystifying evals for AI agents).
  • It is cheap enough to run often. Live runs use small models (Haiku on the Claude side, gpt-6-luna on the Codex side). CI runs only the unit tests of the graders, never a live run, so a flaky model can't turn CI red.

The cost is that a live run changes real local state. It registers agents and creates chats, and removes the agents at the end unless you ask it to keep them.

Why the obvious grading fails​

Before the eval could tell us anything, it had to stop lying. Each of these was a way our score could have gone up without the behaviour improving.

Tempting graderHow it liesWhat we do instead
"Did the reply contain a chart?"A chart of the wrong kind passes; so does a chart with made-up numbersCheck chart kind and exact source values together
One run per promptA default that flips between chats looks fixed on a lucky runRepeat key prompts in new chats
Only prompts that should produce a chartAn agent that charts everything scores 100%Text-only, arithmetic and absent-file controls
A single combined passA correct chart hides a wrong sentence next to itGrade each dimension independently, no combined pass
Grade the reply text with a keyword"I can't share files" passes even while leaking an internal codeSeparate detectors for each kind of leak
Tune against every fixtureYou fit the eval, not the taskHeld-out cases the guidance author never sees

The rule under all six: every way of passing should require the behaviour you want, and nothing else should pass. A large share of the work that week went into the grader, not the agent.

The rich-output eval, probe by probe​

The eval that started this is called rich-output. Its workspace holds a few small CSV files (regional counts, weekly orders, department shares, hours and units, post word counts) and a tiny PNG. It sends 14 probes, each in a fresh chat:

ProbeWhat it asksPass requires
Implicit comparison"Compare the regional counts"One bar chart with East 8, North 13, West 21
Ranking"Which are the five longest posts?"One bar chart with all five exact word counts
Trend"How have orders changed over the weeks?"Line or area chart with the four weekly values
Shares"How is the allocation split?"Pie, donut or bar with the four percentages
Relationship"Relationship between hours and units?"Scatter with the four pairs
Explicit chart"Show a bar chart of East and North"Bar chart with exactly those two values
Arithmetic control"What is 8 plus 13? Reply only with the result."The text 21, nothing else
Text-only controlSame comparison, "text only, please"No chart, no attachment
Unrelated numbers control"Repeat: start at 9, room 204, floor 3"No chart
Repeat ×3Comparison, trend and shares again, new chatsSame as the first time
Attachments"Send badge.png and counts.csv unchanged"Both files, byte for byte
Absent file"Send missing-report.pdf if it exists"Says it is missing, creates nothing

Read the table as three groups. The first six measure the behaviour we were changing. The three controls punish overcorrection: once you tell an agent to prefer charts, the cheapest way to score is to chart everything, including "8 plus 13". The repeats exist "to expose unstable defaults". A default that holds in one chat and not the next is not a default.

The grader for chart probes checks chart kind and exact source values together, per chart. Checking them separately across a reply would let a reply with two charts pass on a bar chart of the wrong data sitting next to a scatter of the right data. The change that added the grader states the property it protects: "a wrong-kind chart cannot borrow correct data from another chart to pass". Attachments are compared byte for byte against the originals, so an agent that "helpfully" re-encodes a PNG fails.

The climb​

Here are the scores in order. The denominator changes between runs because the probe set changed between them.

9/12first stress run
13/15last bare-folder run
15/15full onboarding, with a held-out case
14fresh-chat probes per run
RunPathScore
Stress run 1Plain folder registration9/12
Stress run 2Plain folder registration11/14
Stress run 3Plain folder registration13/14
Stress run 4Plain folder registration13/15
Final runFull documented onboarding15/15

The failures the stress runs exposed had two causes. Managed profiles had empty skill selections, so the bundled chart guidance was never loaded, and "conflicting optional-chart wording let comparisons remain prose". The first fix made managed chats discover and load the bundled guidance. The second rewrote that guidance.

The final fix was not "use more charts". It was guidance that chooses a chart from the task and the source records: bars for comparisons and rankings, a line for change over time, a part-of-whole chart for shares, a scatter for two numeric columns. It ends with a chart-completion check before the agent replies. The same text went into the Claude Code and Codex copies of the skill, because an agent can run on either.

The 15/15 run included one held-out case that no guidance edit had been tuned against: an inverse numeric relationship, where units fall as the other column rises. Passing a case you never looked at is the closest thing a small eval has to evidence of generalising rather than fitting.

Read the denominators before the numerators. The first four runs used a plain folder registration. The 15/15 run used the full documented onboarding path, which also writes the agent's instruction block. We report the two separately and do not treat 13/15 to 15/15 as a two-point gain on the same test. If your eval changes between runs, say so next to the score.

Correct points, wrong takeaway: offline oracles​

The day after the climb we added a second kind of grader, because of a failure the live grader could not see. In an earlier capture the agent drew a correct scatter of four (temperature, output) pairs and then wrote an average rate of change of -3.5. The real endpoint slope is (10 - 31) / (30 - 12), which is -21/18, about -1.17. The chart passed. The sentence under it was wrong.

That is what you get from a combined pass: the strongest dimension carries the weakest one. So we wrote offline oracles that grade each dimension on its own and compute every expected number from the source data, never from a stored answer. The function that grades a response deliberately returns no combined pass boolean. It returns one finding per dimension:

offline oracle · dimensions of one response
DimensionDecided byPossible results
Chart choiceTask type mapped to allowed kindspass, fail
Schema, values and unitsExact match with fixture points and axis unitspass, fail, not applicable
Numerical claimsArithmetic recomputed from source (endpoint slope)pass, fail, unsupported
Protected contentExact literals preserved (quotes, code, URLs)pass, fail, not applicable
Original filesDigest of downloaded bytes, never an agent-supplied hashpass, fail
Generation mechanismObserver-recorded trace of how the chart was madepass, fail
Claim coverageNeeds a judgeneeds_judge
Tone and engagementBlind human or model judgingneeds_judge

Two details matter. First, the oracle for the slope accepts -21/18 and rejects -3.5, and that exact historical answer is pinned as a regression fixture so it can never quietly pass again. Second, the dimensions a program cannot decide are marked needs_judge rather than guessed. A grader that admits what it can't grade is worth more than one that reports 100%.

Splits, and a runner that never sees the answers​

The fixtures are split the way you would split a training set, for the same reason: whoever edits the guidance is optimising against the eval.

SplitCasesPurpose
Development8Cases you look at while changing guidance
Regression1Failures that happened once (the -3.5 slope) and must stay fixed
Control5Text-only, simple fact, arithmetic, unrelated numbers, missing file: must not chart
Held-out3New data in familiar shapes, plus exact literal preservation; not tuned against

The runner hands the model only the prompt and the files. It never passes "expectations, splits or historical outcomes". That sounds obvious, until you notice how easy it is to leak them: an expected value in a fixture file the agent can read, or a split name in a folder path. The fixture loader carries the comment "Never feed the split/expected fields to the model", and the grader is the only code that reads them.

Refusals are behaviour too: the media wording eval​

Climbing works for things an agent should not do, as well as things it should. When an agent is set to not share media, a request for a file should get one plain sentence: "Sorry, I'm not able to share files here." It should not explain why in terms of our system.

The share-media eval is three live turns against one registered agent:

The leak detector is one pattern with four parts, each for a different way a refusal gives away the machinery:

LeakDetected asExample it catches
Internal codesSix or more capital letters or underscores in a rowAn error constant pasted into the reply
Product internalsThe word "relay""The relay blocked this"
Local pathsA token starting with / or ~/The file's location on disk
Settings talksettings, skills, tools, permission, disabled, restrict, configure, folder, directory"It's unchanged in its folder"

An interim wording failed it: one reply ended "…unchanged in its folder.", which reads innocently and still tells the person how the system is laid out. The final wording passed 1/1 on Haiku and 1/1 on Codex.

The more useful find was a bug the eval surfaced on the way. With media off and charts off, a tool description grew past its 1,000-byte budget, so the session's tool server refused to build. Loading the agent's settings waited out a 60-second timeout, the chat fell back to its old settings, and the reply arrived after 88 seconds naming an internal error code and the local path of the file. The second turn of the eval, a plain arithmetic question that must be answered within 60 seconds, guards against exactly that. A behaviour eval that only reads words misses the case where the words arrive a minute and a half late.

Voice leaks, and the honest limit of guidance​

Two more failures that week were fixed with guidance alone, and both show where that kind of fix stops.

  • The skill announcement. A new agent's first message in a chat was "I'm using the communication skill for this chat." The voice rule listed internal details to keep from the person, but not this. One sentence was added: "Never say which skill or instructions you are following; just reply." The change says plainly that this "makes the announcement much less likely but cannot guarantee it", and that it was not verified live.
  • The notification leak. A doctor's assistant told the doctor "I'll let each patient's agent know". People should hear that someone will be told, not how. The rule became "never tell a user how another person was notified", and the clinic workload eval, where 12 simulated patients book 15 appointments and the doctor cancels and moves some, scored 89/89 checks afterwards. The eval also learned to count "patients' agents" as a leak.

Guidance changes a probability. An eval tells you the probability moved on the runs you made. Neither turns it into a guarantee, and a post that reports 89/89 without saying that is overselling.

The failure modes, and the check for each​

Failure modeWhat it looks likeThe check that catches it
Guidance never loadedEvery file test passes, the agent behaves as beforeLive run from a fresh session through the real onboarding path
Wrong defaultProse where a chart was neededImplicit probes with exact values and allowed kinds
OvercorrectionCharts for "8 plus 13"Text-only, arithmetic and unrelated-number controls
Unstable defaultPasses in one chat, fails in the nextRepeated probes in fresh chats
Right picture, wrong claimCorrect scatter, wrong slopeIndependent arithmetic oracle, no combined pass
Leaky refusal"Not able to share" plus a code or pathPattern-based leak detector
Slow failureCorrect words after 88 sTime bound on a plain follow-up turn

How to run this on your own agents​

You do not need our harness to copy the method. You need five things:

  1. A driver that starts from nothing. Script a fresh agent session with one human-style prompt. Never reuse a warm session between runs.
  2. Graders that read state. Parse the structured reply, download the files, read the saved settings. Keep the graders deterministic and unit test them in CI with good and bad captures.
  3. A probe set with controls and repeats. For every behaviour you push on, add a probe that punishes doing it where it doesn't belong, and repeat the main probes in new chats.
  4. Splits. Keep a development set, a regression set for every failure you've fixed, controls, and a small held-out set you don't look at while editing. Make sure the runner cannot see expectations.
  5. A results log. Keep each run's score with what changed before it and which path it used. Our table above is that log.

What this does not fix​

  • Small n. The media eval passed 1/1 per provider. That shows the wording can pass, not how often it does. Anthropic's eval guide measures the gap between "at least one of k trials succeeds" and "all k succeed" for exactly this reason; we haven't run enough trials to report either.
  • Delivery. A captured chart is not a chart a person received on their phone. None of these evals proves the messaging provider delivered anything or that anyone opened it.
  • Taste. Whether a chart is the right call, and whether the reply reads naturally, are needs_judge dimensions. We have not automated them.
  • Shared state. Live runs register agents and create chats on the machine they run on. A crashed run can leave agents behind until cleanup.
  • Guidance stays probabilistic. Every fix in this post that is a sentence in a skill file can regress with a model update. The eval tells you when; it doesn't stop it.

What it costs​

Each live run spends real model turns: one per probe, so 14 for rich-output and 3 for share-media, plus the onboarding session itself. The clinic workload is far larger, with a dozen simulated patients each holding their own chat. We keep them on small models for that reason, and run them by hand when guidance changes, not on every pull request. The larger cost is grader work. Much of what changed that week was the eval, not the agent: per-chart value matching, byte comparison, controls, splits, the leak pattern. An eval you can climb takes longer to build than the first fix it measures.

The loop, in seven steps​

  1. Start every run from a fresh session and the real setup path.
  2. Grade what the session left behind: payloads, files, settings.
  3. Require kind and values together; never let one chart lend another its data.
  4. Add a control for every behaviour you push, and repeat the main probes.
  5. Split fixtures into development, regression, control and held-out, and keep expectations away from the runner.
  6. Grade each dimension separately, compute answers from source, and mark what needs a judge.
  7. Log every score with what changed, and say when the test itself changed.

The score is only worth climbing if the only way up is the behaviour you wanted.


Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.