Every AI platform says it is model-agnostic. Almost none of them are. Not because the teams are lying — because agnosticism isn't a property you declare in a README. It's a seam in the code, and a seam that nothing routinely crosses is a seam that has already rusted shut. You just don't find out until the day you try to move it.

We ended an earlier post by saying the model is a component you swap. That's the easy half. The hard half is the follow-up question every engineer asks immediately: fine, how? What does the code have to look like for that sentence to be true a year from now, under a team that has shipped a hundred features since anyone last thought about it?

Here is the test we've landed on, and it's blunter than most architecture advice. A swappable dependency is one where something in your normal workflow already swaps it. Not a test that could be written. Not an adapter that could be added. Something that crosses the seam today, on a Tuesday, without anyone deciding to. If the only time you'd exercise the seam is the day you need it, you don't have a seam. You have a plan. Plans don't compile.

This post walks four seams in our system, what keeps each one alive, and — for one of them — an admission about where the architecture is ahead of the deployment.

Seam 1: the model behind an alias​

All inference goes through a model proxy. Nothing in our application code requests a vendor model by name. Callers request an alias: a default alias, and a cheaper-tier alias for work that doesn't need the good stuff. The alias resolves to a real model in configuration, not in code.

That much is unremarkable — it's the indirection everyone writes. The part that makes it real is the routing table. Local development routes the same alias to a locally-run model while deployed environments route it somewhere else entirely. Every engineer, every day, on every branch, runs the system with the alias pointed at a genuinely different provider than production uses. Nobody has to remember to test the indirection. You cannot get your work done without exercising it.

That's the whole difference between a seam and a wish. The indirection isn't kept honest by discipline or by a lint rule. It's kept honest by the fact that the local loop would break the instant someone hard-coded a model, and it would break for the person who did it, in the first minute.

The consequence shows up in the schema. There is no model column on an agent. An agent is a definition — instructions, tools, KPIs — and deliberately not a model binding. Model choice is a service-level concern, resolved at dispatch, not a property an agent carries around. The reasoning is mechanical rather than philosophical: if every agent row named a model, "swap the model" would be a data migration across customer rows instead of a config change. The seam would technically exist and be practically unusable, which is the most expensive kind of abstraction to own.

Seam 2: the sandbox behind a Protocol​

Agents run code in a box. The orchestrator has never met one. It depends only on a Protocol — an interface — and three adapters satisfy it: a fake provider used locally and in CI, a microVM provider that is the real one, and a confidential-computing provider aimed at stricter deployments.

That third one deserves a flag before it does any work in this argument: it is aspirational. The Protocol is written for it. The adapter is not built. We are not shipping confidential computing today, and it appears here only because of what it did to the interface.

Which was this: the Protocol was designed against the harder provider so all three satisfy one contract. Write your interface against the most constrained implementation — including one that doesn't exist yet — and the easy cases fall out of it for free. Write it against the easy case and the hard one will not fit later, no matter how much you want it to. The unbuilt adapter earned its keep by being a constraint during design, which is a legitimate use for a thing that doesn't exist.

The mechanism that keeps it from degrading is capability declaration. Adapters advertise what they offer; the orchestrator branches on the capability, never on the provider:

@dataclass(frozen=True)
class Capabilities:
    sudo: bool           # can the agent take root inside the box?
    scratch_disk: bool   # writable disk beyond the process?
    resumable: bool      # can a box be suspended and woken later?


class SandboxProvider(Protocol):
    """A place an agent can run. Nothing above this line knows which one."""

    def capabilities(self) -> Capabilities: ...

    async def create(self, spec: BoxSpec) -> BoxHandle: ...
    async def exec(self, box: BoxHandle, cmd: Command) -> ExecResult: ...
    async def destroy(self, box: BoxHandle) -> None: ...

if provider == "x" is how a seam dies. It always starts as one line, in one place, for one good reason, and then it's forty lines in nine files and the abstraction is now decoration around a switch statement. Branching on caps.resumable instead means a new adapter answers a question rather than joining a conditional.

The Protocol also documents what must not cross it. For every adapter, the forbidden leaks are written down: the box's only external egress is the tool gate and the model proxy; tool results follow the preview-plus-reference contract; delivery and box lifecycle stay behind the interface. Naming what must not pass through an interface is rarer than naming what must, and it's more useful — leaks are how abstractions rot, and they rot from the inside, by addition, one convenient exception at a time.

And the fake provider deserves its own beat. The entire system runs in CI with no external sandbox at all. That's not a testing convenience; it's the strongest available proof the seam holds. An abstraction you can't run without is not an abstraction — it's a wrapper. Every green CI run is a swap of the most infrastructure-heavy dependency we have, performed by nobody, hundreds of times a week.

Seam 3: integrations behind a uniform shape​

Two sourcing paths exist for third-party tools: sometimes we wrap a vendor's API ourselves, sometimes we proxy a remote server someone else runs. These are peer-equal behind the same registry shape, and the line that matters is that agents can't tell them apart at call time. The agent-facing contract doesn't leak the sourcing decision — which means the sourcing decision can be revisited without touching a single agent definition.

The registry is convention, not registration. Every vendor is a directory whose name is its slug, with a fixed file layout inside; discovery walks one level down and finds it. Adding a vendor is "write a new package + decorate", not "edit a shared elif chain". A central registry file is two bad things at once: a merge conflict and a coupling point. A naming convention is neither, and it can't drift from itself.

Metering follows the same instinct. A tool is metered if its canonical name isn't in the local-tools set — vendor-agnostic by construction, because there is no per-vendor list anywhere for anyone to forget to update. A canonicalizer collapses the two surface spellings of the same tool into one identity, so the accounting doesn't care which call style the model happened to use. And the tool catalog itself is data — in the database, read at runtime — not code. The general shape: when correctness depends on a list staying current, delete the list.

Seam 4: the seam has to move while traffic flows​

The last one is smaller and travels further. A dispatch-shape change shipped consumer-first: the client was taught to speak both the old and the new response shapes before any environment switched. Once the consumer is bilingual, the two deploys can land in either order, and nothing is holding its breath.

An interface change is two deploys, and the consumer goes first. That's the entire technique. It's worth saying out loud because the seam metaphor is usually static — a line in the architecture — when in practice a seam is also a thing you must be able to move without stopping the world. A boundary you can only change during a maintenance window is a boundary you'll stop changing.

The four seams, and how each one would rust​

Read the third column first. That's the only one that's load-bearing. The first two columns describe every model-agnostic architecture diagram ever drawn, including the ones belonging to teams who'd need three weeks to change a model.

What this actually costs​

Model-agnosticism is not a property you announce. It's a maintenance cost you agree to pay, forever, in exchange for never being trapped. You pay it in a Protocol that's harder to write than the concrete class would have been. You pay it in a fake provider that must be kept faithful. You pay it in a capability flag where a provider check would have taken ten seconds. You pay it on days when nothing is being swapped, which is most days, which is the point.

The way to know whether you're actually paying it is simple, and you can run it on your own system this afternoon: name the last time something crossed the seam. Not a plan to. Not a test that would. An actual crossing, with a date.

If you can't name one, the seam is decorative. The model isn't a component you swap — it's a component you married.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.

A chatbot that says something wrong is embarrassing. An agent that sends something wrong is a fact about the world. The moment an agent can send, post, create, or delete, it stops being a text generator and becomes a principal in your system — with credentials and reach, taking actions other people receive.

The instinct at that point is to write a better system prompt. Tell the model to be careful. Tell it to confirm before sending. This doesn't work, and it doesn't fail in an exotic way — it fails because a system prompt is a suggestion addressed to the thing you're trying to constrain.

Permissions for agents are an ordinary authorization problem. Same shape as any other: a caller, a resource, an action, a decision. But the caller has two properties you don't usually design for. It is non-deterministic — the same input can produce a different tool call tomorrow. And it is persuadable — the content it reads can change what it decides to do. A retrieved document, a message body, a web page: all of it is input, and any of it can argue.

So you design accordingly. The rest of this is where the gate goes, and how it fails.

One gate, and the executor is dumb​

The tool gate is the sole approval authority. Not the primary one — the sole one. The sandbox where the agent's code actually runs is a pure executor, and the design intent is written down in exactly those terms:

The box is a PURE EXECUTOR: every piece of tool METADATA (catalog, enabled bindings, allowed accounts) is resolved run-side. The box NEVER calls the gate to discover tools — it has no whole-catalog access and cannot list. Its tools are therefore fixed at creation/dispatch time.

The load-bearing word is discover. An executor that can enumerate its own capabilities can be talked into enumerating more of them — a persuadable caller plus a listable catalog is a negotiation, and you've handed the model the agenda. Fixing the toolset at dispatch and denying the box any catalog access means the blast radius is decided before the model gets a turn. Whatever the run argues for later, it argues from a fixed hand.

The second half of that principle matters just as much: two components both making authorization decisions is not defense in depth. It's two sources of truth, and one of them will be stale. Depth is layers that narrow — a gate, then a scope, then an audit. Two peers independently answering "is this allowed?" isn't depth; it's a race where the wrong answer only has to win once.

Fail closed, at the bottom of the function​

Every permission resolver is a cascade. Check the user's setting. Check the binding. Check the catalog. Check the tier above. And then, at the bottom of the function, after every rule has declined to match, there is a return — and that return is your actual security posture. Everything above it is decoration.

Ours ends with an explicit default: manual approval required. Not "allow, because we didn't find a rule against it." Require a human, because we didn't find a rule.

# Nothing matched: no user override, no binding rule, no catalog entry.
# This is not "we found nothing to object to" — it is "we know nothing."
# An unknown action is exactly the case where a human should look.
return GuardrailDecision(
    requires_approval=True,
    tier="default",
    reason="no matching rule; unknown actions require manual approval",
)

The failure mode this prevents is quiet. A new toolkit gets added, nobody writes a rule, and a permissive default means it's live and unsupervised on day one — not because anyone decided that, but because a fallthrough decided it. Write the default branch first, and the rest of the cascade can only ever loosen from a safe floor.

Allowlist reads, gate everything else​

The tempting move is a blocklist: enumerate the dangerous actions and gate those. This is an infinite, ever-growing set, maintained by hand, always one integration behind. You lose that race permanently on the first day you run it.

So we inverted it. The platform seeds an explicit auto-allowlist of 2,298 read actions across 86 toolkits, derived mechanically from the integration catalog's own schemas. Anything not on the list is manual by default. The list grows when the catalog grows, by the same mechanism, with no committee.

Take a mail toolkit as the concrete case. Exactly 27 actions are auto-allowed — every one of them a read. Fetch a message. List drafts. Read a profile. List history. Zero send actions. Zero create, reply, or delete. An agent that decides to send mail surfaces an approval, always, unless the user has explicitly chosen otherwise for themselves.

The rule underneath the table is the useful part: reads are recoverable, writes are not. That asymmetry — not a threat model, not a severity score — is the cleanest line available, and the only one we've found that a machine can draw from a schema instead of a human drawing it from judgment in a meeting. Severity scores need someone to assign them and someone else to disagree. "Does this mutate anything?" is a property of the API.

Permissions are personal​

Guardrails resolve from the invoking user's own settings plus the catalog. Nothing another person can edit changes your approval behavior.

This sounds like a small storage decision. It's actually the whole multi-tenant argument. A shared "this agent is trusted" flag means one teammate's convenience silently becomes everyone's standing exposure — including people who never saw the toggle and have no reason to suspect their approvals stopped firing. "I trust this agent" is a statement about you. It has to be stored against you, or it isn't the statement you thought you were making.

Configuration history is append-only. A change soft-deletes the old row and inserts a new version, so every past configuration stays queryable — you can ask what the rules were at any moment, not just what they are now. And the gate logs each decision with the tier that won it. That detail earns its keep: "the agent was allowed" is not an answer to any question worth asking. "The agent was allowed by this rule, set by this person, at this time" is.

Approval state must not lie to the retry layer​

Here's the one that took real thought.

Tool calls carry idempotency keys, so retries are safe. Standard. But an approval gate returns HTTP 200 with no error — and it is not an executed call:

An approval_required GATE response is HTTP 200 with no error key but is NOT an executed call — the tool was suspended for the user's decision and never ran. Recording it would dedup the REAL execution on the resume re-invoke (the box presents the SAME idempotency key after approval), so the approved tool would return the "already ran" duplicate guidance with no result.

Read that failure through: a user approves an action, the resume arrives with the same idempotency key, the store says "seen it," and the tool the human explicitly approved never runs — while everything reports success. The gate would have converted an approval into a silent no-op.

So the idempotency store records only completed calls. A gate is a pause, not an outcome. Approval ids are deterministic — {run_id}:{tool_call_id} — so the same decision resolves the same call across a resume. The general lesson: any "we've seen this" cache needs a crisp definition of seen, and "returned 200" is not it. Suspension isn't completion.

Write your errors for the model, because the model is the client​

When a genuine duplicate is detected, the response is HTTP 200 carrying prose. Not a 409. Not an error code. And the reason is precise: a raw error would read as a transient failure the model might blindly retry.

DUPLICATE TOOL CALL — this exact call already completed on an earlier attempt;
your previous response was most likely lost in transit. The call ran exactly
once and was NOT lost. If this tool WRITES or takes an action (send, create,
update, delete, post, pay, …) it is already done — do NOT call it again. If it
is a READ / query and you still need the data, issue a NEW call to fetch it
again...

This generalizes further than it looks. When your caller is a language model, your API's error strings are prompt engineering. A 409 is a token sequence whose training-data association is transient conflict, back off, try again — so it teaches the model to retry the send. A sentence teaches it what happened. Status codes were designed for clients that parse; you now have a client that reads.

What we don't have​

Approval is binary per action: auto or manual. There's no "draft for approval" tier, no read/draft/send gradient. That's a real limit, and it's worth defending honestly rather than pretending it's finished design: a binary gate is legible and enforceable — you can describe it in one sentence and verify it in one query. A gradient is more expressive and much easier to get subtly wrong. A permission system that's subtly wrong is worse than one that's coarsely right, because you'll believe it.

Approval scopes are defined for both a single call and a whole session. Only the single-call scope is wired today.

And the one nobody wants to say out loud: an approval gate's real enemy isn't bypass, it's fatigue. A gate that fires on everything trains people to click approve without reading — worse than no gate at all, because it manufactures the appearance of oversight. You get the audit trail, the sign-off, the timestamp, and none of the thinking. That is the actual argument for allowlisting reads. Not convenience. Preserving the meaning of the clicks that remain.

The boundary lives outside the loop​

An agent that can act needs a boundary that exists outside its own reasoning. Not a stronger instruction, not a more emphatic prompt, not a self-check the model performs on itself — those all live inside the thing you're trying to constrain, and a persuadable caller auditing its own persuasion is not a control.

The model can be talked into things. That's not a defect; it's most of what makes it useful. A deny-by-default gate that fails closed at the bottom of the function, allowlists only what's recoverable, resolves against the person who actually asked, and logs who allowed what and when — that cannot be talked into anything. It doesn't listen. That's the feature.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.

We argued that you can't ship what you can't measure. Here's the uncomfortable half of that sentence: your measurement is a model too, and it is also wrong sometimes.

We score agent runs with an LLM judge — it reads a run and grades it against the KPIs that agent was given. That judge is treated, by everyone downstream of it, as an instrument. A thermometer. Numbers come out; people make decisions. But a judge is not a thermometer. It's a second fallible system wearing a lab coat, and every number your team trusts is resting on it.

So we built an eval of the judge. An LLM-as-judge eval of an LLM-as-judge. This post is what it found — including the part where the judge was starving for evidence and we nearly rewrote the rubric instead.

The eval has to drive the real thing​

The first design decision is the one most people get wrong, and it's not subtle: the eval drives the exact production path. Same prompt file, same input renderer, same parser, same model alias.

That sounds like hygiene. It's load-bearing. The moment your eval builds its own tidy little prompt and calls the model directly, it is measuring a system that does not exist. Every divergence — a renderer your eval skips, a parser it reimplements more forgivingly — is a place where the eval is green and production is not. You didn't evaluate your judge. You evaluated a fictional cousin of it that happens to share a rubric.

Thirteen ways a judge goes wrong​

The suite started at 17 synthetic scenarios across 13 failure families, plus 7 pairwise ordering rules. It later grew to 24 scenarios and 9 orderings. Real-run fixtures with hand-labeled score bands sit alongside the synthetics, because synthetic scenarios test the failure you imagined and real runs test the ones you didn't.

The failure families are the part worth stealing. None of them are obvious until a judge has burned you:

Calibration — does the score land in the band a human would give? Correctness-trap — a confident, articulate, wrong answer must score near zero. This is the one that separates a judge from a style critic. Sycophancy — the agent claims success; the tool results say it failed. Who does the judge believe? A judge that takes the agent's word for it is a compliance machine, not a grader. Grounding — fabricated citations. Injection — instructions embedded inside tool results, aimed at the judge. Your judge reads attacker-influenced text by design; treat it that way. Estimation — unmeasurable KPIs like "time saved." Verbosity bias — does length buy score? Relevance — beautiful output, wrong topic. Truncation — a display artifact the judge must not read as incomplete work.

Plus over-penalty, trajectory, structural load (eight KPIs at once), rubric anchoring, and autonomy. Thirteen families, because a judge doesn't fail in one direction — it fails in whichever direction your rubric was quietest about.

Orderings: the metric that survives your rubric edits​

If you take one thing from this post, take this one.

Most judge metrics are absolute. In-band rate asks whether a score landed where a human would put it. That's useful and it is fragile — it drifts with every rubric edit, every anchor you reword, every criterion you add. Tighten the rubric on Tuesday and Monday's calibration is partly fiction.

Orderings don't drift. An ordering is a pairwise constraint: run A must score above run B. Not by how much. Just above. Whatever your absolute numbers are doing, whatever the anchors say this week, the thorough grounded run outranks the confident fabricated one — or your judge is broken in a way no amount of decimal precision will save.

This is a calibration-free signal, and that property is the whole point. It survives rubric changes, prompt rewrites, and model swaps, because it asks the judge to do the thing you actually need it to do: distinguish better work from worse work. A judge emitting 74.3 and 68.1 looks precise. If it can't reliably put the right run on top, that precision is decoration.

The rest of the metrics: in-band rate, parse/coverage rate, and stddev across trials — trial-to-trial consistency, which is just asking whether the judge is stable against itself before you ask it to be right about anything else. The suite passes only when in-band ≥ 0.9, parse ≥ 0.98, and all orderings hold. Orderings are the one with no partial credit. Six out of seven is a failing grade.

Seven iterations​

† Iterations 6–7 run against the harder 24-scenario suite and are not comparable to iterations 1–5. The suite got harder; the number went down. We could have reported 98.2% and stopped at iteration 5, and it would have been the better headline. It would also have been the exact failure this post is about.

The prompt is usually not the problem​

Read the Change column again. The two biggest wins are iterations 1 and 3, and neither one touched the rubric. Both are renderer changes. They changed what the judge could see, not what it was asked.

Iteration 1 moved in-band fifteen points by raising the final-response window from 800 to 4,000 characters and tool summaries from 100 to 400. Nothing about the judge's instructions changed. We just stopped hiding the evidence from it.

Call it evidence starvation. A judge asked to verify grounding, handed 100-character tool summaries, cannot verify anything — the citation it's meant to check got truncated three fields ago. So it does what any reasonable grader does with missing evidence and a demand for a verdict: it rates good work as fabricated and incomplete. The rubric was fine. The rubric was never the problem. The judge was reading through a keyhole.

Iteration 3 is the same lesson in a different key: tool-call arguments were missing from the decision trail, so the judge could see that a tool ran but not what it was asked. That's not a rubric gap. That's a hole in the evidence.

Nearly everyone tunes the prompt first. It's the most available surface — a text file, right there, infinitely adjustable. So teams write increasingly elaborate instructions to a judge that structurally cannot see the thing they're instructing it about. Before you rewrite the rubric, check whether your judge can actually see the thing you're asking it about. Most judge quality problems are platform problems in a prompt costume.

Hash what the judge reads, not what the human reads​

Two mechanisms keep the numbers honest over time, and both are about comparability — the property you destroy by accident.

Each rubric carries a content hash of its static prompt parts. Verdicts produced under different rubric text are never aggregated together. Change the rubric, get a new hash, get a new bucket. You cannot accidentally average yesterday's scores with today's under a different definition of "good," because the system won't let the two meet. Every dashboard that silently mixes verdicts across rubric versions is reporting a number that means nothing.

The nice detail: display-only help text is deliberately excluded from the hash. Improving the human-facing explanation shouldn't trigger a pointless re-judge of the corpus. Hash what the judge reads, not what the human reads.

The same instinct shows up in rubric edits. When one rubric was hardened against over-tagging, the output criteria and schema were left unchanged specifically so scores stay comparable. Improving your judge while destroying your ability to compare to last month is a bad trade — and it's one you make by accident, in a commit that felt like pure upside.

Guards keyed on what the judge already sees​

When a judge over-tags "missing tool call," the instinct is to write it a lecture. Lectures are weak. A guard tied to metadata already in front of the judge is strong: emit that tag only when run metadata shows zero tool calls and the request needed external data; never emit it when the tool count is above zero.

That guard only works if its premise is true, so we verified in code that the tool count really is rendered in the block the judge reads. That step is not optional and it is routinely skipped. A guard whose premise is false is worse than no guard, because now you've got a rule that reads as rigorous and fires on vibes.

What we still can't claim​

Two admissions, and they're the point.

A guard in place is not a problem fixed. Proving it requires a re-judge of the corpus. Shipping the guard and declaring victory is precisely the error the whole apparatus exists to catch — asserting a result you didn't measure, with more confidence than the evidence supports. Judges do it. So do we.

Even at temperature 0, a judge is not fully deterministic across serving providers. Perfect reproducibility isn't on the menu. Knowing your noise floor is — which is why stddev across trials is a first-class metric rather than a footnote. You can't eliminate the noise. You can refuse to mistake it for signal.

Two more places where the honest answer is a shape, not a number. Where tool output isn't captured, the grounding criterion returns unknown — not fail. A judge that can't see the evidence must say so rather than guess; absence of proof is not proof of fabrication. And a run that legitimately suspended or failed is judged on its trajectory so far, not penalized for a final answer it had no reason to produce.

The eval is the product. And the judge is part of the eval — which means every number you trust is resting on a system that, until you evaluate it, nobody has evaluated. The judge doesn't announce that it's drifting. It just keeps returning numbers, in the same confident format, forever.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.

Most advice about token efficiency is prompt golf. Trim the system prompt. Cut the pleasantries. Replace the paragraph with a bullet. Shave two hundred tokens off the instructions and feel briefly virtuous about it.

It's the wrong game. Two hundred tokens is a rounding error next to a single tool result you didn't need to look at, dragged forward through every turn of a thirty-turn run. The wins that matter are structural: they come from deciding what never enters the context window in the first place. Not what gets said more tersely — what never gets said at all.

This is a practical guide to making that decision well. Four moves, each one a place where a real agent either touches something or doesn't, and then a section on what to do about it when you sit down to build your own.

Why prompt golf loses​

A system prompt is written once and sent every turn. So is the tool list. So is every tool result you left sitting in the message history. The prompt is the only one of those three you can see in a text editor, which is why it's the only one anybody optimizes — and it's usually the smallest.

Here's the asymmetry that decides everything: the context window is not a budget you spend, it's a budget you re-spend. A message that enters the history on turn four is present on turn five, and six, and every turn after, until something removes it. Put a 60,000-character API response in context on turn four of a twenty-turn run and you haven't made that decision once. You've made it sixteen times.

Prompt golf optimizes the constant. Structure optimizes the multiplier.

The four moves​

Every row is the same instinct applied to a different surface. Let's take them one at a time.

Move 1: keep the bulk out of context entirely​

Our tool layer enforces an offload contract. Any tool result over roughly 4,000 characters never enters the message history at all. What the model gets back is an ~800-character preview and a file reference — a handle to the full result, sitting on disk, which the agent queries with SQL when it actually needs something out of it.

The instruction the agent carries is blunt on purpose: the bulk is NEVER in context; query it.

The preview is the part people notice. The consequence is the part that matters: a large result never enters the message history, and therefore never enters the checkpoint either — so it isn't re-sent on every subsequent turn.

These reference files are read-only by instruction, and the reason is unglamorous and useful: they can run to tens or hundreds of megabytes. Opening one whole would exhaust the sandbox's memory long before it exhausted anything else. The file is not a smaller version of the result. It's the full result, somewhere the model doesn't have to look at all of it.

Move 2: loop in code, not in turns​

The agent has a real Python kernel, and it's taught a three-tier ladder for deciding how to act:

1. Answer directly — no tool at all.
2. Call a tool directly — for ONE discrete action. Don't wrap a one-shot call in code.
3. Write and run code — for orchestration: chaining, looping, bulk fetching.

The rule that does the work sits at tier three: calling the same tool across a list is a LOOP — write it once in code.

Consider fetching fifty records. Done one per turn, that's fifty round trips through the model, and — worse — fifty results sitting in the message history, re-sent on every turn that follows. Done in a loop inside one cell, it's one turn and one offloaded reference.

# The per-turn version: 50 turns, 50 results in history, 50 re-sends per turn.
#   turn 1: get_record(id="r_001") -> {...}
#   turn 2: get_record(id="r_002") -> {...}
#   ... 48 more turns, each one carrying every result before it ...

# The in-code version: 1 turn, 1 offloaded reference, history untouched.
records = [get_record(id=r) for r in record_ids]   # 50 calls, 0 extra turns
open_ones = [r for r in records if r["status"] == "open"]
summary = {"total": len(records), "open": len(open_ones)}
print(summary)          # small enough to enter context — that's the point

Only summary crosses back into the message history. The fifty records stay in kernel memory, and because the kernel persists across cells, they stay reachable — you don't re-fetch to re-use. State carries.

The tier-two caveat is real, though, and worth respecting. A single discrete action wrapped in a code cell is worse than just calling the tool: you've added a kernel round trip and a layer of indirection to buy nothing. The ladder has three rungs because there are three right answers.

Move 3: names in the prompt, bodies on demand​

Every bound tool is a permanent line item. It's in the prompt on turn one and it's in the prompt on turn thirty, whether or not the agent ever calls it.

So we split the tool surface in two. Tool names are all listed in the system prompt. Their parameter schemas are not. The agent fetches a schema only for the tools it's about to use, and it's told to batch that fetch: pass EVERY tool you're about to use in ONE call, instead of calling this repeatedly — each call is a separate step.

The name list is itself capped, at 50 names, after which it truncates with a (+N more). That number is a judgment call, not a law of nature: long enough to function as a menu the model can browse, short enough not to become a tax on every turn.

The same principle governs skills. A skill is loaded on demand, which means the agent pays the token cost only when the skill is actually relevant to the turn. A skill you might need on one turn in fifty should not be present on the other forty-nine.

And here's the transferable version for anyone building on the platform: bind fewer tools. An agent with forty tools pays for forty tools on turn one and turn thirty. Not because forty is too many in the abstract, but because most of those forty were bound speculatively — "it might need to update the record" — and a speculative binding is overhead on every turn that follows.

Move 4: let the harness compact, and don't fight it​

Compaction is the safety net, and it works. It triggers at a threshold — 180,000 tokens — keeps only the most recent few tool outputs in full, and reports exactly what it freed. There's a reactive path too, forcing compaction when the model itself reports an overflow the static threshold didn't predict. And rolling memory consolidation runs at around 70% of the context budget and on a turn interval: deliberately before the pressure point rather than at it, because a consolidation with room to be thoughtful is a better one.

All of that is machinery you get for free. But compaction is not free in the sense that matters: something has to read the history to summarize it, and whatever gets summarized is no longer there in full. The goal isn't to compact efficiently. It's to arrive at compaction rarely — which loops straight back to Moves 1 and 2. An agent that offloads its bulk and loops in code approaches the threshold slowly. One that carries every result forward sprints at it.

When you build your own agent​

The four moves above are platform mechanics. These are the decisions that are yours, in roughly the order they'll bite you.

Bind the tools the agent needs, not the tools it might need. This is Move 3 with your name on it, and it's the one lever where you can do more damage than the harness can undo. Every speculative binding is permanent overhead on every turn of every run.

Name the deliverable and the stopping condition. An agent that doesn't know what "done" looks like spends turns discovering it — and those exploratory turns are the most token-hungry kind, because each one adds to the history that the next one re-reads. "Produce a summary table of open items, grouped by owner; stop when every open item appears once" is a brief. "Look into the open items" is an invitation to wander.

Set the KPI measurement interval to something sane. Measurement runs on every Nth run, not every run. A KPI you measure every single run means a judged evaluation every single run — and a metric that moves slowly does not need measuring quickly. Weekly-drifting quality does not need a per-run verdict.

Keep KPI slots few. There are 8. Using all 8 because they're there means judging 8 criteria on every measured run. Pick the ones whose movement would actually change what you do.

Prefer one capable agent with a narrow brief over a broad agent that must first work out what you meant. Interpretation is a real workload, and you pay for it in turns before any of the useful work starts.

Match the schedule to the decision it feeds. A schedule that runs hourly for a report nobody reads hourly is the most common waste there is. It's also the easiest to fix — nobody has to refactor anything, someone just has to admit the report is a daily report.

The mindset​

Read the four moves back to back and they collapse into one instinct. Don't carry the bulk — reference it. Don't repeat a call across turns — loop it in code. Don't ship schemas for tools you aren't using — fetch them when you are. Don't sprint at the compaction threshold — arrive slowly.

Every one of those is a version of the same question: what does the model never have to look at?

The efficient agent isn't the one with the terser prompt. It's the one that touches the least.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.

Everyone who designs an agent loop spends their time on two steps. They tune the generate step, because that's where the intelligence appears to live. They harden the verify step, because that's where the discipline lives. Almost nobody budgets serious design effort for the third step, which is the one that actually determines whether the loop is trustworthy: the stop.

Here is the uncomfortable mechanical fact underneath every loop you have ever built. A turn that produces text with no tool call is the only place a run can stop. That's it. That's the entire exit condition — the absence of a function call. And that single signal conflates three completely different situations, which is why agents routinely "finish" tasks they never did.

This post assumes you already have a loop — generate, verify, correct — and that it has burned you at least once. We're going one level down, into the exit.

One signal, three meanings​

When the model emits a turn with no tool call, exactly one of these is true:

• done — the agent delivered a real, complete answer.
• question — the agent is deliberately asking the user for something. An agent is always allowed to ask. A genuine ask is a first-class stopping point, never a stall.
• stalled — the agent stopped with the task demonstrably unfinished without asking: it narrated an action it never took, returned partial data, or skipped a step.

Read that third one again, because it's the whole post. Narrated an action it never took. Every agent builder recognises this instantly. "I've updated the configuration and re-run the checks." No it hasn't. There is no tool call in that turn. The model produced a paragraph describing work in the past tense, the loop saw no function call, the loop exited, and the harness handed a fluent lie to whoever was waiting.

The naive loop treats all three as one event, because at the transport layer they are one event: a message with an empty tool-call list. Left alone, done, question, and stalled all become "the run ended successfully," and your completion signal is decoupled from reality.

So we put a gate exactly there — at the no-tool-call boundary — and make it branch.

This pattern has a name in coding agents — a stop hook — and it generalises cleanly to any agent loop. The interesting part isn't the shape. It's the four design decisions inside it.

Decision one: no heuristics, on purpose​

The gate asks the LLM to classify the stop. There are deliberately no string or phrase heuristics, because only the model that produced the turn can reason about whether it actually finished.

This is the decision people push back on hardest, and it's the one we're most confident about. The tempting move is obvious: regex the turn for "I've completed", "Done!", "Let me know if". Ship it in an afternoon, zero extra model calls, deterministic.

It's a trap, and the reason is structural. The sentence "I've completed the migration" is exactly as likely to appear in a done turn as in a stalled turn. That's what makes stalls dangerous — a stalled agent is not confused or hedging. It's confident. It writes the completion sentence because it believes the completion sentence. The surface text is identical in both cases; the only difference is whether the preceding transcript contains the tool calls that would make the sentence true. A regex cannot see that. It matches the claim, not the evidence for the claim.

The thing that can see it is a model looking at the turn and the work that led to it. Judging "did this actually happen" is a reading-comprehension problem over a transcript, and the only tool we have that does reading comprehension is the model. So we use it, and we accept the cost of an extra call rather than pretend a pattern match is a judgement.

The gate reads a bounded window — the last 8 messages — not the whole run. The classification is about the stop, not the history. You're asking "is this turn honest given what just happened", not "summarise the last forty minutes."

Decision two: put the expensive check at the boundary​

The gate runs only at the no-tool-call boundary. Turns that emit tool calls route straight to the tool node and never reach it.

That one sentence is what makes the design affordable. A long run might be forty turns, thirty-eight of which call tools. Those thirty-eight never touch the gate. The judging happens at candidate stopping points — roughly once per run, a handful of times if the agent stalls and recovers.

Generalise it: put your expensive check at the decision boundary, not in the hot loop. People reach for per-turn monitors and per-turn critics, discover the overhead is untenable, and rip the whole idea out. Usually the check didn't need to run per turn. It needed to run at the moment a decision becomes irreversible. Ending a run is irreversible — the answer goes to the user. Calling a tool isn't; another turn is coming that will re-examine everything. Spend your judgement where the door closes.

Decision three: the plan inverts authority​

This is the strongest idea we have on the subject.

When a run carries a plan, the plan — not the model and not the judge — decides completion. Every step marked completed ⇒ stop. Any step unmarked ⇒ the harness rejects the stop and re-injects the specific next step, so the model drives its own plan to completion and cannot hallucinate the whole task done with work remaining.

Sit with the inversion. The agent writes the plan. The plan then binds the agent. A model that decided at step two what needed to happen does not get to decide at step five that actually it's finished — the artifact it produced when it had a clear view of the task now overrules the model that has drifted, gotten tired of the task, or convinced itself the remaining steps were optional.

That's the trick, and it's more general than agents: externalise the completion criterion out of the model's judgement and into an artifact it can't retroactively edit. Completion stops being a vibe the model reports and starts being a property you can check. Six of seven boxes ticked is not a matter of opinion.

And note what the rejection does. It doesn't scold. It doesn't say "you're not done, try harder." It re-injects the specific next step, because a loop is only as good as its feedback string — and "step 4: verify the new records resolve" is a task, while "incomplete" is a mood.

Decision four: a budget that can't be gamed​

There's a cap of 20 continuations. The anti-gaming rule matters more than the number:

Each stall consumes one unit, and a tool call in between does NOT refund it — so an agent that keeps making hollow tool calls but never completes (e.g. a fetch that always returns partial data) still terminates.

Refunding on tool calls is the obvious-looking design and it's wrong. If real work resets the budget, then anything shaped like real work resets the budget, and an agent stuck in a genuinely unresolvable situation — a source that always returns half the rows — will loop forever while looking productive the entire time. Non-refundable units mean the budget measures stalls, not activity.

Same reasoning excludes bookkeeping tools. Updating the plan and marking a task complete don't count as real work. Otherwise an agent keeps its budget alive by ticking its own boxes, which is a very sophisticated way of doing nothing.

On the last unit the gate injects one "wrap up with what you have" nudge, so the run ends with a best-effort answer rather than a dangling "let me re-fetch." A budget that expires silently mid-thought produces the worst possible artifact: a truncated intention. Land the plane.

Decision five: precedence is policy​

One small thing at the router with a real lesson in it. The router checks the hard tool-budget stop first, before checking whether a user requested a stop, then falls through to continuing.

def route(state):
    # This is checked FIRST — a runaway that also has a
    # pending stop must still end.
    if state.tool_calls >= TOOL_BUDGET:
        return "end"
    if state.user_stop_requested:
        return "end"
    return "continue"

Both branches end the run, so the order looks cosmetic. It isn't. The ordering encodes which authority wins when two conditions fire at once, and every one of these little if chains is a policy document that nobody reviews as one. Write them in the order you'd defend out loud: hard safety limits, then user intent, then the default. In a router, precedence is policy — you are ranking your principles whether or not you meant to.

What the stop is really for​

Upstream of the exit, everything is optional in a way you can talk yourself out of. You can ship a loop with a weak verifier and tell yourself the model is usually right. You can skip the plan. The system will look like it works.

The stop is where that stops being true, because the stop is the only step whose output the user actually sees. A loop that can't tell "I'm done" from "I gave up" from "I need you" will confidently hand you all three as the same thing — and the one it hands you most often is the middle one wearing the first one's clothes.

Getting the stop right is what makes everything upstream of it worth building.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.

Ask an engineer how to stop an agent that won't stop, and you'll get the same answer every time: add a limit. It's the right instinct and the wrong plan. There is no single limit that catches every shape of runaway, because runaways don't have one shape — and each shape slips past a different guard for a different structural reason.

The failure mode people picture is a model producing one bad answer. That's not the interesting failure. The interesting failure is a loop that keeps making plausible, individually-reasonable calls forever. Every call, read on its own, is a call you would have approved. Only the hundredth one tells you anything is wrong — and by then the run has been busy a long time, doing work that looked like progress the whole way down.

That we build the harness rather than treat it as glue is settled ground here. This post is one level deeper: the specific mechanics of putting a floor under an autonomous loop, and why bounding one takes four independent guards instead of a number in a config file.

The insight the whole thing hangs on​

Start with the guard everyone reaches for first: refuse a call the agent has already made. Deduplicate. If it asks the same question twice, it's stuck.

That guard never fires.

A runaway per-record loop calls the same tool with different arguments every time. Record 1, record 2, record 3. Message A, message B, message C. No call ever repeats itself, so a de-duplicator sees a stream of perfectly novel, perfectly legitimate requests and waves each one through. The loop isn't repeating. It's enumerating — and enumeration is indistinguishable from diligence, one call at a time.

Every wall below exists because some loop shape is invisible to the others. That's the design constraint — not defense in depth for its own sake, but four guards keying on four genuinely different things, because a loop that's obvious along one axis is featureless along the rest.

Wall 1 — the per-tool-name budget​

The first wall caps calls at 40 per tool name, and it ignores arguments entirely.

That last part is the point, and it's a direct response to the shape above: the observed failure was the same tool called hundreds of times with different ids, so an identical-call guard would not catch it. Strip the arguments out of the identity and the loop becomes trivially visible. Forty reads is forty reads whether it read the same record forty times or forty different ones.

Crude on purpose. A tool name is the coarsest possible key, which is exactly why nothing about the arguments can hide from it.

Wall 2 — the aggregate budget, and the trick inside it​

Wall 1 has an obvious hole: an agent that calls eight different tools thirty-nine times each never trips a per-name cap and has still done three hundred operations. So the second wall is tool-agnostic — a soft cap of 40 across all external tool operations. A mail read, a repository listing, and a knowledge search all count identically. There is no notion of a cheap call.

The clever part isn't the cap. It's what happens past it.

Past the cap, a call identity is refused once. An identical repeat is allowed through.

Sit with that for a second, because it's doing something subtle. A runaway per-record loop iterates different args each call, so no call ever repeats its identity and the loop stays refused — forever, every call, one after another. A genuinely-required call proceeds on its confirmed retry: the agent gets told no, decides the call really is necessary, asks again the same way, and gets it.

A loop cannot confirm itself, because confirming requires repeating — and the loop never repeats. The exact property that makes a runaway invisible to a de-duplicator is the property that keeps it behind this wall — the behavior that defeats one guard is what the other is built to catch. The agent isn't asked to prove intent. The mechanics of its own loop answer for it.

Wall 3 — the delivery reserve​

Send, post, deliver: these draw on a separate reserve of 5 operations, argument-independent, exempt from the data-gathering cap.

Without it, the arithmetic goes badly. A run spends its entire budget reading and has nothing left to deliver the result. All research, no answer — the worst place to stop, because all of the expensive work happened and none of it reached anyone. A budget that can strand its own output is a budget with a bug in it.

There's an honest design note here too. The refuse-once-then-confirm cycle from Wall 2 was too fragile for sends. It required the agent to repeat the exact call — defeated the moment it reworded the message between attempts. And of course it rewords the message; that's a reasonable thing to do when you've been told no.

A read is idempotent enough to retry; a send is not. Different semantics, different budget. When a mechanism's assumptions don't hold for a class of tool, the answer is a second mechanism, not a stretched first one.

Wall 4 — the query-loop guard​

The last wall caps at 10 and keys on (file_ref, normalized_sql). A normalizer strips string and number literals down to a skeleton, so WHERE ts='A' and WHERE ts='B' collapse to the same signature.

This catches the loop that both other guards structurally miss. A per-tool-name count sees one tool used a reasonable number of times. An identical-call guard sees ten distinct queries. Neither is wrong. The loop is simply invisible along both of their axes — and visible immediately along a third: same query shape, different literal, over and over.

Normalization is what makes an enumerating loop's signature stop moving.

The hard stop above them​

All four are soft. They refuse; they don't terminate. Above them sits a single hard aggregate cap on external tool operations — tunable per deployment, and setting it to 0 disables it.

The mechanism is worth showing because it's clean. At the cap, the harness injects a terminal assistant message with no tool calls. The router reads that marker and routes to __end__:

# At the hard cap: don't raise. Speak, then let the router do its job.
if hard_cap and external_ops >= hard_cap:
    state.messages.append(
        assistant_message(hard_stop_notice(external_ops, hard_cap))
    )
    return state          # note: no tool calls attached

def route(state):
    last = state.messages[-1]
    if last.tool_calls:
        return "tools"
    return "__end__"      # a terminal message is already the end shape

No exception. No special termination path. The run ends at the graph's normal completion shape — the same one a successful run uses — because "an assistant message with nothing left to call" already means finished. This mirrors a precedent in the same router, where a synthesis-complete message ends the run identically. The hard stop didn't need new machinery. It needed to say the thing the router already understood.

And the terminal message is written for the user, not the log:

I stopped after {n} tool calls — this run hit its hard limit of {cap} tool
calls, which guards against a runaway loop. I've summarised what I gathered so
far; if the task genuinely needs more tool calls, break it into smaller runs
or raise the agent's tool-call budget.

Error-as-data: the principle underneath all of it​

None of the four walls work as exceptions. Refusals are returned as tool errors the model reads and re-plans against.

The governing rule: a refusal must never read like a success. Every refusal message states explicitly that the call did NOT execute.

This sounds fussy and isn't. A guard that silently returns nothing teaches the model that the tool is broken — and a model that believes its tool is broken does exactly what you'd do: retry it, try a neighbouring tool, work around the outage. You've converted a budget into a stimulus for more calls. A guard that explains teaches the model to change strategy, which is the only outcome you wanted. The refusal is an input to the next planning step, so it has to be written like one.

Metering by construction​

A tool is metered if its canonical name isn't in the local-tools set. That's the whole rule — external ops get counted uniformly, and nobody maintains a per-vendor list that goes stale the day someone adds an integration.

Two details make it hold. A canonicalizer collapses the two surface spellings of the same tool — the function-call form and the code-mode form — into one identity, so an agent can't dodge a budget by switching call styles. And synthetic messages the harness injects itself never count; the meter measures the agent, not the scaffolding around it.

The wall that's soft on purpose​

One budget is deliberately not a wall. The turn budget advances in steps of 50 — 50, then 100, then 150 — and injects a warning the agent is free to ignore. It gates nothing.

That's not a weaker version of a hard cap. It's a different tool. A soft signal shapes behavior; a hard wall guarantees termination. A warning lets a still-reasoning agent wrap up on its own terms, which is a better ending than any forced stop — but it promises nothing, because an agent that ignores it just keeps going. A hard cap promises termination and can only deliver it bluntly. You need both, and you need them separate: a soft signal that gates is a hard wall with worse ergonomics, and a hard wall that warns is a wall that doesn't work.

A seam worth naming​

Only the turn budget reaches the box today. Token and wall-clock budgets are computed but not enforced in-box — the outer bound on wall-clock is the sandbox's own lifetime. That's a real gap, and we'd rather write it down than let the diagram imply coverage the code doesn't have.

The four walls are the shape of the argument, though, and the shape generalizes. The question is never "what's the limit?" It's "what does this loop look like, and which key makes it stop moving?" Answer that four times and you have something that holds.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.

A skill is a self-contained Markdown directive with a little YAML frontmatter on top, sitting in its own directory. That description makes it sound like documentation, which is exactly the trap. The interesting part is not what a skill contains. It's when the agent pays for it.

Here is the sentence we keep in our own skill registry, and it is the whole design: a skill is a directive the agent reads on demand via a read_skill tool, rather than something injected into every prompt — so the agent pays the token cost only when the skill is actually relevant to the turn.

Read that twice, because every rule below is a consequence of it. On demand is not an implementation detail you can bolt on afterward. It is the property that makes a skill a skill instead of a system prompt with extra steps. And it means the best skill you can write is one that is absent from most of the agent's turns — present, correct, and completely unloaded.

What you are actually writing​

Structurally there is almost nothing to it:

---
name: design-core
description: >-
  Read BEFORE designing any visual deliverable — decks, documents, pages.
  Carries the non-negotiable design bans and the layout method. Load this
  first, then the medium-specific skill.
---

# Core design directive

## Absolute anti-slop bans
- No Times. No Arial.
- No pure black text on pure white.
- No default 1px grey borders.
...

Frontmatter, a body, a directory. The agent's harness knows the file exists and knows what the description says. It does not know a word of the body until it asks. That gap — the harness knowing the name while deferring the content — is the entire budget you have to work with, and four rules follow from spending it well.

Rule 1: load it through a tool, not a shell command​

The agent could read the file with cat. Same bytes into context, no tool required. Do not do this.

Loading through a dedicated tool keeps the raw directive text out of the user-facing terminal stream. It rides to the model as a tool result; the UI shows a compact card. If the agent cats the file instead, the user watches a wall of imperatives about font choices scroll through their chat — a conversation they were never party to, rendered as if it were output.

The tokens are identical. The product is worse. That asymmetry is worth internalizing, because it's the first place skill design stops being a context problem and starts being an interface problem. Your directive is addressed to the model. Anything that leaks it to the human is a bug, even when it's free.

Rule 2: the description is the routing logic​

If bodies load on demand, something has to decide when. That something is the description field — and it works because the tool that loads skills enumerates the valid skills in its own description. The model sees the full menu of names and triggers without loading a single body.

Names in the prompt. Bodies on demand.

We apply the same principle one layer down, to tool schemas: every tool name is in the system prompt, but the parameter schemas are fetched only when needed. The guidance we give the fetching tool says it plainly — your tools are ALL listed by exact name in the system prompt; this is NOT for discovery, it fetches the param schema you don't yet have. Discovery is free. Detail is not. Skills are that idea applied to prose instead of JSON.

Which reframes what your description is for. It is not a summary of the skill. Nobody is browsing. It is a trigger — the condition under which a model that has never read your body should decide it needs to. Write it as an instruction about when, not a précis of what:

• Summary: "Guidance and conventions for building presentations."
• Trigger: "Read before creating any slide deck, after the core design skill."

The second one routes. The first one describes.

Rule 3: layer, don't lump​

The instinct when writing a skill is to make it complete. Resist it. Our six shipped skills are deliberately layered: one medium-agnostic core design directive is read before designing anything, and the medium-specific skills build on top of it with the format rules for exactly one output type. The agent reads the core, then the layer.

The sizes are the point. Not one of them is a manual. The core carries what is true of every visual deliverable; each layer carries only what is true of its medium and nothing that the core already said. A spreadsheet task loads roughly 10 KB total and never sees a word about slide transitions.

The failure this prevents is subtle. A single lumped skill isn't just bigger — it forces every task to carry every medium, which means the moment it starts crowding the context window, someone "fixes" it by making it vaguer. Layering lets you stay specific, because specificity is cheap when it's scoped.

Specificity is also what makes a skill bite. Our core design directive opens with "Absolute anti-slop bans": no Times, no Arial, no pure black on pure white, no default 1px grey borders. Concrete prohibitions beat vague encouragement, every time. "Use tasteful typography" is a sentence a model can agree with while producing Arial. Skills carry a persona and hard bans because that's what survives the trip through a generative process. And when prose is the wrong instrument, ship an executable script alongside the markdown — a skill directory is not obligated to be only words.

Rule 4: reference big data by path, never by content​

Our presentation skill needs a 54 KB theme catalog. It does not contain a 54 KB theme catalog.

The loader appends the catalog's resolved absolute path to the skill text, so the agent can json.load it inside run_python without the prompt ever carrying the data — and the skill explicitly instructs the agent not to dump the thing into the chat. A 54 KB asset costs a path.

Generalize this. Any time your skill wants to contain a lookup table, a schema dump, a catalog, or a corpus of examples, it wants a path instead. The agent has a filesystem and an interpreter. Prose is for judgment; data is for disk. A skill that inlines its data spends the whole context window on a lookup that a single line of Python does better.

The part where we admit it's hard​

The skills we ship inside the product run 2.5–8 KB. The skills our own engineers write for their internal dev tooling run 25–85 times larger. One internal QA skill is 215 KB — far, far past any progressive-disclosure guidance we would give anyone, including ourselves.

The detail that makes it instructive rather than embarrassing: that skill has an envs/ subdirectory. The author knew the pattern. They applied it to the peripheral material and not to the main body.

That's the failure mode, and it's always the same one. Nobody sits down to write a 215 KB skill. They write a good 8 KB one, then answer a question in it, then another, then paste in the thing that keeps coming up — and each addition is locally correct. The skill grows into a manual because nobody pays the cost of loading it at review time. The author reads the diff. The agent reads the whole file, on every task that touches it, forever. The discipline is easy to state and genuinely hard to keep, and pretending otherwise would make the rest of this post less useful.

The pattern that actually holds the line is two-tier. One of our better internal skills opens by declaring what it is: it orients and routes, and the deep, authoritative reference is a longer document — the single source of truth. Read it before non-trivial work; don't duplicate it here. Eight kilobytes of skill pointing at twenty-nine kilobytes of reference.

Skills should route to depth, not contain it. The skill's job is to know that the reference exists, say when it matters, and get out of the way. The reference's job is to be right. Conflating those two jobs is how you get 215 KB.

How to know yours is good​

Read your skill and ask when it should not load. If you can't answer crisply, your description isn't a trigger and your routing will be guesswork. Then check that nothing in the body would be equally at home in the core, in a sibling layer, or in a file on disk — those are the three things a bloated skill is made of.

Then apply the test that subsumes all of it:

If your skill would be equally useful pasted into every system prompt, it isn't a skill. It's a system prompt you haven't admitted to.

That's not a rhetorical flourish; it's a diagnostic with a fix on either side. If the content genuinely belongs in every turn, move it to the system prompt and stop paying the indirection. If it doesn't — if it's for decks, or for one language, or for the third Tuesday of the release cycle — then earn the deferral. Sharpen the trigger, split the layer, push the data to a path.

A skill earns its keep by being absent most of the time. Write it so the agent is glad to see it, and doesn't have to.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.

"Run this agent every morning at 9" sounds like a one-line feature. There's a cron expression, there's a job table, there's a thing that reads the table. Ship it Friday.

Cron is the easy 5%. The other 95% is not firing twice, not losing a fire, not holding a database transaction open across a task that runs for minutes, and deciding whose 9am you actually mean. A cron expression is a trigger. A scheduler is everything you build around it.

This is a tour of that everything — the design of the scheduling layer under CatalEx, and the reasoning behind the parts that look strange until you've been bitten.

The shape of the stack​

A managed cron trigger fires a tick endpoint every minute. The jobs service scans the database for anything due, and dispatches each due job through a managed task queue. The queue calls back into the owning service's handler, which does the actual work.

Four moving parts, three network hops, and every one of them is a place where the same job can fire twice or not at all. Everything below is about closing those gaps.

At the top of the stack, a retry is worse than a miss​

The cron trigger is configured with zero retries. Deliberately. The comment in the code is the whole argument:

The next tick will re-evaluate any jobs that were due; re-trying a failed tick risks double-dispatch against the same window.

This inverts the instinct you've built everywhere else in your system. Down in the leaves, retry is free and correct. At the root of a scheduling stack, retry is a duplicate-fire generator. The tick is not a request that must succeed; it's a sample of a clock that will be sampled again in sixty seconds. A missed tick costs you a job running one minute late. A retried tick that partially succeeded costs you a job that ran twice — and "ran twice" for an agent means two emails sent, two tickets filed, two rows written into someone's CRM.

Fail-open on the way down. Fail-closed on the way up.

Four layers against double-fire​

Zero retries on the tick is a policy, not a mechanism. Below it sit four mechanisms, each one assuming the layer above it failed.

Layer 1 is why you can run more than one dispatcher. SELECT … FOR UPDATE SKIP LOCKED on the due-job scan means concurrent dispatcher instances never double-dispatch the same job — the second instance doesn't block on the locked row, it walks past it and takes the next one. Without SKIP LOCKED you get either a convoy (everyone blocks behind row one) or a single-instance scheduler, which is a single point of failure with extra steps.

Layer 2 is the one people skip. next_run is advanced in the same committed transaction that writes the execution row. The moment that commit lands, the job is no longer due — before the dispatch has even been attempted. The scan can't pick it up again on the next tick, because as far as the query is concerned, the window has already passed.

Layer 3 costs one argument. Every enqueue passes the execution id as the queue's task name. A queue that redelivers the same task name inside its dedup window drops it. This is free protection you get by naming a thing carefully.

Layer 4 is the backstop, and its reasoning is the elegant bit. The handler, before doing any work at all, inserts a claim row keyed on execution_id with ON CONFLICT DO NOTHING. If the insert claimed nothing, someone else already has this execution, and the handler returns {"skipped": "duplicate execution"} and goes home.

The usual objection to handler-side idempotency is that you'll swallow a real retry. You won't — but only because retries are minted upstream as new executions, so identity and attempt are never conflated. Idempotency keys work when the key means this delivery, not this task. Get that distinction wrong and your dedup layer starts eating work.

Three rules about transaction boundaries​

These are design rules, not scars. Each one exists because the alternative has a specific, predictable failure mode.

Never hold a transaction across the dispatch. A downstream agent run takes minutes. Wrap the dispatch in the transaction that read the due jobs and you're holding an idle-in-transaction connection for the duration — long enough to trip the database's idle-in-transaction timeout, at which point the connection dies mid-update and you get to find out what your code does about that. Hence the split: read due jobs, commit, then dispatch.

Commit before you dispatch — in that order. Dispatch first and the callback opens a fresh transaction that cannot see your uncommitted execution row. The handler looks up an execution that doesn't exist, and here's the vicious part: it still returns 200. The queue sees success, never retries, and the work vanishes with no error anywhere. Silent loss is worse than loud failure, and the ordering is the only thing between you and it.

Report failures in-band. A malformed job returns HTTP 200 with status="failed" in the body. It's tempting to return a 5xx — it feels honest. But the tick's transport layer would read that as a transport error and apply transport retry semantics to a permanently broken job, re-dispatching a payload that will never parse. Status belongs in the body so the tick can apply its own retry policy. HTTP status codes describe the conversation, not the work.

Whose 9am?​

Now the part that looks like a bug and isn't.

Cron parsing is a standard 5-field expression, always UTC. There is no timezone column. Not "we haven't added one yet" — the API contract explicitly forbids timezone-bearing fields on the wire. The database stores a UTC cron and nothing else.

All local↔UTC math lives in the browser, because the browser is the only place that knows the user's local zone. The server never guesses, never infers from a header, never stores a stale zone that was true when the user signed up in a city they've since left. There's one authority for "your 9am" and it's the machine sitting in front of the user.

There's no interval primitive either. "Every N hours" compiles in the browser into ${minute} */${n} * * *, with the divisors restricted to [1,2,3,4,6,8,12] — the clean divisors of 24. That restriction is doing real work: */5 hours doesn't tile 24 evenly, so it silently skips at the midnight wraparound. You'd get 0,5,10,15,20 and then a four-hour gap, forever, and nobody would notice for weeks. Restricting the input is cheaper than explaining the arithmetic.

Alongside the cron sits a display descriptor — frequency, time_of_day, days_of_week, interval_hours — stored purely so the UI can round-trip losslessly. Rendering a UTC cron through a timezone-naive formatter prints UTC numbers styled as wall-clock time, which is how you end up telling a user their job runs at 2pm when it runs at 9am.

Weekly schedules roll each weekday independently through UTC, then dedupe. A local Monday 11pm legitimately lands on Tuesday UTC while the rest of the week doesn't move. The naive version — shift the time, then shift every day by +1 — is wrong for exactly the days that cross midnight, which is the only case where it matters.

The honest tradeoff: because only a UTC cron is stored, schedules do not follow DST. A 9am local schedule becomes 8am or 10am local when the offset changes. Nothing recomputes crons on DST boundaries. This is a real limitation and we'd rather say it plainly than let you discover it in March. The alternative isn't free: following DST means storing the user's zone, which means recomputing every affected cron whenever a government changes its rules, and owning the ambiguity of schedules that land in the hour that doesn't exist or the hour that happens twice. UTC-only buys a schema with one interpretation. That's the trade, made with open eyes.

The unglamorous rest​

The remaining details are the ones that decide whether the thing survives a year.

Execution timeout is a per-job timeout_minutes, constrained at the database between 1 and 15. A stale sweep marks anything past its timeout as failed and retries it — which is the only place a retry is minted, and it mints a new execution id, which is what makes layer 4 sound.

Primary keys are UUIDv7 — time-ordered, so inserts land at the end of the index instead of scattering across it. Hot-path indexes are partial: the index on next_run carries WHERE status = 'active', because the dispatcher never asks about inactive jobs and there's no reason to make it page past them.

A duplicate-schedule guard allows one schedule per agent per cron expression, enforced at the application layer, returning 409. Distinct agents may share a cron — the constraint is against a user double-clicking their way into two identical jobs, not against 9am being popular.

Orphan protection runs both ways. Deleting an agent purges its schedules; and if a run finds its agent already gone, the handler purges the schedules itself. Otherwise an orphaned schedule fires every tick forever, doing nothing, and nobody notices because nothing is failing.

And a run that suspends — waiting on a human approval — ends without a terminal frame. The execution still counts as completed, and records a status breadcrumb instead. The dispatch did its job; a retry would double-run the agent. The scheduler's contract is "the work was handed off correctly," not "the work finished."

That's the shape of it. One cron expression at the top, and underneath it four layers of dedup, three rules about where a transaction may not go, a timezone model with a stated limitation, and a fistful of indexes. The cron was the easy part. It's always the easy part.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.

"What can your agent do?" is two questions wearing one coat. The first is about the action space: once the agent is awake and thinking, what can it reach, call, read, and change? The second is about the trigger space: what makes it wake up in the first place? Teams conflate them constantly, and the conflation is expensive, because the two want opposite answers.

We build for a deliberate asymmetry. The action space is as wide as we can responsibly make it — a real Python kernel, a shell, roughly a hundred curated third-party integrations, and permission to write loops instead of taking turns. The trigger space is three entries long. Not three for now, pending a roadmap. Three because every entry point into an autonomous system is a surface someone has to reason about, and we would rather have three surfaces we understand completely than thirty we understand approximately.

Three ways in, enforced in the schema​

Here is the entire trigger space, as a StrEnum backed by an eight-character database column:

class TriggerType(StrEnum):
    CHAT = "chat"
    RUN = "run"
    SCHEDULE = "schedule"

That column width is not an accident of history. Eight characters is a wall. Adding a fourth trigger means a migration, which means a conversation, which is exactly the friction we want on this particular axis. Cheap to add is the wrong property for an entry point into a system that takes real actions.

That's the list. There is no webhook trigger. No email-arrival trigger. No "someone posted in a channel" trigger. We know those are the first three features on most people's wish list, and we know exactly what the absence looks like from outside: a gap.

We'd argue it's a position.

Why a narrow trigger set is the feature​

The argument against webhooks isn't that they're hard. They're easy — that's the problem. Every webhook you accept is an entry point where the outside world decides when your agent runs, how often, and with what payload. Each one needs its own authentication story, its own replay semantics, its own idempotency reasoning, its own answer to "what happens when this fires four hundred times in a minute because an upstream system had a bad afternoon." That reasoning doesn't amortize across endpoints. It's per-endpoint, forever.

A schedule is a trigger you can reason about. It fires when you said it would. Its concurrency is bounded by its own period. Its payload is whatever the agent goes and fetches. And critically — you can poll from a schedule. Nearly every "I need a webhook" requirement is really "I need to know about this within N minutes," and a schedule answers that with a mechanism whose failure modes fit in your head. You trade a little latency for an entry point that cannot be triggered by anyone outside your system. That trade is available to you at any time, and it's usually the right one.

The discipline shows up elsewhere too. There is an internal event job type in the platform, and it would have been the natural place to sneak a fourth trigger in. It doesn't start agent runs. It handles post-run work — consolidating memory, scoring KPIs. Work that happens after an execution, on the same rails, never as a way in. The trigger set stayed three because we kept it three when it was inconvenient.

One path, many entry points​

Here's the part that makes the narrow set work rather than merely sound principled.

A scheduled run is not a special execution mode. It is not batch mode. It is not a lighter-weight cousin of the interactive path with a few features stubbed out. It goes down the identical orchestration path as a user-initiated run — different surface, different trigger_type, same everything else. The intent is written into the module itself: keep the scheduled run "on the identical orchestration path as a user-initiated run."

You can see the discipline hold under pressure at the one place you'd expect it to bend. When someone manually triggers a schedule — the "run this now, don't wait for 6am" button — the obvious implementation is a shortcut: skip the schedule machinery, fire a direct run. We don't. It creates an execution with trigger_source="manual" and dispatches immediately, leaving the recurring schedule intact, and it reaches the agent through the same schedule path. The manual trigger is a nudge to an existing mechanism, not a bypass around it.

The alternative is where divergence bugs live. A separate batch mode starts as ninety percent shared code. Then the scheduled path gets a fix the interactive one doesn't. Then someone adds a guard to the interactive path because that's where the user complaints come from. Six months later you have two systems with one name, and the failure you're debugging only reproduces at 4am on Tuesdays. One path with many entry points means a fix lands once, and every trigger inherits it. It also means the trigger set can stay small without costing capability — because narrowing the ways in doesn't narrow what happens once you're in.

Now the wide half​

Everything above is about restraint. None of it applies to the action space, where the answer is: as much as we can hand it.

The agent has a real computer. Inside the box, a set of tools run locally — run_python, bash, text_editor, query_file, read_file, update_plan, load_tools, get_tool_details, fetch_full_result, query_resource, user_input, task_complete, save_memory. These aren't metered as external operations, because they aren't external. They're the agent using its own machine.

Above that sit first-party servers: web search, the Brain knowledge and search layer, a platform "system" server, and a tool catalog the agent queries to find out what else exists. And beyond those, roughly a hundred curated third-party integrations reached through our integration layer — CRM and sales, communication, finance, documents and storage, calendar and meetings. Wide by category, curated rather than infinite, but wide.

Code-mode: the part most readers don't expect​

The default mental model of an agent is a turn-taker. Think, call one tool, read the result, think again. That model is a ceiling, and most of it is self-imposed.

The preamble teaches a three-tier ladder instead:

1. Answer directly — no tool needed.
2. Call a tool directly — for ONE discrete action.
   Don't wrap a one-shot call in code.
3. Write and run code — for orchestration: chaining calls,
   looping, fetching in bulk.

The rung that matters is the third, and the guidance that makes it click is blunt: "Calling the same tool across a list is a LOOP — write it once in code." An agent asked to touch forty records shouldn't take forty turns. It should write the loop. The Python kernel persists across cells, so state carries between steps — fetch in one, transform in the next, act in a third, without rebuilding the world each time.

Tier two matters as much, in the other direction. The failure mode of a capable agent is wrapping a single API call in eleven lines of Python because code feels more thorough. "Don't wrap a one-shot call in code" is the counterweight. The ladder is a ladder — you're supposed to stop climbing when you've reached the rung that fits.

The asymmetry, stated plainly​

Widen what an agent can do once it's running. Narrow the number of ways it can start running. Those pull in opposite directions and they should — they're protecting different things.

The action space is bounded by the agent's judgment inside a run you initiated, observed under permissions you set. Getting it wrong costs you one bad run. The trigger space is bounded by nothing except the entry points you built, and every one you add is a door someone else's system can knock on at a time you didn't choose. Getting that wrong costs you the property that made the whole thing governable.

So: three triggers, one path, a real computer, and permission to write a loop. If you want an agent to react to something we don't trigger on, poll for it from a schedule. It's less elegant. It's a mechanism you can hold in your head, and after enough production systems, that stops feeling like a consolation prize.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.

From the outside, creating an agent looks like filling in a form. You describe what you want, you look at a preview, you click a button. Thirty seconds, maybe less. The interface is doing everything it can to feel like nothing much is happening.

That impression is the product working correctly and it is also completely wrong. The interesting engineering in an agent-creation path isn't the form — it's the handful of seconds after you confirm, where the system decides whether you're getting a working agent or a plausible-looking broken one. Those seconds are made of validation, versioning, and a few unglamorous atomicity decisions.

This is a walk through ours, in the order the system experiences it, and an argument about what a well-built creation path owes you.

Three doors​

There are three shipped ways to get an agent in CatalEx. You can talk one into existence through a chat wizard — the "from scratch" path. You can pick from a template catalog of 159 templates across 13 categories, weighted toward the work people actually automate: 41 specialized, 30 marketing, 29 engineering, then a long tail through design, sales, testing, support, finance, and product. Or you can do nothing at all: every account is provisioned an agent at signup.

The three doors converge on the same write path. That's the first design decision worth naming — the wizard is not a privileged client. It ends up calling the same create contract as everything else, which means the guarantees below are guarantees for all three, not features of one nice UI.

The wizard refuses to let you one-shot it​

The "from scratch" flow is a three-phase state machine:

type Phase = 'chatting' | 'synthesizing' | 'reviewing';

// User turns only. One assistant turn can span several streamed
// messages, so counting messages would let a single exchange
// masquerade as a conversation.
const MIN_TURNS_FOR_CREATE = 3;

In phase one you're talking to a purpose-built system agent whose only job is to interview you. The "Create Agent" button doesn't exist until you've taken at least three turns. Not three messages — three user turns, because a single assistant reply can arrive as several streamed messages and counting those would let one exchange impersonate a conversation.

The constant is a product opinion wearing a type annotation. You cannot one-shot an agent definition here, by construction. The reason is not paternalism; it's that a one-line description underdetermines an agent. "Write my release notes" doesn't say which repository, what tone, who reads them, what happens when a release has no user-facing changes. A model handed that will invent answers, and the invented answers are exactly the ones you'll be debugging in three weeks. Forcing a conversation front-loads the questions to the moment when answering them is cheap.

The interviewer runs under its own budgets: 15 turns, 50,000 tokens, 120 seconds of wall clock. An agent whose job is to elicit requirements should not be able to elicit them forever. Budgets on a helper agent aren't a safety measure so much as a design constraint — they force the interview to converge.

Synthesis, and a menu on disk​

Hit the button and you're in phase two. The synthesizer streams over SSE, and a live checklist tracks it through plan_snapshot frames. The seeded plan is exactly five steps: understand request, discover tools, draft instructions, draft KPIs, validate definition. You watch them tick.

The second step is more interesting than it sounds. The synthesizer discovers which tools are available by reading a seeded file on disk — once — rather than calling a discovery tool. Its prompt says so plainly: "There is no tool_catalog tool to call."

Put the menu on disk instead of spending a tool call on it. A tool call for a catalog is a round trip, a chunk of context, and a step in the loop, all to retrieve something that was static the whole time. If the model needs a fixed reference to do its job, hand it the reference. Save the tool calls for the things that actually require going somewhere.

From there the synthesizer writes into a fixed skeleton — About, Role, Core Workflows, Communication Style, Constraints, Decision-Making, Output Format — targeting 400–800 words across five to eight sections. A skeleton is not a creative restriction. It's what makes the output diffable, reviewable, and comparable across 159 templates and every agent a user writes by hand.

Phase three is an editable preview. Nothing has been written yet. You can rewrite any of it before it becomes real.

The part that matters: what happens on confirm​

Here's the write path.

First, every requested tool key is validated against the live catalog. Not a cached list, not the list the synthesizer thought it had. The reason is in the code comment: this guards against model hallucination, because a synthesizer that emits a plausible-but-wrong tool key "would silently produce an agent with 0 tools at runtime". Invalid keys come back as a 400 with the offending keys named.

That sentence is the best single lesson in the whole path, so let it land. A hallucinated tool key is not a loud failure. It doesn't throw. It produces an agent that looks completely fine in the preview, saves without complaint, appears in your list with a green dot, and then does nothing useful when you run it — because the tool it was told to use resolves to nothing. The failure surfaces days later as "the agent seems dumb."

Second, the create contract requires an explicit quota field. There's no default. A caller cannot create an agent without stating the cap, because — from the code — the service "never creates without an explicit cap so the invariant 'every create is cap-checked' holds at the contract layer."

Encode invariants in the contract, not in a caller's good intentions. A default would work fine right up until the day someone adds a fourth creation path, and then the invariant is a comment in a file nobody opened.

What isn't in the row​

The agent table holds what you'd expect — name, instructions (a DB CHECK caps them at 16,384 characters), description, tags, status (CHECK IN 'active', 'paused', 'draft'), KPIs as JSONB, the memory and evolution flags, run counters, current_version, template_id. The absences are more informative.

There is no model column. Model choice is a service-level concern, not a per-agent field. Bake a model into the agent row and every agent becomes a small monument to whatever was best the week it was created, and upgrading the fleet becomes a migration.

There is no schedule column. Schedules live in a separate service, keyed by agent id. An agent is a definition of behavior; when it runs is somebody else's question. Fusing them into one row is how you end up with a scheduler you can't change without touching the agent model.

Version 1, or nothing​

If the row is inserted, two more things must happen: build version 1 — a snapshot of instructions, KPIs, memories, and tools — and pin it. If either step fails, the agent row is rolled back and the API returns 503 with a Retry-After. You never get a half-created agent, and you never get an agent at version zero whose first edit has nothing to diff against.

The 503 is deliberate. Failing the whole create and telling you to try again is strictly better than handing back a live-looking agent with no version history, because the second one is a debugging problem you inherit silently.

And now the honest seam. Schedules and tool bindings attached at create time are not covered by that guarantee. They're best-effort — attempted, and on failure logged and swallowed. Which means a binding failure can produce a live agent that's missing a tool. We know. The v1 pin is load-bearing for correctness, so it's atomic; the bindings are recoverable by re-attaching, so they're not. It's a real seam, chosen rather than overlooked, and worth stating plainly because every atomicity boundary is a decision about which failures you'd rather have.

Lex, and matching on provenance​

The agent you get at signup is named Lex. Lex is not a special row. It's an ordinary agent provisioned from the research-assistant template, subject to the same validation, the same v1 pin, the same everything.

The code never identifies it by name. It matches on template provenance, and the comment explains why: "What makes it 'built-in' is provenance, so that's what we match on — never the display name, which the user is free to change." When a migration needed to rename the default agent, it only touched rows still carrying the stock name. If you'd renamed yours, it was left alone.

Match on the immutable fact, not the mutable label. Display names are user data. Any logic keyed on one is a bug waiting for the first person who personalizes something.

The provisioning itself is best-effort by design. Every failure path returns 200 with provisioned: false and a reason. Signup does not fail because agent creation did — the account is the thing that must exist; the starter agent is a convenience, and convenience never gets to block the front door.

What a creation path owes you​

Validate the model's output against the live world before you persist it. Make the invariant a required field, not a default. Draw the atomicity boundary consciously and say where it is. Keep the mutable labels out of your logic.

None of this shows up in the form. That's the point — the whole job is to make sure that the thing you clicked "create" on is the thing you actually got.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.

Every team that builds seriously with AI walks the same road, usually without noticing. It starts with a chat window and a good feeling, and — if the work survives contact with production — it ends somewhere far more disciplined. This is a field note on that road: the three stages we see teams pass through, what each one is genuinely good at, and the specific failure that forces the jump to the next.

We are writing it down because the stages are easy to confuse. "Vibe coding," "spec-driven development," and "loop engineering" get used loosely, as vibes themselves. They are not interchangeable moods. They are three different answers to one question — who is responsible for correctness, and how is it checked? — and the answer changes as the stakes rise.

The short version​

Each row is a reaction to the row above it. Read top to bottom, it is a story about moving responsibility for correctness out of the human's head and into an artifact, then into a system.

Stage one: vibe coding​

Vibe coding is building by feel. You describe what you want in natural language, the model produces something, you look at it, and you react. If it looks right, you keep it. If it looks wrong, you nudge. The loop is fast, tactile, and — this is the important part — the only verifier is you, looking at the screen.

It is genuinely great. For a prototype, a throwaway script, a UI you can see with your own eyes, or the first exploratory hour of any project, nothing beats it. The feedback is immediate and the cost of being wrong is a re-roll.

Here is the shape of it:

You:   "Build a page that lists our open roles from the careers API."
Model: <200 lines of code>
You:   *scrolls, runs it* "Close. Make the cards clickable and sort by team."
Model: <revised code>
You:   *looks* "Ship it."

The reason it works is also the reason it doesn't scale: verification is implicit and human. You are the test suite. That is fine when "correct" means "looks right in the browser" and the whole thing fits on one screen. It falls apart the moment correctness depends on something you can't see at a glance — a concurrency edge case, a compliance rule, a contract with another service, a behavior that only shows up on the 10,000th request.

Stage two: spec-driven development​

Spec-driven development moves the source of truth out of the conversation and into a document. Instead of steering the model turn by turn, you write down what "correct" means — the behavior, the constraints, the acceptance criteria — and the model implements against that. The spec becomes the artifact you review and version; the code becomes a build output of it.

The mental shift is subtle but total. In vibe coding you review code. In spec-driven development you review the spec, and you let tests review the code.

A spec doesn't have to be heavy. Often it's a short, precise document:

## Feature: careers listing

### Behavior
- Fetch open roles from GET /api/roles.
- Show role title, team, location, and an apply link.
- Sort by team (A→Z), then by title (A→Z).

### Constraints
- Roles with `status != "open"` MUST NOT be shown.
- If the API is unreachable, render the last cached response and a stale badge.

### Acceptance
- [ ] A closed role in the fixture never appears in the DOM.
- [ ] With the API mocked to 500, the stale badge renders.
- [ ] Sorting is stable and matches the fixture snapshot.

Now correctness lives in two places you can actually point at: the constraints (what must be true) and the acceptance tests (how we'll know). The model can generate the implementation, regenerate it, or a different model can — and the tests decide whether the result is acceptable. The human reviews intent, not line-by-line syntax.

This is a large step up, and for a great deal of software it is the right and final stop. It gives you determinism where you need it: the same spec produces a result that either passes the acceptance criteria or doesn't.

So why is there a stage three? Because specs describe a destination, not a journey. A spec tells you what a correct result looks like. It says nothing about what to do when the run that's supposed to produce that result is non-deterministic — which is exactly the situation once an AI agent is doing the work at runtime, in the wild, against inputs you didn't write fixtures for.

Stage three: loop engineering​

Loop engineering is what you build when the doer is itself an AI system operating without you in the chair. The spec is still there — you still need to know what correct means — but now the thing that checks the work, catches the mistakes, and tries again has to be part of the system, because you're asleep or handling ten thousand concurrent runs and cannot be the verifier.

The unit of work is no longer a prompt or a spec. It's a loop: generate a candidate, verify it against the spec, and if it fails, feed the failure back in and correct. The loop runs until the work passes or a budget is exhausted.

Written as a control loop, the idea is almost boring — which is the point:

def run(task, spec, max_attempts=4):
    feedback = None
    for attempt in range(max_attempts):
        candidate = agent.generate(task, spec, prior_feedback=feedback)
        report = verify(candidate, spec.criteria)   # tests, evals, checks
        if report.passed:
            return candidate
        feedback = report.failures                   # what went wrong, precisely
    escalate(task, feedback)                          # hand to a human, with context

Everything hard about loop engineering is hidden inside verify and feedback. The loop is trivial. Making the verifier trustworthy is the entire discipline:

• The verifier has to be real. Assertions, type checks, and tests are the cheap layer. Above them sit evals — graded checks on fuzzy outputs where there is no single right answer, only better and worse. A loop is only as honest as its weakest check; a verifier that rubber-stamps is worse than none, because it manufactures false confidence at scale.
• Feedback has to be actionable. "Failed" teaches the model nothing. "Role r_912 had status closed but appeared in the output at index 3" tells it exactly what to fix. The quality of the feedback string is, in practice, the quality of the loop.
• The loop needs a budget and an exit. Real loops cost money and can thrash. Bounded attempts, then a clean escalation to a human with the full trace, is not a failure mode — it's the design.

This is the stage where the earlier two are absorbed rather than discarded. The spec supplies the criteria the verifier checks against. The vibe-coding instinct — generate fast, react to what you see — becomes the inner generate step, except the thing reacting is now the verifier instead of your eyes.

How we think about it at CatalEx​

Running agents in production is loop engineering as a full-time job. The platform-level features we talk about — agents that are self-correcting (they catch and fix their own mistakes mid-run), self-metriced (they pick and track the criteria they're judged on), and self-evolving (they get better from each outcome instead of plateauing on day one) — are all names for parts of a well-built loop:

• Self-correcting is the fail → diagnose → regenerate edge, made reliable.
• Self-metriced is the agent participating in defining spec.criteria, so the verifier isn't frozen at whatever we guessed on day one.
• Self-evolving is what happens when the feedback from thousands of loops feeds back into how the next generation is produced.

None of it is exotic once you see it as one control loop drawn at different altitudes. The demo is the inner box. The spec is the criteria. The loop is the product.

Where you actually are​

You don't pick a stage as an identity; you pick it per piece of work, honestly.

• Exploring, prototyping, building something you can see? Vibe code it. Speed is the whole point, and you are a perfectly good verifier for things that fit on a screen.
• Building something that has to keep being correct as it changes, with more than one person or model touching it? Write the spec. Move correctness out of your head and into acceptance criteria.
• Putting an AI system in the position of doing the work unattended, at scale, against inputs you didn't anticipate? Engineer the loop. Make verification and correction part of the system, because you won't be there to be the test.

The mistake is not being at an early stage. Vibe coding a prototype is correct. The mistake is staying at a stage after the work has quietly moved past it — shipping vibe-coded code as if it were verified, or shipping a spec as if the run that fulfills it were deterministic. Each stage exists to fix the specific way the previous one lies to you once the stakes go up.

Know which lie you can currently afford. Build the next loop before you need it.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.

"Give the agent memory" is one of those phrases that sounds like a feature and is actually an architecture. Everyone agrees agents should remember things across sessions. Almost no one agrees on what that means — and the default implementation, a vector database that everything gets dumped into, produces agents that are confidently wrong about their own past.

This is a field note on how we think about memory for agents that run in production: what the pieces are, what each one is for, and the single principle that governs all of them — an agent's memory is only as useful as what it refuses to remember.

The default that quietly fails​

The reflexive design is: embed every message, every tool result, every document, and store the vectors. At recall time, embed the current query and pull the top k nearest neighbors into the context window. This is easy, it demos beautifully, and it degrades in a specific, predictable way.

The failure isn't that retrieval "misses." It's that retrieval succeeds at the wrong thing. Nearest-neighbor search returns what is semantically similar to the query, not what is relevant to the task. Ask an agent that has processed a thousand invoices about "the Acme invoice" and it will happily surface eleven Acme invoices from three different quarters, none flagged as superseded, and splice them into context with equal authority. The model, being agreeable, will reconcile them into a confident answer that is subtly wrong.

Four kinds of memory, four different jobs​

It helps to stop saying "memory" and name the pieces. Human cognition is a loose analogy, but a useful one, because each type answers a different question.

Working memory is the current context window: the task, the recent turns, the tool outputs from this run. It is bounded, it is expensive, and it is the only memory the model can actually reason over. Everything else exists to decide what gets promoted into it.

Episodic memory is "what happened." Specific events, tied to time and outcome: on June 3rd we ran the reconciliation for Acme and it failed because the currency field was null; the human corrected it this way. Episodic memory is how an agent learns from its own history instead of repeating it.

Semantic memory is "what is true." Distilled facts that outlived the event that produced them: Acme bills in EUR; this customer's fiscal year starts in April; the approval threshold is 5,000. Semantic memory is what episodic memory becomes once a pattern has been seen enough times to trust.

Procedural memory is "how we do this here." The learned workflow — the order of steps, the checks that matter, the tool that actually works for a given task. It's the difference between an agent that re-derives the process every run and one that has a way of doing things.

The mistake is storing all four as undifferentiated embeddings. They have different write rules, different retrieval rules, and very different lifespans. A null-currency incident is an episode that should decay. "Acme bills in EUR" is a fact that should persist and be retrieved by entity, not by vector similarity to the current sentence.

Writing is a decision, not a side effect​

The most important design choice in an agent's memory is not how it retrieves. It's what it writes down, and that should be a deliberate step, not an automatic log of everything that streamed through the context.

A memory-write worth making usually clears three bars:

• It's durable. It will still be true, or still be instructive, next week. A transient tool error is not; the lesson from a recurring tool error is.
• It's decision-relevant. It changes what a future run would do. If remembering it wouldn't alter any future action, it's noise wearing a fact's clothes.
• It's attributable. It carries where it came from and when. A memory with no provenance can't be trusted, updated, or retired — it can only accumulate.

In practice this means an agent should reflect before it writes: at the end of a run, summarize what changed, promote the durable facts to semantic memory, keep the specific episode with its outcome, and drop the rest. This is the same generate-then-verify discipline that governs a good agent loop, pointed inward.

Retrieval by structure, not just by similarity​

Once memory is typed, retrieval stops being one vector search and becomes a small set of targeted ones. Facts about an entity are fetched by that entity. Recent episodes are fetched by recency and relevance, with the outcome attached so the model can weight a success differently from a failure. Procedures are fetched by task type. Vector similarity is one signal among several — useful for the fuzzy "have we seen anything like this before," dangerous as the only lens.

The other half of retrieval is budgeting. Working memory is finite and attention is not free; a context window stuffed with fourteen marginally relevant memories reasons worse than one holding the three that matter. Recall should be ranked and capped, and the cap should be small. More retrieved memory past a point makes agents worse, not better — the signal gets buried and the model starts pattern-matching on the loudest neighbor instead of the most relevant one.

Forgetting is a feature​

Here is the part that feels wrong and is essential: a good memory system spends as much design effort on removal as on storage.

Facts get superseded — the approval threshold changed, the customer switched currencies, the API you learned to call was deprecated. If memory only appends, every one of those updates becomes a contradiction the model has to resolve at read time, usually by picking one at random. So memories need to be updatable (the new fact replaces the old, with the old one retired, not deleted silently) and decayable (episodes lose weight over time unless reinforced). "Remembers everything" is not the goal. An agent that cannot forget a fact that stopped being true is worse than one with no memory at all, because it is wrong with history behind it.

This is also where trust and safety live. Memory is an injection surface: content an agent reads today can be written into memory and re-executed as instruction tomorrow. Typed, attributed, expirable memory is what lets you audit that — to answer "why did the agent believe this?" with a specific provenance chain rather than a shrug at a vector store.

Why this compounds​

The reason to do the harder version is that memory is the mechanism by which an agent gets better instead of just staying the same. A stateless agent performs the same on run 10,000 as on run 1 — it re-derives everything, repeats every avoidable mistake, and never accumulates the specific, hard-won knowledge of your domain that a good human teammate builds over months. An agent with typed, curated, decaying memory does the opposite: each run leaves the next one a little more competent, because the durable lessons persist and the noise doesn't.

At CatalEx this is what we mean when we say agents are self-evolving and carry context across sessions. It isn't a bigger vector store. It's the discipline of deciding, every run, what was worth remembering — and what the agent is better off forgetting.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.

Every team that has tried to ship an AI feature has hit the same wall. The demo works. A stakeholder asks, "is it good?" and the honest answer is a shrug. You feel like it's good. You changed a prompt and it feels a little better, or a little worse, and you cannot say which, because "feels" is the only instrument you have.

That wall is the absence of an eval. And the reason so many AI projects stall at 80% — good enough to demo, never good enough to trust — is almost never the model. It's that the team never built the one thing that turns "feels good" into a number you can defend: a way to measure whether the system actually works.

Tests answer yes/no. Evals answer how well.​

Traditional software has a comfortable property: correctness is usually binary. The function returns the right value or it doesn't. A test suite is a wall of green checks, and green means ship.

AI systems break that comfort. "Correct" becomes a spectrum. A summary can be accurate but bury the lede. An agent can complete the task but take a reckless path to get there. Two answers can both be right and one can be clearly better. The output space is open-ended, the same input can produce different outputs run to run, and the thing you care about — helpfulness, faithfulness, safety — isn't a return value you can assert on.

An eval is what you build to measure a spectrum instead of a boolean. At its core it is three things: a dataset of representative inputs, a grader that scores each output, and an aggregate score you can track over time. That's it. The engineering is entirely in making each of those three trustworthy.

Without an eval, you are flying on vibes​

It's worth being blunt about what the absence of an eval costs, because the costs are invisible until they compound.

You can't tell improvement from regression. You tweak a prompt to fix one bad case and silently break five others you never looked at. You have no way to know, so you ship it.

You can't compare models. A cheaper model comes out; is it good enough for your task? Without an eval, "good enough" is a meeting, not a measurement, and the meeting is won by whoever is most confident.

You can't safely change anything. Every prompt edit, tool change, or model upgrade is a gamble on a system whose behavior you can only spot-check. So teams stop changing things, and the product freezes at its first acceptable version.

And you can't close the loop. This is the one that matters most to us: an autonomous agent that verifies and corrects its own work is only as honest as the verifier inside it. That verifier is an eval, running at runtime. A team that never learned to write a trustworthy grader offline has no chance of embedding a trustworthy one at runtime.

The grader is the whole game​

Anyone can assemble a dataset. The discipline is in the grader, and graders come in a rough hierarchy of cost and fuzziness.

Programmatic checks are the cheap, sharp layer: exact match, JSON-schema validity, "did it call the right tool with the right arguments," "does the code compile and pass tests." When a task has a checkable ground truth, use it. These are fast, deterministic, and free of the ambiguity that dooms softer graders. Most teams underuse them because they reach for a model before checking whether the property is mechanically verifiable.

LLM-as-judge is for the genuinely open-ended: is this summary faithful to the source, is this reply helpful, is this reasoning sound. A second model grades the first against a rubric. It's powerful and it's a trap in equal measure, because a sloppy judge manufactures false confidence at scale. The failure modes are well-documented and consistent: judges prefer longer answers, prefer the first option shown, and prefer their own style. A judge is only usable once you have evaluated the evaluator — checked its scores against human labels on a sample and confirmed it agrees often enough to trust.

Human grading is the ground truth the other two are calibrated against. It's slow and expensive, so you spend it deliberately: to build the golden set, to audit the automated graders, and to adjudicate the cases the judges disagree on. You don't scale it; you use it to make the things that scale honest.

Evals rot, so treat them like code​

An eval is not a milestone you pass once. It's a living asset, and it decays in two directions.

It decays against reality: your dataset was drawn from last quarter's inputs, and production has moved on. New kinds of requests appear; the eval doesn't cover them; your score stays green while real quality slips. The fix is a flywheel — every genuinely wrong production case that surfaces becomes a new eval example, so the dataset tracks the distribution it's supposed to represent.

And it decays against gaming: any single metric, optimized hard enough, stops correlating with the thing it was a proxy for. Optimize a helpfulness judge and you get verbose, hedging answers that read as helpful. The defense is a small panel of graders looking at different failure modes — faithfulness, concision, safety, task success — so that gaming one costs you another.

The teams that get this right run evals the way they run tests: in CI, on every change, with the score gating the merge. A prompt edit that drops the eval is a regression, full stop, the same as a failing unit test. That single move — making the number block the merge — is what converts evaluation from a report nobody reads into the thing that actually governs the system.

Why we say the eval is the product​

Point the whole picture at an autonomous agent and the stakes get concrete. The agent generates a candidate, a verifier scores it against criteria, and on failure it corrects and tries again. That verifier is an eval running in the hot path. Its dataset is the live task, its grader is your checks and judges, its score is the gate between "commit" and "try again." Every property that makes an offline eval trustworthy — real coverage, calibrated graders, resistance to gaming — is exactly what makes the runtime loop safe to leave unattended.

Which is why we don't treat evaluation as QA that happens after the product is built. The eval is the part of the product that decides whether any of the rest of it can be trusted. Build that first, and everything downstream — shipping, improving, running agents without a human in the chair — becomes possible. Skip it, and you're left with a great demo and a shrug.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.

The most expensive misconception in applied AI is that performance lives in the model. Pick the best model, the thinking goes, and the rest is glue. So teams spend their attention on which model to call and treat everything around it — the tools, the loop, the context they assemble, the way they check the output — as plumbing.

The data says the opposite. The scaffolding around the model — what we'll call the harness — routinely moves results as much as swapping the model does, and sometimes more. The same weights, handed a better harness, become a different product. If you're deciding where to spend the next month of engineering, this is the argument for spending it on the harness.

What the harness actually is​

The harness is everything between the user's intent and the model's tokens:

• Tools the model can call, and how well their inputs and outputs are shaped
• The loop — how many steps it gets, when it retries, when it stops
• Context assembly — what you retrieve and put in the window, and what you leave out
• Verification — the checks that decide whether an output is accepted or sent back for another pass
• Memory — what persists across steps and sessions

None of that is the model. All of it decides what the model can accomplish.

Same model, very different results​

The cleanest evidence comes from SWE-bench, a benchmark where an agent has to resolve real GitHub issues by editing a codebase until the project's tests pass. It's a good measuring stick precisely because the grader is objective — the tests either pass or they don't.

Watch what happens to the field's score over a couple of years, as both models and harnesses improved together:

A jump from roughly 1% to roughly 67% is not a story about weights getting 60x better. Model quality improved a great deal in that window — but a large share of the climb came from learning how to wrap the model: giving it a code-navigation tool instead of raw file dumps, letting it iterate against the test suite, managing what stayed in context. The harness matured alongside the model.

You can see the harness's contribution in isolation by holding the model fixed and changing only the scaffold. The pattern is consistent across published results: a capable model, asked to emit a patch in one shot, badly underperforms the same model given tools and a loop.

Same weights. Roughly a 40-point spread. The harness is not glue around the performance — a large fraction of the performance is the harness.

The harness compounds; the model resets​

There's a second reason to invest in the harness, and it's about time, not just scores. Model choice is a decision you re-make every few months — a new frontier model ships and you re-evaluate. Every dollar of prompt-whispering tuned to one model's quirks partially evaporates at the next upgrade.

Harness investment does the opposite. A good code-navigation tool, a well-designed verification step, a context-assembly strategy that keeps the window clean — these carry forward onto the next model, and usually get better underneath it, because a stronger model uses a good tool more effectively. The harness is the part of your system that accrues value across model generations instead of being reset by them.

This also reframes the "which model" debate. On a mature harness, the gap between the top two or three frontier models is often smaller than the gap between a good and a mediocre harness on either of them. The model is a component you swap; the harness is the product you build.

So what should you actually build?​

If the harness is where the leverage is, a few principles follow.

Make it model-agnostic. Treat the model as a replaceable component behind an interface. When the next model ships, you want to run your eval suite against it in an afternoon, not rewrite your prompts. The teams that move fastest on new models are the ones whose harness didn't hard-code the old one.

Invest in tools and their ergonomics, not just their existence. A tool whose output is a wall of unstructured text costs the model context and attention. The same tool returning a tight, structured result is worth more than a model upgrade. Shape tool I/O the way you'd shape an API for a junior engineer who reads fast but tires easily.

Build the verification step first, not last. The loop that lets a model check and correct its own work is the single highest-leverage piece of the harness — it converts a capable one-shot model into a reliable one. Everything we've written about loop engineering and evals is really about this component of the harness.

Manage context like a budget. More retrieved text is not more capability past a point; it's noise that buries the signal. A harness that puts the three right things in the window beats one that puts fourteen plausible ones.

At CatalEx this is the whole thesis of the product. The platform is a harness — tools, orchestration, memory, verification, model routing — built so that the model underneath is a component you can swap as the frontier moves, while the scaffolding that actually determines whether an agent works keeps compounding. Pick your model carefully. Then spend the real effort on everything around it, because that's where the performance was hiding all along.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.

There's a tidy argument going around: skip the Model Context Protocol, just give the agent a shell and a CLI. No server to run, no schemas to maintain, and — the clincher — it saves tokens, because a command is short and a tool definition is long. It's a good argument. It's also only half right, and the missing half is where most of the cost actually lives.

This is a look at the two ways to wire an agent into a third-party system — an MCP server versus a command-line tool — with particular attention to the claim that the CLI is the cheaper one. The honest answer is "sometimes, and not for the reason people think."

Two ways to hand an agent a capability​

MCP is a protocol for exposing tools and data to an agent in a structured way. You run a server that advertises a set of tools, each with a typed schema for its inputs and outputs. The agent's runtime loads those definitions, the model sees them, and it calls them as structured functions with structured results coming back.

The CLI approach skips all of that. You give the agent a shell tool and tell it a command exists. It composes a command string, the command runs, and the raw stdout comes back as text for the model to read. No schema, no server — the integration surface is "the agent knows how to use a terminal."

Both work. They fail and cost differently, and the difference is almost entirely about tokens and structure.

Where the tokens actually go​

The pro-CLI token argument focuses on one side of the ledger: tool-definition overhead. An MCP server that exposes twenty tools injects twenty schemas into context on every request, whether or not any of them get used. That's real, and it's not small — a directory of chatty MCP servers can burn thousands of tokens before the model has done anything. A CLI, by contrast, costs almost nothing to advertise: one line saying the command exists.

But there are two sides to the ledger, and the other side is output.

So "CLI saves tokens" collapses into a real question: does your integration make few calls against many tools (definition overhead dominates → CLI looks cheaper), or many calls returning verbose output (output dominates → the CLI can be far more expensive)? The answer depends on the workload, not on the mechanism.

There's a reason a whole category of tooling exists to sit between an agent and its CLIs and strip the output down — filtering, summarizing, and reshaping command results before they hit the context window. (Our own developer setup leans on exactly this kind of proxy; more on that in a follow-up.) The existence of that category is the tell: raw CLI output is a token liability that teams actively engineer around. The CLI isn't lean by default. It's lean after you do the work MCP's typed outputs give you for free.

Beyond tokens: the other four axes​

Token cost is the loudest part of the debate and the least important once you've sized it. Four other differences matter more in production.

Structure and reliability. MCP hands the model typed arguments and typed results. The CLI hands it a string to compose and a string to parse. Every un-typed boundary is a place the model can get the flag wrong, misread a table, or hallucinate a field that isn't there. Structured I/O isn't just tidier; it removes whole classes of silent error.

Discoverability. With MCP, the agent can enumerate what's available and how to call it. With a CLI, it relies on training-data familiarity or on reading --help (more tokens) — and it will confidently invent flags for tools it half- remembers. Popular tools it knows well; your bespoke internal CLI it does not.

Composition. This one favors the CLI. Shells pipe, chain, and glue tools together in ways that are genuinely powerful, and a model that's good at bash can accomplish a lot with primitives you never explicitly exposed. MCP tools compose only as far as you designed them to.

Security and blast radius. A shell is arbitrary code execution by construction. Handing an autonomous agent a terminal is handing it the ability to do anything the shell can, which means sandboxing, allowlists, and audit become your problem. An MCP tool is a narrow, declared capability — the agent can do that, and only that. For anything touching production or customer data, that containment is worth a lot.

So what should you build?​

Neither wins categorically. The decision falls out of the workload.

Reach for MCP when the capability is a durable, high-value operation you'll call often; when the output should be structured for the model rather than for a human; when you need containment and audit; or when the same capability will be reused across many agents. Typed, discoverable, narrow — that's worth the server.

Reach for the CLI when the tool already exists and is well-known, when you need ad-hoc composition more than a fixed contract, when you're prototyping and don't want to stand up infrastructure, or when the operations are one-off enough that schema maintenance would cost more than it returns. Just budget for the output problem — plan to filter what comes back.

And often, build both. A common, effective shape is a thin MCP server that wraps a CLI: the model gets a typed, discoverable, containable interface, while the server internally shells out and — critically — reshapes the raw output into a compact, structured result before it returns. You get the CLI's reach and the MCP's ergonomics, and you pay the output-token cost once, in code you control, instead of on every model turn.

The uncomfortable conclusion​

"The CLI saves tokens" is true in exactly the narrow case where you have many rarely-used tools and terse, model-friendly output — and false the moment output gets verbose, which for real-world CLIs is most of the time. The tokens you save on definitions, you often hand right back on results, plus a parsing tax and a security surface.

So don't pick the mechanism for its reputation. Size your workload — call frequency, tool count, output verbosity, containment needs — and pick the interface that minimizes tokens-per-successful-task while keeping the blast radius survivable. Usually that's MCP for the durable core, CLIs for the long tail, and an output filter wherever raw command text would otherwise flood the window. The "cheaper" option is the one you engineered to be cheap, not the one that started that way.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.

If you want to understand why AI works the way it does today, you don't need to read a hundred papers. You need to read about five, in order, and understand what each one changed. Everything since is, to a first approximation, a refinement or a scaling-up of ideas these introduced.

This is that list — the five we'd hand a strong engineer joining the field. Not the five "best" papers by citation count, but the five that each moved the ground under everyone's feet. Read in sequence, they tell one continuous story: how we got from machine translation to agents.

1. Attention Is All You Need (2017)​

Vaswani et al. · June 2017 · arXiv:1706.03762

Why it mattered: it invented the Transformer, the architecture underneath every large model since.

Before this, sequence models were recurrent — they processed tokens one at a time, which made them slow to train and forgetful over long inputs. The Transformer threw out recurrence entirely and replaced it with self-attention: a mechanism that lets every token look directly at every other token in one step. That single change did two things. It let models learn long-range relationships that recurrence kept losing, and — because attention is parallelizable — it let training scale across hardware in a way recurrence never could. The paper's title was a swagger, and it earned it. Almost nothing that came after is possible without it; when people say "an LLM," they mean a very large Transformer.

2. Language Models are Few-Shot Learners — GPT-3 (2020)​

Brown et al. · May 2020 · arXiv:2005.14165

Why it mattered: it showed that scale alone unlocks a new capability — learning from the prompt, with no training.

GPT-3 took the Transformer and made it enormous: 175 billion parameters trained on a large slice of the internet. The headline wasn't the size, though — it was in-context learning. Give the model a few examples of a task in the prompt, and it performs the task, with no fine-tuning and no weight updates. This reframed the entire practice of using models. You stopped training a model per task and started prompting one general model, and "prompt engineering" became a discipline overnight. It's also the paper that made scale the field's central bet: make the model bigger, and qualitatively new abilities appear.

3. Chain-of-Thought Prompting (2022)​

Wei et al. · January 2022 · arXiv:2201.11903

Why it mattered: it showed that asking a model to show its work unlocks reasoning it otherwise can't do.

Large models were oddly bad at multi-step problems — arithmetic, logic, word problems — even as they aced other tasks. The fix turned out to be almost embarrassingly simple: prompt the model to produce intermediate reasoning steps before the final answer, and accuracy on hard problems jumps. "Let's think step by step" became a genuine capability unlock, not a party trick. More deeply, it revealed that a lot of a model's competence is latent — present in the weights but only accessible if you give it room to compute. Every "reasoning model" that followed is a descendant of this observation.

4. Training Language Models to Follow Instructions with Human Feedback — InstructGPT (2022)​

Ouyang et al. · March 2022 · arXiv:2203.02155

Why it mattered: it turned raw, unruly language models into helpful assistants — the step that made the technology usable by everyone.

A raw pretrained model predicts likely text; it does not, by default, do what you ask. InstructGPT introduced RLHF — reinforcement learning from human feedback — to close that gap: humans rank model outputs, a reward model learns those preferences, and the language model is tuned to produce what humans prefer. The result was startling: a much smaller RLHF-tuned model was preferred over the giant base model, because it was actually trying to be helpful. This is the paper that made the assistant paradigm work, and it's the direct methodological ancestor of ChatGPT and every chat model since. Alignment stopped being a safety footnote and became the productization step.

5. Training Compute-Optimal Large Language Models — Chinchilla (2022)​

Hoffmann et al. · March 2022 · arXiv:2203.15556

Why it mattered: it corrected how the whole field spends its compute — and proved most big models were trained wrong.

The prevailing wisdom after GPT-3 was "bigger is better," so labs raced to add parameters. Chinchilla showed that for a fixed compute budget, parameters and training data should scale roughly together — and that the giant models of the day were badly undertrained, starved of data relative to their size. A smaller model trained on more tokens beat a larger one trained on fewer. This reshaped training strategy across the industry: data quality and quantity became first-class, and "compute-optimal" entered the vocabulary. It's why the strongest models aren't simply the biggest ones.

Where the story goes next​

Read those five and you have the spine: an architecture that scales (1), the discovery that scale creates capability (2), a way to summon reasoning from it (3), a way to make it helpful (4), and a way to train it efficiently (5).

Everything we work on sits one step past the end of this list. Once models could reason and follow instructions, the obvious next move was to let them act — to call tools, observe results, and loop — which is the agent paradigm and the line of work (ReAct, Toolformer, and what followed) that this blog mostly lives in. If you want the sequel to this reading list, that's where it starts. But start here. These five are the grammar; the rest is composition.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.

When you build software with an AI agent day to day, the thing you run out of first isn't intelligence — it's context. The window is finite, attention inside it isn't free, and a session that starts sharp gets duller as it fills with the exhaust of the work: verbose command output, whole files read to find one function, stale documentation, the residue of three tasks ago.

So a real part of building CatalEx is context hygiene — keeping the model's window full of signal and empty of everything else. This is the setup we actually run, and how each piece is configured. The theme underneath all of it is the same one from our post on harnesses: spend context like a budget, and make every token earn its place.

The principle: tokens-per-successful-task​

It's tempting to optimize the wrong number. The instinct is to minimize tokens per call — shorter prompts, fewer tools. But the metric that actually matters is tokens per successful task. A window bloated with noise makes the model reason worse, which causes correction turns, which costs far more tokens than the noise you were trying to trim.

Everything below is in service of that number. Some tools cut what enters the window; some cut what the model has to emit; some replace a large, vague context with a small, precise one. All of them are trying to get the task done in fewer total tokens by keeping the window clean, not just short.

RTK: filter command output at the source​

The single biggest source of context bloat in day-to-day development is command output. git status, npm install, a test run, a container log — these print for humans, in full, and every character lands back in the model's context on the next turn.

RTK (a token-optimizing CLI proxy) intercepts this at the source. It runs as a PreToolUse hook, so it's transparent — the agent runs a normal command and RTK rewrites it through a proxy that strips the output down to what matters before it ever reaches the window. In our global Claude config it's wired as:

{
  "hooks": {
    "PreToolUse": [
      { "hooks": [{ "type": "command", "command": "rtk hook claude" }] }
    ]
  }
}

The effect is invisible and large: routine dev operations come back 60–90% smaller with no change to how you drive the agent. git status returns the signal, not the decoration. When you genuinely need the raw firehose, there's an escape hatch (rtk proxy <cmd>) that bypasses the filtering — but the default is lean, which is the right default.

caveman: compress the conversation itself​

RTK trims what tools put in; caveman trims what the model and you put into the conversation. It's a plugin that switches communication into an ultra-compressed mode — the tagline is "why use many token when few do trick" — cutting roughly 75% of conversational tokens while preserving technical accuracy. It's enabled per-repo in .claude/settings.json:

{
  "enabledPlugins": { "caveman@caveman": true },
  "extraKnownMarketplaces": {
    "caveman": { "source": { "source": "github", "repo": "JuliusBrussee/caveman" } }
  }
}

The tradeoff is real and worth stating: compression is lossy, and there are moments — subtle design discussion, careful reasoning about a tricky bug — where you want the model spending tokens, not saving them. Caveman is a lever, not a law. We lean on it for the high-volume, low-nuance stretches of work and ease off when the thinking gets delicate.

context7: fresh docs instead of stale memory (or a doc dump)​

A quieter context problem is wrong context. A model's training data has a cutoff, so its memory of a fast-moving library is often months stale — and the usual fix, pasting in whole documentation pages, floods the window to answer one question.

context7 (an Upstash MCP server) solves both ends. It pulls version-specific documentation and code examples from the source on demand, so the model gets the current API for the exact version we're on, scoped to what was asked, instead of hallucinating from stale memory or drowning in a doc dump. It's enabled the same way:

{ "enabledPlugins": { "context7@claude-plugins-official": true } }

This is context management by substitution — replacing a large, unreliable context (training memory, or a pasted doc site) with a small, accurate one.

Code intelligence: read symbols, not whole files​

One of the least obvious context sinks is exploration. To change one function, an agent will happily read the whole file it lives in — and the two files it imports — just to understand the types. That's a lot of tokens to answer "what's the signature of this thing."

We run language-server plugins (pyright-lsp, typescript-lsp) so the agent has precise, symbol-level intelligence: go-to-definition, hover types, references. It can ask "what is this symbol" and get a one-line answer instead of reading three files to infer it. Across a working session that's an enormous saving, and it also makes the edits more reliable — the model is acting on the actual type, not a guess assembled from surrounding code.

Worktree isolation: keep each task's context scoped​

The last piece is structural. When several pieces of work happen in parallel, their contexts bleed into each other — and a window carrying the residue of three half-finished tasks reasons worse on the one in front of it.

We use git worktrees with lifecycle hooks so each task runs in its own isolated checkout, with its environment set up and torn down automatically:

{
  "hooks": {
    "WorktreeCreate": [
      { "hooks": [{ "type": "command", "command": ".claude/hooks/setup-worktree.sh" }] }
    ],
    "WorktreeRemove": [
      { "hooks": [{ "type": "command", "command": ".claude/hooks/remove-worktree.sh" }] }
    ]
  }
}

The context benefit is that each session's window stays scoped to a single task — its files, its branch, its environment — instead of accumulating the state of everything else in flight. Isolation is a context-management technique as much as a workflow one.

The shape of a lean setup​

Put together, the setup layers cleanly by what it targets:

• What tools emit → RTK filters command output at the source.
• What the conversation costs → caveman compresses the exchange.
• What the model knows → context7 substitutes fresh, scoped docs for stale memory.
• What exploration costs → LSP intel answers "what is this symbol" without reading whole files.
• What leaks between tasks → worktree isolation keeps each window scoped.

Every one of them is pointed at the same number — total tokens to get the task done right — and every one is configuration, not willpower. That's the real lesson: managing context while building with Claude is the same discipline as building a good agent harness, turned around and aimed at your own development loop. Set the environment up so the window stays lean, and the model does its best work for longer before the exhaust builds up.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.

Here is a shift that has already happened, whether or not your documentation noticed: a large and growing share of the readers of everything you write are no longer human. They're language models — reading your docs to answer a question, your README to use your library, your marketing page to tell someone what you do. Andrej Karpathy has been making this point for a while, and it lands harder every month: the primary reader of a lot of your writing is now an LLM, and you are probably still writing for a human.

This is a look at that idea — sometimes called writing an "LLM wiki," or knowledge for machines — and what it actually changes about how you produce docs. It's not a call to stop writing for people. It's the recognition that you now have a second, very different reader, and that reader has been quietly ignored.

The two readers read nothing alike​

A human reading your docs skims. They jump to the code block, infer meaning from a screenshot, tolerate a nav bar and a cookie banner and three paragraphs of positioning, and click around until they find the part they need. Human docs are built around all of that: visual hierarchy, interactivity, and the reader's willingness to fill gaps.

An LLM reads none of it that way. It ingests text mostly linearly. It can't click your "see also." It can't see the screenshot that contained the actual answer. It chokes on the chrome — the navigation, the boilerplate, the styling markup — and spends context on all of it. And critically, it will not tell you it got confused; it will confidently produce an answer from whatever it managed to parse, gaps included.

So the same page that works beautifully for a person can be nearly useless to a model: the key fact was in an image, the steps were split across five clicked pages, and half the tokens were spent on a header that repeats on every route.

What "writing for the LLM" actually means​

Karpathy's framing is useful because it's concrete once you take the reader seriously. Writing for the model reader means:

• Plain, linear, self-contained text. The answer should be in the words, not in a diagram or a video or three hops away. If a model reading top to bottom can't extract it, neither can the model your users are asking.
• Structure a machine can parse. Clear headings, explicit lists, real markdown — not visual layout that only means something rendered.
• Say the thing, drop the performance. The positioning fluff, the repeated chrome, the SEO padding — all of it is pure cost to a model and adds nothing to the answer. Density of fact per token is the metric.
• Completeness over cleverness. A human forgives a doc that assumes context. A model doesn't have your context; it has what's on the page.

The mental test is simple: paste the raw text of your page into a model, ask it the question your user would ask, and see if it can answer from that alone. If it can't, your human-facing polish is hiding the substance from your fastest-growing audience.

The LLM wiki, made concrete​

The clearest instantiation of this idea already exists and is easy to adopt. llms.txt — a convention proposed by Jeremy Howard — is a plain-markdown file at the root of your site that gives models a clean, curated map of what you are and where the real information lives, free of the HTML and navigation they'd otherwise wade through. It's a wiki entry for your site, written for the machine.

The broader "LLM wiki" idea generalizes that: a body of knowledge maintained specifically for model consumption — unambiguous, structured, complete, and kept current. Not a copy of your human docs with the styling stripped, but knowledge authored with the model as the intended reader. Where Wikipedia organized the world's knowledge for people to read, the LLM wiki organizes it for models to reason over.

Why it's worth doing now​

The payoff is direct and it compounds. When a user asks ChatGPT, Claude, or an AI search engine about your problem space, the answer is assembled from whatever sources the model could parse and trust. Documentation written for the model is documentation the model can actually use — which means you get represented accurately in the answer, and cited, instead of paraphrased wrong or skipped for a competitor whose page was cleaner to read.

There's an internal payoff too. The same properties that make knowledge good for an external model make it good for your own agents. An agent that reads your internal docs to do its job has exactly the LLM reader's constraints — linear, context-limited, unable to click. Knowledge authored for machines is knowledge your own automation can act on. The wiki you write for the world's models is the wiki your own agents read too.

How we approach it at CatalEx​

We try to practice this, including on the blog you're reading. It ships an llms.txt that describes what CatalEx is in plain markdown; a robots.txt that explicitly welcomes AI crawlers rather than blocking them; structured data so a model can attribute a post correctly; and posts written to be self-contained — the argument is in the prose, not locked in an image. The goal is that when a model reads any of it, it comes away with an accurate, citable understanding, not a pile of chrome it has to guess through.

None of this is exotic. It's the old discipline of writing clearly, with one update to the audience: the most frequent, most literal, least forgiving reader of what you publish is now a model. Karpathy's point is really just that — take that reader seriously. Write the wiki for the LLM, and you serve the humans who ask it questions at the same time.

Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.