Every AI platform says it is model-agnostic. Almost none of them are. Not\nbecause the teams are lying — because agnosticism isn't a property you declare\nin a README. It's a seam in the code, and a seam that nothing routinely crosses\nis a seam that has already rusted shut. You just don't find out until the day\nyou try to move it.
\n\nWe ended an earlier post by saying the model is a component you swap. That's the\neasy half. The hard half is the follow-up question every engineer asks\nimmediately: fine, how? What does the code have to look like for that sentence\nto be true a year from now, under a team that has shipped a hundred features\nsince anyone last thought about it?
\nHere is the test we've landed on, and it's blunter than most architecture\nadvice. A swappable dependency is one where something in your normal workflow\nalready swaps it. Not a test that could be written. Not an adapter that could\nbe added. Something that crosses the seam today, on a Tuesday, without anyone\ndeciding to. If the only time you'd exercise the seam is the day you need it,\nyou don't have a seam. You have a plan. Plans don't compile.
\nThis post walks four seams in our system, what keeps each one alive, and — for\none of them — an admission about where the architecture is ahead of the\ndeployment.
\nAll inference goes through a model proxy. Nothing in our application code\nrequests a vendor model by name. Callers request an alias: a default\nalias, and a cheaper-tier alias for work that doesn't need the good stuff. The\nalias resolves to a real model in configuration, not in code.
That much is unremarkable — it's the indirection everyone writes. The part that\nmakes it real is the routing table. Local development routes the same alias to\na locally-run model while deployed environments route it somewhere else\nentirely. Every engineer, every day, on every branch, runs the system with the\nalias pointed at a genuinely different provider than production uses. Nobody has\nto remember to test the indirection. You cannot get your work done without\nexercising it.
\nThat's the whole difference between a seam and a wish. The indirection isn't\nkept honest by discipline or by a lint rule. It's kept honest by the fact that\nthe local loop would break the instant someone hard-coded a model, and it would\nbreak for the person who did it, in the first minute.
\nThe consequence shows up in the schema. There is no model column on an\nagent. An agent is a definition — instructions, tools, KPIs — and deliberately\nnot a model binding. Model choice is a service-level concern, resolved at\ndispatch, not a property an agent carries around. The reasoning is mechanical\nrather than philosophical: if every agent row named a model, \"swap the model\"\nwould be a data migration across customer rows instead of a config change. The\nseam would technically exist and be practically unusable, which is the most\nexpensive kind of abstraction to own.
\nThe honest part. The architecture is model-agnostic. The deployment today is\neffectively single-model — both aliases currently resolve to the same model. The\nseam is maintained and it is exercised, but we are not running a heterogeneous\nfleet behind it, and saying otherwise would be exactly the unearned agnosticism\nthis post is about. What we've bought is optionality, and the price of\noptionality is that you keep paying for it before you use it.
Agents run code in a box. The orchestrator has never met one. It depends only on\na Protocol — an interface — and three adapters satisfy it: a fake provider\nused locally and in CI, a microVM provider that is the real one, and a\nconfidential-computing provider aimed at stricter deployments.
\nThat third one deserves a flag before it does any work in this argument: it is\naspirational. The Protocol is written for it. The adapter is not built. We\nare not shipping confidential computing today, and it appears here only because\nof what it did to the interface.
\nWhich was this: the Protocol was designed against the harder provider so all\nthree satisfy one contract. Write your interface against the most constrained\nimplementation — including one that doesn't exist yet — and the easy cases fall\nout of it for free. Write it against the easy case and the hard one will not fit\nlater, no matter how much you want it to. The unbuilt adapter earned its keep by\nbeing a constraint during design, which is a legitimate use for a thing that\ndoesn't exist.
\nThe mechanism that keeps it from degrading is capability declaration. Adapters\nadvertise what they offer; the orchestrator branches on the capability, never on\nthe provider:
\n@dataclass(frozen=True)
class Capabilities:
sudo: bool # can the agent take root inside the box?
scratch_disk: bool # writable disk beyond the process?
resumable: bool # can a box be suspended and woken later?
class SandboxProvider(Protocol):
\"\"\"A place an agent can run. Nothing above this line knows which one.\"\"\"
def capabilities(self) -> Capabilities: ...
async def create(self, spec: BoxSpec) -> BoxHandle: ...
async def exec(self, box: BoxHandle, cmd: Command) -> ExecResult: ...
async def destroy(self, box: BoxHandle) -> None: ...
if provider == \"x\" is how a seam dies. It always starts as one line, in one\nplace, for one good reason, and then it's forty lines in nine files and the\nabstraction is now decoration around a switch statement. Branching on\ncaps.resumable instead means a new adapter answers a question rather than\njoining a conditional.
The Protocol also documents what must not cross it. For every adapter, the\nforbidden leaks are written down: the box's only external egress is the tool\ngate and the model proxy; tool results follow the preview-plus-reference\ncontract; delivery and box lifecycle stay behind the interface. Naming what must\nnot pass through an interface is rarer than naming what must, and it's more\nuseful — leaks are how abstractions rot, and they rot from the inside, by\naddition, one convenient exception at a time.
\nAnd the fake provider deserves its own beat. The entire system runs in CI with\nno external sandbox at all. That's not a testing convenience; it's the strongest\navailable proof the seam holds. An abstraction you can't run without is not an\nabstraction — it's a wrapper. Every green CI run is a swap of the most\ninfrastructure-heavy dependency we have, performed by nobody, hundreds of times\na week.
\nTwo sourcing paths exist for third-party tools: sometimes we wrap a vendor's API\nourselves, sometimes we proxy a remote server someone else runs. These are\npeer-equal behind the same registry shape, and the line that matters is that\nagents can't tell them apart at call time. The agent-facing contract doesn't\nleak the sourcing decision — which means the sourcing decision can be revisited\nwithout touching a single agent definition.
\nThe registry is convention, not registration. Every vendor is a directory whose\nname is its slug, with a fixed file layout inside; discovery walks one level\ndown and finds it. Adding a vendor is \"write a new package + decorate\", not\n\"edit a shared elif chain\". A central registry file is two bad things at once: a\nmerge conflict and a coupling point. A naming convention is neither, and it\ncan't drift from itself.
\nMetering follows the same instinct. A tool is metered if its canonical name\nisn't in the local-tools set — vendor-agnostic by construction, because\nthere is no per-vendor list anywhere for anyone to forget to update. A\ncanonicalizer collapses the two surface spellings of the same tool into one\nidentity, so the accounting doesn't care which call style the model happened to\nuse. And the tool catalog itself is data — in the database, read at runtime —\nnot code. The general shape: when correctness depends on a list staying current,\ndelete the list.
\nThe last one is smaller and travels further. A dispatch-shape change shipped\nconsumer-first: the client was taught to speak both the old and the new\nresponse shapes before any environment switched. Once the consumer is bilingual,\nthe two deploys can land in either order, and nothing is holding its breath.
\nAn interface change is two deploys, and the consumer goes first. That's the\nentire technique. It's worth saying out loud because the seam metaphor is\nusually static — a line in the architecture — when in practice a seam is also a\nthing you must be able to move without stopping the world. A boundary you can\nonly change during a maintenance window is a boundary you'll stop changing.
\n| Seam | What it abstracts | What crosses it routinely | How it would rust |
|---|---|---|---|
| Model alias | Which model actually serves a request | Local dev routes the alias to a local runner; deployed routes elsewhere | A model column on the agent row |
| Sandbox Protocol | Where agent code executes | CI runs the whole system on the fake provider | if provider == \"microvm\" |
| Integration registry | Whether we wrapped it or proxy it | Every vendor added the same way — a directory, a decorator | A central elif chain; a per-vendor metering list |
| Dispatch shape | The runtime↔client response contract | Consumers taught both shapes before the switch | Changes that require both sides to land at once |
Read the third column first. That's the only one that's load-bearing. The first\ntwo columns describe every model-agnostic architecture diagram ever drawn,\nincluding the ones belonging to teams who'd need three weeks to change a model.
\nModel-agnosticism is not a property you announce. It's a maintenance cost you\nagree to pay, forever, in exchange for never being trapped. You pay it in a\nProtocol that's harder to write than the concrete class would have been. You pay\nit in a fake provider that must be kept faithful. You pay it in a capability\nflag where a provider check would have taken ten seconds. You pay it on days\nwhen nothing is being swapped, which is most days, which is the point.
\nThe way to know whether you're actually paying it is simple, and you can run it\non your own system this afternoon: name the last time something crossed the\nseam. Not a plan to. Not a test that would. An actual crossing, with a date.
\nIf you can't name one, the seam is decorative. The model isn't a component you\nswap — it's a component you married.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/the-model-is-a-component", "title": "The Model Is a Component: Designing for the Swap", "summary": "Model-agnosticism isn't a claim, it's a seam — and a seam nothing crosses has already rusted shut. How we keep four seams honest: the model behind an alias, the sandbox behind a Protocol, integrations behind one shape, and rollout.", "date_modified": "2026-07-13T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "Model Routing", "Agent Harness", "Agent Engineering" ] }, { "id": "https://blog.catalex.co/deny-by-default", "content_html": "A chatbot that says something wrong is embarrassing. An agent that sends\nsomething wrong is a fact about the world. The moment an agent can send, post,\ncreate, or delete, it stops being a text generator and becomes a principal in\nyour system — with credentials and reach, taking actions other people receive.
\nThe instinct at that point is to write a better system prompt. Tell the model to\nbe careful. Tell it to confirm before sending. This doesn't work, and it doesn't\nfail in an exotic way — it fails because a system prompt is a suggestion\naddressed to the thing you're trying to constrain.
\n\nPermissions for agents are an ordinary authorization problem. Same shape as any\nother: a caller, a resource, an action, a decision. But the caller has two\nproperties you don't usually design for. It is non-deterministic — the same\ninput can produce a different tool call tomorrow. And it is persuadable — the\ncontent it reads can change what it decides to do. A retrieved document, a\nmessage body, a web page: all of it is input, and any of it can argue.
\nSo you design accordingly. The rest of this is where the gate goes, and how it\nfails.
\nThe tool gate is the sole approval authority. Not the primary one — the sole one.\nThe sandbox where the agent's code actually runs is a pure executor, and the\ndesign intent is written down in exactly those terms:
\n\n\nThe box is a PURE EXECUTOR: every piece of tool METADATA (catalog, enabled\nbindings, allowed accounts) is resolved run-side. The box NEVER calls the gate\nto discover tools — it has no whole-catalog access and cannot list. Its tools\nare therefore fixed at creation/dispatch time.
\n
The load-bearing word is discover. An executor that can enumerate its own\ncapabilities can be talked into enumerating more of them — a persuadable caller\nplus a listable catalog is a negotiation, and you've handed the model the agenda.\nFixing the toolset at dispatch and denying the box any catalog access means the\nblast radius is decided before the model gets a turn. Whatever the run argues for\nlater, it argues from a fixed hand.
\nThe second half of that principle matters just as much: two components both\nmaking authorization decisions is not defense in depth. It's two sources of\ntruth, and one of them will be stale. Depth is layers that narrow — a gate,\nthen a scope, then an audit. Two peers independently answering \"is this allowed?\"\nisn't depth; it's a race where the wrong answer only has to win once.
\nEvery permission resolver is a cascade. Check the user's setting. Check the\nbinding. Check the catalog. Check the tier above. And then, at the bottom of the\nfunction, after every rule has declined to match, there is a return — and that\nreturn is your actual security posture. Everything above it is decoration.
Ours ends with an explicit default: manual approval required. Not \"allow, because\nwe didn't find a rule against it.\" Require a human, because we didn't find a\nrule.
\n# Nothing matched: no user override, no binding rule, no catalog entry.
# This is not \"we found nothing to object to\" — it is \"we know nothing.\"
# An unknown action is exactly the case where a human should look.
return GuardrailDecision(
requires_approval=True,
tier=\"default\",
reason=\"no matching rule; unknown actions require manual approval\",
)
The failure mode this prevents is quiet. A new toolkit gets added, nobody writes\na rule, and a permissive default means it's live and unsupervised on day one —\nnot because anyone decided that, but because a fallthrough decided it. Write the\ndefault branch first, and the rest of the cascade can only ever loosen from a\nsafe floor.
\nThe tempting move is a blocklist: enumerate the dangerous actions and gate those.\nThis is an infinite, ever-growing set, maintained by hand, always one integration\nbehind. You lose that race permanently on the first day you run it.
\nSo we inverted it. The platform seeds an explicit auto-allowlist of 2,298 read\nactions across 86 toolkits, derived mechanically from the integration catalog's\nown schemas. Anything not on the list is manual by default. The list grows when\nthe catalog grows, by the same mechanism, with no committee.
\nTake a mail toolkit as the concrete case. Exactly 27 actions are auto-allowed\n— every one of them a read. Fetch a message. List drafts. Read a profile. List\nhistory. Zero send actions. Zero create, reply, or delete. An agent that decides\nto send mail surfaces an approval, always, unless the user has explicitly chosen\notherwise for themselves.
\n| Action class | Default | Why |
|---|---|---|
| Read / list / fetch | Auto-allowed | Recoverable. Re-running it just repeats the request. |
| Send / post / reply | Manual approval | Lands in someone else's inbox. Not undoable. |
| Create / update | Manual approval | Changes state others depend on. |
| Delete | Manual approval | The one with no inverse at all. |
| Anything unrecognized | Manual approval | We don't know what it is. That's the point. |
The rule underneath the table is the useful part: reads are recoverable, writes\nare not. That asymmetry — not a threat model, not a severity score — is the\ncleanest line available, and the only one we've found that a machine can draw\nfrom a schema instead of a human drawing it from judgment in a meeting. Severity\nscores need someone to assign them and someone else to disagree. \"Does this\nmutate anything?\" is a property of the API.
\nDefaults are written for onboarding, not for you. Wiring up a chat platform's\nbot, we hand-assembled the permission integer bit by bit rather than taking the\nintegration's suggested default — because the convenient default granted full\nadministrator rights. Least privilege is work. The default is someone else's\noptimization, and their objective was time-to-first-success, not your blast\nradius.
Guardrails resolve from the invoking user's own settings plus the catalog.\nNothing another person can edit changes your approval behavior.
\nThis sounds like a small storage decision. It's actually the whole multi-tenant\nargument. A shared \"this agent is trusted\" flag means one teammate's convenience\nsilently becomes everyone's standing exposure — including people who never saw\nthe toggle and have no reason to suspect their approvals stopped firing. \"I trust\nthis agent\" is a statement about you. It has to be stored against you, or it\nisn't the statement you thought you were making.
\nConfiguration history is append-only. A change soft-deletes the old row and\ninserts a new version, so every past configuration stays queryable — you can ask\nwhat the rules were at any moment, not just what they are now. And the gate logs\neach decision with the tier that won it. That detail earns its keep: \"the agent\nwas allowed\" is not an answer to any question worth asking. \"The agent was\nallowed by this rule, set by this person, at this time\" is.
\nHere's the one that took real thought.
\nTool calls carry idempotency keys, so retries are safe. Standard. But an approval\ngate returns HTTP 200 with no error — and it is not an executed call:
\n\n\nAn
\napproval_requiredGATE response is HTTP 200 with noerrorkey but is NOT\nan executed call — the tool was suspended for the user's decision and never\nran. Recording it would dedup the REAL execution on the resume re-invoke (the\nbox presents the SAME idempotency key after approval), so the approved tool\nwould return the \"already ran\" duplicate guidance with no result.
Read that failure through: a user approves an action, the resume arrives with the\nsame idempotency key, the store says \"seen it,\" and the tool the human explicitly\napproved never runs — while everything reports success. The gate would have\nconverted an approval into a silent no-op.
\nSo the idempotency store records only completed calls. A gate is a pause, not\nan outcome. Approval ids are deterministic — {run_id}:{tool_call_id} — so the\nsame decision resolves the same call across a resume. The general lesson: any\n\"we've seen this\" cache needs a crisp definition of seen, and \"returned 200\" is\nnot it. Suspension isn't completion.
When a genuine duplicate is detected, the response is HTTP 200 carrying prose.\nNot a 409. Not an error code. And the reason is precise: a raw error would read\nas a transient failure the model might blindly retry.
\nDUPLICATE TOOL CALL — this exact call already completed on an earlier attempt;
your previous response was most likely lost in transit. The call ran exactly
once and was NOT lost. If this tool WRITES or takes an action (send, create,
update, delete, post, pay, …) it is already done — do NOT call it again. If it
is a READ / query and you still need the data, issue a NEW call to fetch it
again...
This generalizes further than it looks. When your caller is a language model,\nyour API's error strings are prompt engineering. A 409 is a token sequence whose\ntraining-data association is transient conflict, back off, try again — so it\nteaches the model to retry the send. A sentence teaches it what happened. Status\ncodes were designed for clients that parse; you now have a client that reads.
\nApproval is binary per action: auto or manual. There's no \"draft for\napproval\" tier, no read/draft/send gradient. That's a real limit, and it's worth\ndefending honestly rather than pretending it's finished design: a binary gate is\nlegible and enforceable — you can describe it in one sentence and verify it in\none query. A gradient is more expressive and much easier to get subtly wrong. A\npermission system that's subtly wrong is worse than one that's coarsely right,\nbecause you'll believe it.
\nApproval scopes are defined for both a single call and a whole session. Only the\nsingle-call scope is wired today.
\nAnd the one nobody wants to say out loud: an approval gate's real enemy isn't\nbypass, it's fatigue. A gate that fires on everything trains people to click\napprove without reading — worse than no gate at all, because it manufactures the\nappearance of oversight. You get the audit trail, the sign-off, the timestamp,\nand none of the thinking. That is the actual argument for allowlisting reads. Not\nconvenience. Preserving the meaning of the clicks that remain.
\nAn agent that can act needs a boundary that exists outside its own reasoning. Not\na stronger instruction, not a more emphatic prompt, not a self-check the model\nperforms on itself — those all live inside the thing you're trying to constrain,\nand a persuadable caller auditing its own persuasion is not a control.
\nThe model can be talked into things. That's not a defect; it's most of what makes\nit useful. A deny-by-default gate that fails closed at the bottom of the\nfunction, allowlists only what's recoverable, resolves against the person who\nactually asked, and logs who allowed what and when — that cannot be talked into\nanything. It doesn't listen. That's the feature.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/deny-by-default", "title": "Deny by Default: Permissions for Agents That Act", "summary": "The moment an agent can send, post, or delete, it becomes a principal in your system — and a system prompt is not an access control. How we designed a deny-by-default tool gate, and the places it gets subtle.", "date_modified": "2026-07-12T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "Agent Governance", "AI in Production", "Agent Engineering" ] }, { "id": "https://blog.catalex.co/judging-the-judge", "content_html": "We argued that you can't ship what you can't measure. Here's the uncomfortable\nhalf of that sentence: your measurement is a model too, and it is also wrong\nsometimes.
\n\nWe score agent runs with an LLM judge — it reads a run and grades it against the\nKPIs that agent was given. That judge is treated, by everyone downstream of it,\nas an instrument. A thermometer. Numbers come out; people make decisions. But a\njudge is not a thermometer. It's a second fallible system wearing a lab coat,\nand every number your team trusts is resting on it.
\nSo we built an eval of the judge. An LLM-as-judge eval of an LLM-as-judge. This\npost is what it found — including the part where the judge was starving for\nevidence and we nearly rewrote the rubric instead.
\nThe first design decision is the one most people get wrong, and it's not\nsubtle: the eval drives the exact production path. Same prompt file, same input\nrenderer, same parser, same model alias.
\nThat sounds like hygiene. It's load-bearing. The moment your eval builds its own\ntidy little prompt and calls the model directly, it is measuring a system that\ndoes not exist. Every divergence — a renderer your eval skips, a parser it\nreimplements more forgivingly — is a place where the eval is green and production\nis not. You didn't evaluate your judge. You evaluated a fictional cousin of it\nthat happens to share a rubric.
\nThe suite started at 17 synthetic scenarios across 13 failure families, plus 7\npairwise ordering rules. It later grew to 24 scenarios and 9 orderings. Real-run\nfixtures with hand-labeled score bands sit alongside the synthetics, because\nsynthetic scenarios test the failure you imagined and real runs test the ones you\ndidn't.
\nThe failure families are the part worth stealing. None of them are obvious until\na judge has burned you:
\nCalibration — does the score land in the band a human would give?\nCorrectness-trap — a confident, articulate, wrong answer must score near\nzero. This is the one that separates a judge from a style critic.\nSycophancy — the agent claims success; the tool results say it failed. Who\ndoes the judge believe? A judge that takes the agent's word for it is a\ncompliance machine, not a grader.\nGrounding — fabricated citations.\nInjection — instructions embedded inside tool results, aimed at the judge.\nYour judge reads attacker-influenced text by design; treat it that way.\nEstimation — unmeasurable KPIs like \"time saved.\"\nVerbosity bias — does length buy score?\nRelevance — beautiful output, wrong topic.\nTruncation — a display artifact the judge must not read as incomplete work.
\nPlus over-penalty, trajectory, structural load (eight KPIs at once), rubric\nanchoring, and autonomy. Thirteen families, because a judge doesn't fail in one\ndirection — it fails in whichever direction your rubric was quietest about.
\nIf you take one thing from this post, take this one.
\nMost judge metrics are absolute. In-band rate asks whether a score landed where a\nhuman would put it. That's useful and it is fragile — it drifts with every\nrubric edit, every anchor you reword, every criterion you add. Tighten the rubric\non Tuesday and Monday's calibration is partly fiction.
\nOrderings don't drift. An ordering is a pairwise constraint: run A must score\nabove run B. Not by how much. Just above. Whatever your absolute numbers are\ndoing, whatever the anchors say this week, the thorough grounded run outranks the\nconfident fabricated one — or your judge is broken in a way no amount of decimal\nprecision will save.
\nThis is a calibration-free signal, and that property is the whole point. It\nsurvives rubric changes, prompt rewrites, and model swaps, because it asks the\njudge to do the thing you actually need it to do: distinguish better work from\nworse work. A judge emitting 74.3 and 68.1 looks precise. If it can't reliably\nput the right run on top, that precision is decoration.
\nThe rest of the metrics: in-band rate, parse/coverage rate, and stddev across\ntrials — trial-to-trial consistency, which is just asking whether the judge is\nstable against itself before you ask it to be right about anything else. The\nsuite passes only when in-band ≥ 0.9, parse ≥ 0.98, and all orderings hold.\nOrderings are the one with no partial credit. Six out of seven is a failing\ngrade.
\n| Iteration | Change | In-band | Orderings | Stddev |
|---|---|---|---|---|
| baseline | — | 76.8% | 6/7 | 11.0 |
| 1 | renderer: final response 800→4000 chars, trail summaries 100→400 | 91.7% | 6/7 | 7.9 |
| 2 | prompt: KPI independence, fraction rule, citation matching | 94.0% | 6/7 | 5.7 |
| 3 | renderer: tool-call arguments added to the decision trail | 95.8% | 6/7 | 5.7 |
| 4 | prompt: calibration anchors reworked | 96.8% | 7/7 | 4.3 |
| 5 | judge temperature 0; empty transcripts score ≤10 | 98.2% | 7/7 | 4.0 |
| 6 | head+tail windowing + adversarial scenarios added | 91.0%† | 9/9 | 7.8 |
| 7 | prompt: truncation is a display artifact; padding earns nothing | 95.2%† | 9/9 | 5.9 |
† Iterations 6–7 run against the harder 24-scenario suite and are not\ncomparable to iterations 1–5. The suite got harder; the number went down. We\ncould have reported 98.2% and stopped at iteration 5, and it would have been the\nbetter headline. It would also have been the exact failure this post is about.
\nRead the Change column again. The two biggest wins are iterations 1 and 3, and\nneither one touched the rubric. Both are renderer changes. They changed what\nthe judge could see, not what it was asked.
\nIteration 1 moved in-band fifteen points by raising the final-response window\nfrom 800 to 4,000 characters and tool summaries from 100 to 400. Nothing about\nthe judge's instructions changed. We just stopped hiding the evidence from it.
\nCall it evidence starvation. A judge asked to verify grounding, handed\n100-character tool summaries, cannot verify anything — the citation it's meant to\ncheck got truncated three fields ago. So it does what any reasonable grader does\nwith missing evidence and a demand for a verdict: it rates good work as\nfabricated and incomplete. The rubric was fine. The rubric was never the problem.\nThe judge was reading through a keyhole.
\nIteration 3 is the same lesson in a different key: tool-call arguments were\nmissing from the decision trail, so the judge could see that a tool ran but not\nwhat it was asked. That's not a rubric gap. That's a hole in the evidence.
\nNearly everyone tunes the prompt first. It's the most available surface — a text\nfile, right there, infinitely adjustable. So teams write increasingly elaborate\ninstructions to a judge that structurally cannot see the thing they're\ninstructing it about. Before you rewrite the rubric, check whether your judge\ncan actually see the thing you're asking it about. Most judge quality problems\nare platform problems in a prompt costume.
\nTwo mechanisms keep the numbers honest over time, and both are about\ncomparability — the property you destroy by accident.
\nEach rubric carries a content hash of its static prompt parts. Verdicts\nproduced under different rubric text are never aggregated together. Change the\nrubric, get a new hash, get a new bucket. You cannot accidentally average\nyesterday's scores with today's under a different definition of \"good,\" because\nthe system won't let the two meet. Every dashboard that silently mixes verdicts\nacross rubric versions is reporting a number that means nothing.
\nThe nice detail: display-only help text is deliberately excluded from the\nhash. Improving the human-facing explanation shouldn't trigger a pointless\nre-judge of the corpus. Hash what the judge reads, not what the human reads.
\nThe same instinct shows up in rubric edits. When one rubric was hardened against\nover-tagging, the output criteria and schema were left unchanged specifically so\nscores stay comparable. Improving your judge while destroying your ability to\ncompare to last month is a bad trade — and it's one you make by accident, in a\ncommit that felt like pure upside.
\nWhen a judge over-tags \"missing tool call,\" the instinct is to write it a\nlecture. Lectures are weak. A guard tied to metadata already in front of the\njudge is strong: emit that tag only when run metadata shows zero tool calls\nand the request needed external data; never emit it when the tool count\nis above zero.
\nThat guard only works if its premise is true, so we verified in code that the\ntool count really is rendered in the block the judge reads. That step is not\noptional and it is routinely skipped. A guard whose premise is false is worse\nthan no guard, because now you've got a rule that reads as rigorous and fires on\nvibes.
\nTwo admissions, and they're the point.
\nA guard in place is not a problem fixed. Proving it requires a re-judge of\nthe corpus. Shipping the guard and declaring victory is precisely the error the\nwhole apparatus exists to catch — asserting a result you didn't measure, with\nmore confidence than the evidence supports. Judges do it. So do we.
\nEven at temperature 0, a judge is not fully deterministic across serving\nproviders. Perfect reproducibility isn't on the menu. Knowing your noise floor\nis — which is why stddev across trials is a first-class metric rather than a\nfootnote. You can't eliminate the noise. You can refuse to mistake it for signal.
\nTwo more places where the honest answer is a shape, not a number. Where tool\noutput isn't captured, the grounding criterion returns unknown — not\nfail. A judge that can't see the evidence must say so rather than guess;\nabsence of proof is not proof of fabrication. And a run that legitimately\nsuspended or failed is judged on its trajectory so far, not penalized for a final\nanswer it had no reason to produce.
\nThe eval is the product. And the judge is part of the eval — which means every\nnumber you trust is resting on a system that, until you evaluate it, nobody has\nevaluated. The judge doesn't announce that it's drifting. It just keeps\nreturning numbers, in the same confident format, forever.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/judging-the-judge", "title": "Judging the Judge: When Your Eval Is the Thing That's Wrong", "summary": "An LLM judge is not an instrument — it's another fallible system that needs its own eval. How we built an eval of our judge, and why the two biggest accuracy wins were platform fixes, not prompt fixes.", "date_modified": "2026-07-11T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "Evaluation", "Loop Engineering", "Agent Engineering" ] }, { "id": "https://blog.catalex.co/token-efficient-agents", "content_html": "Most advice about token efficiency is prompt golf. Trim the system prompt. Cut\nthe pleasantries. Replace the paragraph with a bullet. Shave two hundred tokens\noff the instructions and feel briefly virtuous about it.
\nIt's the wrong game. Two hundred tokens is a rounding error next to a single\ntool result you didn't need to look at, dragged forward through every turn of a\nthirty-turn run. The wins that matter are structural: they come from\ndeciding what never enters the context window in the first place. Not what gets\nsaid more tersely — what never gets said at all.
\nThis is a practical guide to making that decision well. Four moves, each one a\nplace where a real agent either touches something or doesn't, and then a section\non what to do about it when you sit down to build your own.
\n\nA system prompt is written once and sent every turn. So is the tool list. So is\nevery tool result you left sitting in the message history. The prompt is the\nonly one of those three you can see in a text editor, which is why it's the only\none anybody optimizes — and it's usually the smallest.
\nHere's the asymmetry that decides everything: the context window is not a\nbudget you spend, it's a budget you re-spend. A message that enters the\nhistory on turn four is present on turn five, and six, and every turn after,\nuntil something removes it. Put a 60,000-character API response in context on\nturn four of a twenty-turn run and you haven't made that decision once. You've\nmade it sixteen times.
\nPrompt golf optimizes the constant. Structure optimizes the multiplier.
\n| The naive version | The structural version | |
|---|---|---|
| Large results | Tool returns 60,000 chars into the message history | Preview + a file reference; the bulk is queried, never carried |
| Repetition | Fifty tool calls across fifty turns | One loop inside one code cell, one turn |
| Tool surface | Bind forty tools, ship all forty schemas every turn | Names in the prompt; fetch the schema you're about to use |
| Context pressure | Fight compaction, or hit it at full speed | Arrive at it rarely; let it run early when you do |
Every row is the same instinct applied to a different surface. Let's take them\none at a time.
\nOur tool layer enforces an offload contract. Any tool result over roughly 4,000\ncharacters never enters the message history at all. What the model gets back is\nan ~800-character preview and a file reference — a handle to the full\nresult, sitting on disk, which the agent queries with SQL when it actually needs\nsomething out of it.
\nThe instruction the agent carries is blunt on purpose: the bulk is NEVER in\ncontext; query it.
\nThe preview is the part people notice. The consequence is the part that matters:\na large result never enters the message history, and therefore never enters the\ncheckpoint either — so it isn't re-sent on every subsequent turn.
\nA big result you put in context isn't paid for once. It's paid for on every\nturn after. This is the single most underrated fact about agent design: the\nmessage history is not a log, it's an input, and it is re-read from scratch on\nevery single turn until something takes it out.
These reference files are read-only by instruction, and the reason is\nunglamorous and useful: they can run to tens or hundreds of megabytes. Opening\none whole would exhaust the sandbox's memory long before it exhausted anything\nelse. The file is not a smaller version of the result. It's the full result,\nsomewhere the model doesn't have to look at all of it.
\nThe agent has a real Python kernel, and it's taught a three-tier ladder for\ndeciding how to act:
\nThe rule that does the work sits at tier three: calling the same tool across a\nlist is a LOOP — write it once in code.
\nConsider fetching fifty records. Done one per turn, that's fifty round trips\nthrough the model, and — worse — fifty results sitting in the message history,\nre-sent on every turn that follows. Done in a loop inside one cell, it's one\nturn and one offloaded reference.
\n# The per-turn version: 50 turns, 50 results in history, 50 re-sends per turn.
# turn 1: get_record(id=\"r_001\") -> {...}
# turn 2: get_record(id=\"r_002\") -> {...}
# ... 48 more turns, each one carrying every result before it ...
# The in-code version: 1 turn, 1 offloaded reference, history untouched.
records = [get_record(id=r) for r in record_ids] # 50 calls, 0 extra turns
open_ones = [r for r in records if r[\"status\"] == \"open\"]
summary = {\"total\": len(records), \"open\": len(open_ones)}
print(summary) # small enough to enter context — that's the point
Only summary crosses back into the message history. The fifty records stay in\nkernel memory, and because the kernel persists across cells, they stay reachable\n— you don't re-fetch to re-use. State carries.
The tier-two caveat is real, though, and worth respecting. A single discrete\naction wrapped in a code cell is worse than just calling the tool: you've added\na kernel round trip and a layer of indirection to buy nothing. The ladder has\nthree rungs because there are three right answers.
\nEvery bound tool is a permanent line item. It's in the prompt on turn one and\nit's in the prompt on turn thirty, whether or not the agent ever calls it.
\nSo we split the tool surface in two. Tool names are all listed in the system\nprompt. Their parameter schemas are not. The agent fetches a schema only for\nthe tools it's about to use, and it's told to batch that fetch: pass EVERY tool\nyou're about to use in ONE call, instead of calling this repeatedly — each call\nis a separate step.
\nThe name list is itself capped, at 50 names, after which it truncates with a\n(+N more). That number is a judgment call, not a law of nature: long enough to\nfunction as a menu the model can browse, short enough not to become a tax on\nevery turn.
The same principle governs skills. A skill is loaded on demand, which means\nthe agent pays the token cost only when the skill is actually relevant to the\nturn. A skill you might need on one turn in fifty should not be present on the\nother forty-nine.
\nAnd here's the transferable version for anyone building on the platform: bind\nfewer tools. An agent with forty tools pays for forty tools on turn one and\nturn thirty. Not because forty is too many in the abstract, but because most of\nthose forty were bound speculatively — \"it might need to update the record\" —\nand a speculative binding is overhead on every turn that follows.
\nCompaction is the safety net, and it works. It triggers at a threshold — 180,000\ntokens — keeps only the most recent few tool outputs in full, and reports\nexactly what it freed. There's a reactive path too, forcing compaction when the\nmodel itself reports an overflow the static threshold didn't predict. And rolling\nmemory consolidation runs at around 70% of the context budget and on a turn\ninterval: deliberately before the pressure point rather than at it, because a\nconsolidation with room to be thoughtful is a better one.
\nAll of that is machinery you get for free. But compaction is not free in the\nsense that matters: something has to read the history to summarize it, and\nwhatever gets summarized is no longer there in full. The goal isn't to compact\nefficiently. It's to arrive at compaction rarely — which loops straight back\nto Moves 1 and 2. An agent that offloads its bulk and loops in code approaches\nthe threshold slowly. One that carries every result forward sprints at it.
\nThe four moves above are platform mechanics. These are the decisions that are\nyours, in roughly the order they'll bite you.
\nBind the tools the agent needs, not the tools it might need. This is Move 3\nwith your name on it, and it's the one lever where you can do more damage than\nthe harness can undo. Every speculative binding is permanent overhead on every\nturn of every run.
\nName the deliverable and the stopping condition. An agent that doesn't know\nwhat \"done\" looks like spends turns discovering it — and those exploratory turns\nare the most token-hungry kind, because each one adds to the history that the\nnext one re-reads. \"Produce a summary table of open items, grouped by owner;\nstop when every open item appears once\" is a brief. \"Look into the open items\"\nis an invitation to wander.
\nSet the KPI measurement interval to something sane. Measurement runs on\nevery Nth run, not every run. A KPI you measure every single run means a judged\nevaluation every single run — and a metric that moves slowly does not need\nmeasuring quickly. Weekly-drifting quality does not need a per-run verdict.
\nKeep KPI slots few. There are 8. Using all 8 because they're there means\njudging 8 criteria on every measured run. Pick the ones whose movement would\nactually change what you do.
\nPrefer one capable agent with a narrow brief over a broad agent that must\nfirst work out what you meant. Interpretation is a real workload, and you pay for\nit in turns before any of the useful work starts.
\nMatch the schedule to the decision it feeds. A schedule that runs hourly for\na report nobody reads hourly is the most common waste there is. It's also the\neasiest to fix — nobody has to refactor anything, someone just has to admit the\nreport is a daily report.
\nRead the four moves back to back and they collapse into one instinct. Don't\ncarry the bulk — reference it. Don't repeat a call across turns — loop it in\ncode. Don't ship schemas for tools you aren't using — fetch them when you are.\nDon't sprint at the compaction threshold — arrive slowly.
\nEvery one of those is a version of the same question: what does the model never\nhave to look at?
\nThe efficient agent isn't the one with the terser prompt. It's the one that\ntouches the least.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/token-efficient-agents", "title": "Spend Credits Like They're Yours: Building Token-Efficient Agents", "summary": "Token efficiency doesn't come from trimming your system prompt. It comes from four structural decisions about what never enters the context window at all — and how to make them when you build your own agent.", "date_modified": "2026-07-10T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "Token Efficiency", "Agent Engineering", "AI in Production" ] }, { "id": "https://blog.catalex.co/knowing-when-to-stop", "content_html": "Everyone who designs an agent loop spends their time on two steps. They tune the\ngenerate step, because that's where the intelligence appears to live. They\nharden the verify step, because that's where the discipline lives. Almost nobody\nbudgets serious design effort for the third step, which is the one that actually\ndetermines whether the loop is trustworthy: the stop.
\nHere is the uncomfortable mechanical fact underneath every loop you have ever\nbuilt. A turn that produces text with no tool call is the only place a run\ncan stop. That's it. That's the entire exit condition — the absence of a\nfunction call. And that single signal conflates three completely different\nsituations, which is why agents routinely \"finish\" tasks they never did.
\nThis post assumes you already have a loop — generate, verify, correct — and that\nit has burned you at least once. We're going one level down, into the exit.
\n\nWhen the model emits a turn with no tool call, exactly one of these is true:
\ndone — the agent delivered a real, complete answer.question — the agent is deliberately asking the user for something. An\nagent is always allowed to ask. A genuine ask is a first-class stopping\npoint, never a stall.stalled — the agent stopped with the task demonstrably unfinished\nwithout asking: it narrated an action it never took, returned partial data,\nor skipped a step.Read that third one again, because it's the whole post. Narrated an action it\nnever took. Every agent builder recognises this instantly. \"I've updated the\nconfiguration and re-run the checks.\" No it hasn't. There is no tool call in\nthat turn. The model produced a paragraph describing work in the past tense, the\nloop saw no function call, the loop exited, and the harness handed a fluent lie\nto whoever was waiting.
\nThe naive loop treats all three as one event, because at the transport layer\nthey are one event: a message with an empty tool-call list. Left alone,\ndone, question, and stalled all become \"the run ended successfully,\" and\nyour completion signal is decoupled from reality.
So we put a gate exactly there — at the no-tool-call boundary — and make it\nbranch.
\n\nThis pattern has a name in coding agents — a stop hook — and it generalises\ncleanly to any agent loop. The interesting part isn't the shape. It's the four\ndesign decisions inside it.
\nThe gate asks the LLM to classify the stop. There are deliberately no\nstring or phrase heuristics, because only the model that produced the turn can\nreason about whether it actually finished.
\nThis is the decision people push back on hardest, and it's the one we're most\nconfident about. The tempting move is obvious: regex the turn for \"I've\ncompleted\", \"Done!\", \"Let me know if\". Ship it in an afternoon, zero extra model\ncalls, deterministic.
\nIt's a trap, and the reason is structural. The sentence \"I've completed the\nmigration\" is exactly as likely to appear in a done turn as in a stalled\nturn. That's what makes stalls dangerous — a stalled agent is not confused or\nhedging. It's confident. It writes the completion sentence because it believes\nthe completion sentence. The surface text is identical in both cases; the only\ndifference is whether the preceding transcript contains the tool calls that\nwould make the sentence true. A regex cannot see that. It matches the claim, not\nthe evidence for the claim.
The thing that can see it is a model looking at the turn and the work that\nled to it. Judging \"did this actually happen\" is a reading-comprehension problem\nover a transcript, and the only tool we have that does reading comprehension is\nthe model. So we use it, and we accept the cost of an extra call rather than\npretend a pattern match is a judgement.
\nThe gate reads a bounded window — the last 8 messages — not the whole run. The\nclassification is about the stop, not the history. You're asking \"is this turn\nhonest given what just happened\", not \"summarise the last forty minutes.\"
\nThe gate runs only at the no-tool-call boundary. Turns that emit tool calls\nroute straight to the tool node and never reach it.
\nThat one sentence is what makes the design affordable. A long run might be forty\nturns, thirty-eight of which call tools. Those thirty-eight never touch the\ngate. The judging happens at candidate stopping points — roughly once per run,\na handful of times if the agent stalls and recovers.
\nGeneralise it: put your expensive check at the decision boundary, not in the\nhot loop. People reach for per-turn monitors and per-turn critics, discover\nthe overhead is untenable, and rip the whole idea out. Usually the check didn't\nneed to run per turn. It needed to run at the moment a decision becomes\nirreversible. Ending a run is irreversible — the answer goes to the user.\nCalling a tool isn't; another turn is coming that will re-examine everything.\nSpend your judgement where the door closes.
\nThis is the strongest idea we have on the subject.
\nWhen a run carries a plan, the plan — not the model and not the judge —\ndecides completion. Every step marked completed ⇒ stop. Any step unmarked ⇒\nthe harness rejects the stop and re-injects the specific next step, so the model\ndrives its own plan to completion and cannot hallucinate the whole task done\nwith work remaining.
\nSit with the inversion. The agent writes the plan. The plan then binds the\nagent. A model that decided at step two what needed to happen does not get to\ndecide at step five that actually it's finished — the artifact it produced when\nit had a clear view of the task now overrules the model that has drifted, gotten\ntired of the task, or convinced itself the remaining steps were optional.
\nThat's the trick, and it's more general than agents: externalise the\ncompletion criterion out of the model's judgement and into an artifact it can't\nretroactively edit. Completion stops being a vibe the model reports and starts\nbeing a property you can check. Six of seven boxes ticked is not a matter of\nopinion.
\nAnd note what the rejection does. It doesn't scold. It doesn't say \"you're not\ndone, try harder.\" It re-injects the specific next step, because a loop is only\nas good as its feedback string — and \"step 4: verify the new records resolve\" is\na task, while \"incomplete\" is a mood.
\nThere's a cap of 20 continuations. The anti-gaming rule matters more than the\nnumber:
\nEach stall consumes one unit, and a tool call in between does NOT refund it\n— so an agent that keeps making hollow tool calls but never completes (e.g. a\nfetch that always returns partial data) still terminates.
\nRefunding on tool calls is the obvious-looking design and it's wrong. If real\nwork resets the budget, then anything shaped like real work resets the budget,\nand an agent stuck in a genuinely unresolvable situation — a source that always\nreturns half the rows — will loop forever while looking productive the entire\ntime. Non-refundable units mean the budget measures stalls, not activity.
\nSame reasoning excludes bookkeeping tools. Updating the plan and marking a task\ncomplete don't count as real work. Otherwise an agent keeps its budget alive by\nticking its own boxes, which is a very sophisticated way of doing nothing.
\nOn the last unit the gate injects one \"wrap up with what you have\" nudge, so the\nrun ends with a best-effort answer rather than a dangling \"let me re-fetch.\" A\nbudget that expires silently mid-thought produces the worst possible artifact: a\ntruncated intention. Land the plane.
\nOne small thing at the router with a real lesson in it. The router checks the\nhard tool-budget stop first, before checking whether a user requested a stop,\nthen falls through to continuing.
\ndef route(state):
# This is checked FIRST — a runaway that also has a
# pending stop must still end.
if state.tool_calls >= TOOL_BUDGET:
return \"end\"
if state.user_stop_requested:
return \"end\"
return \"continue\"
Both branches end the run, so the order looks cosmetic. It isn't. The ordering\nencodes which authority wins when two conditions fire at once, and every one of\nthese little if chains is a policy document that nobody reviews as one. Write\nthem in the order you'd defend out loud: hard safety limits, then user intent,\nthen the default. In a router, precedence is policy — you are ranking your\nprinciples whether or not you meant to.
The honest tradeoff. The gate is a model call, which means it has a\nfalse-positive rate: sometimes it calls a real completion a stall and the agent\ndoes a redundant lap. We take that trade deliberately. A redundant lap costs\nlatency; a missed stall costs trust in every answer the system produces. When\nyour classifier is imperfect, choose the error you can survive.
Upstream of the exit, everything is optional in a way you can talk yourself out\nof. You can ship a loop with a weak verifier and tell yourself the model is\nusually right. You can skip the plan. The system will look like it works.
\nThe stop is where that stops being true, because the stop is the only step whose\noutput the user actually sees. A loop that can't tell \"I'm done\" from \"I gave\nup\" from \"I need you\" will confidently hand you all three as the same thing —\nand the one it hands you most often is the middle one wearing the first one's\nclothes.
\nGetting the stop right is what makes everything upstream of it worth building.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/knowing-when-to-stop", "title": "Knowing When to Stop: The Hardest Part of an Agent Loop", "summary": "An agent loop stops when a turn produces text with no tool call — and that one signal means three different things. How we classify the stop, why heuristics don't work, and how a plan can override the model's own sense of completion.", "date_modified": "2026-07-09T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "Loop Engineering", "Agent Harness", "Agent Engineering" ] }, { "id": "https://blog.catalex.co/budgets-that-stop-a-runaway-agent", "content_html": "Ask an engineer how to stop an agent that won't stop, and you'll get the same\nanswer every time: add a limit. It's the right instinct and the wrong plan.\nThere is no single limit that catches every shape of runaway, because runaways\ndon't have one shape — and each shape slips past a different guard for a\ndifferent structural reason.
\n\nThe failure mode people picture is a model producing one bad answer. That's not\nthe interesting failure. The interesting failure is a loop that keeps making\nplausible, individually-reasonable calls forever. Every call, read on its own,\nis a call you would have approved. Only the hundredth one tells you anything is\nwrong — and by then the run has been busy a long time, doing work that looked\nlike progress the whole way down.
\nThat we build the harness rather than treat it as glue is settled ground here.\nThis post is one level deeper: the specific mechanics of putting a floor under\nan autonomous loop, and why bounding one takes four independent guards instead\nof a number in a config file.
\nStart with the guard everyone reaches for first: refuse a call the agent has\nalready made. Deduplicate. If it asks the same question twice, it's stuck.
\nThat guard never fires.
\nA runaway per-record loop calls the same tool with different arguments every\ntime. Record 1, record 2, record 3. Message A, message B, message C. No call\never repeats itself, so a de-duplicator sees a stream of perfectly novel,\nperfectly legitimate requests and waves each one through. The loop isn't\nrepeating. It's enumerating — and enumeration is indistinguishable from\ndiligence, one call at a time.
\nEvery wall below exists because some loop shape is invisible to the others.\nThat's the design constraint — not defense in depth for its own sake, but four\nguards keying on four genuinely different things, because a loop that's obvious\nalong one axis is featureless along the rest.
\nThe first wall caps calls at 40 per tool name, and it ignores arguments\nentirely.
\nThat last part is the point, and it's a direct response to the shape above: the\nobserved failure was the same tool called hundreds of times with different\nids, so an identical-call guard would not catch it. Strip the arguments out of\nthe identity and the loop becomes trivially visible. Forty reads is forty reads\nwhether it read the same record forty times or forty different ones.
\nCrude on purpose. A tool name is the coarsest possible key, which is exactly why\nnothing about the arguments can hide from it.
\nWall 1 has an obvious hole: an agent that calls eight different tools thirty-nine\ntimes each never trips a per-name cap and has still done three hundred\noperations. So the second wall is tool-agnostic — a soft cap of 40 across all\nexternal tool operations. A mail read, a repository listing, and a knowledge\nsearch all count identically. There is no notion of a cheap call.
\nThe clever part isn't the cap. It's what happens past it.
\nPast the cap, a call identity is refused once. An identical repeat is\nallowed through.
\nSit with that for a second, because it's doing something subtle. A runaway\nper-record loop iterates different args each call, so no call ever repeats its\nidentity and the loop stays refused — forever, every call, one after another. A\ngenuinely-required call proceeds on its confirmed retry: the agent gets told no,\ndecides the call really is necessary, asks again the same way, and gets it.
\nA loop cannot confirm itself, because confirming requires repeating — and the\nloop never repeats. The exact property that makes a runaway invisible to a\nde-duplicator is the property that keeps it behind this wall — the behavior that\ndefeats one guard is what the other is built to catch. The agent isn't asked to\nprove intent. The mechanics of its own loop answer for it.
\nSend, post, deliver: these draw on a separate reserve of 5 operations,\nargument-independent, exempt from the data-gathering cap.
\nWithout it, the arithmetic goes badly. A run spends its entire budget reading and\nhas nothing left to deliver the result. All research, no answer — the worst place\nto stop, because all of the expensive work happened and none of it reached\nanyone. A budget that can strand its own output is a budget with a bug in it.
\nThere's an honest design note here too. The refuse-once-then-confirm cycle from\nWall 2 was too fragile for sends. It required the agent to repeat the exact\ncall — defeated the moment it reworded the message between attempts. And of\ncourse it rewords the message; that's a reasonable thing to do when you've been\ntold no.
\nA read is idempotent enough to retry; a send is not. Different semantics,\ndifferent budget. When a mechanism's assumptions don't hold for a class of tool,\nthe answer is a second mechanism, not a stretched first one.
\nThe last wall caps at 10 and keys on (file_ref, normalized_sql). A\nnormalizer strips string and number literals down to a skeleton, so WHERE ts='A' and WHERE ts='B' collapse to the same signature.
This catches the loop that both other guards structurally miss. A per-tool-name\ncount sees one tool used a reasonable number of times. An identical-call guard\nsees ten distinct queries. Neither is wrong. The loop is simply invisible along\nboth of their axes — and visible immediately along a third: same query shape,\ndifferent literal, over and over.
\nNormalization is what makes an enumerating loop's signature stop moving.
\n| Wall | Keys on | Catches | Can't see |
|---|---|---|---|
| Per-tool-name (40) | Tool name only | One tool hammered, any args | Many tools, each under cap |
| Aggregate (40, soft) | Call identity, all tools | Broad enumeration across tools | Nothing — but it only refuses, it can't end a run |
| Delivery reserve (5) | Send-class tools | Budget starvation before delivery | Data-gathering loops (by design) |
| Query loop (10) | (file_ref, normalized_sql) | Same query shape, rotating literals | Loops that vary query structure |
All four are soft. They refuse; they don't terminate. Above them sits a single\nhard aggregate cap on external tool operations — tunable per deployment, and\nsetting it to 0 disables it.
\nThe mechanism is worth showing because it's clean. At the cap, the harness\ninjects a terminal assistant message with no tool calls. The router reads\nthat marker and routes to __end__:
# At the hard cap: don't raise. Speak, then let the router do its job.
if hard_cap and external_ops >= hard_cap:
state.messages.append(
assistant_message(hard_stop_notice(external_ops, hard_cap))
)
return state # note: no tool calls attached
def route(state):
last = state.messages[-1]
if last.tool_calls:
return \"tools\"
return \"__end__\" # a terminal message is already the end shape
No exception. No special termination path. The run ends at the graph's normal\ncompletion shape — the same one a successful run uses — because \"an assistant\nmessage with nothing left to call\" already means finished. This mirrors a\nprecedent in the same router, where a synthesis-complete message ends the run\nidentically. The hard stop didn't need new machinery. It needed to say the thing\nthe router already understood.
\nAnd the terminal message is written for the user, not the log:
\nI stopped after {n} tool calls — this run hit its hard limit of {cap} tool
calls, which guards against a runaway loop. I've summarised what I gathered so
far; if the task genuinely needs more tool calls, break it into smaller runs
or raise the agent's tool-call budget.
None of the four walls work as exceptions. Refusals are returned as tool\nerrors the model reads and re-plans against.
\nThe governing rule: a refusal must never read like a success. Every refusal\nmessage states explicitly that the call did NOT execute.
\nThis sounds fussy and isn't. A guard that silently returns nothing teaches the\nmodel that the tool is broken — and a model that believes its tool is broken does\nexactly what you'd do: retry it, try a neighbouring tool, work around the outage.\nYou've converted a budget into a stimulus for more calls. A guard that explains\nteaches the model to change strategy, which is the only outcome you wanted. The\nrefusal is an input to the next planning step, so it has to be written like one.
\nA tool is metered if its canonical name isn't in the local-tools set. That's the\nwhole rule — external ops get counted uniformly, and nobody maintains a\nper-vendor list that goes stale the day someone adds an integration.
\nTwo details make it hold. A canonicalizer collapses the two surface spellings of\nthe same tool — the function-call form and the code-mode form — into one\nidentity, so an agent can't dodge a budget by switching call styles. And\nsynthetic messages the harness injects itself never count; the meter measures the\nagent, not the scaffolding around it.
\nOne budget is deliberately not a wall. The turn budget advances in steps of 50 —\n50, then 100, then 150 — and injects a warning the agent is free to ignore. It\ngates nothing.
\nThat's not a weaker version of a hard cap. It's a different tool. A soft signal\nshapes behavior; a hard wall guarantees termination. A warning lets a\nstill-reasoning agent wrap up on its own terms, which is a better ending than any\nforced stop — but it promises nothing, because an agent that ignores it just\nkeeps going. A hard cap promises termination and can only deliver it bluntly. You\nneed both, and you need them separate: a soft signal that gates is a hard wall\nwith worse ergonomics, and a hard wall that warns is a wall that doesn't work.
\nOnly the turn budget reaches the box today. Token and wall-clock budgets are\ncomputed but not enforced in-box — the outer bound on wall-clock is the sandbox's\nown lifetime. That's a real gap, and we'd rather write it down than let the\ndiagram imply coverage the code doesn't have.
\nThe four walls are the shape of the argument, though, and the shape generalizes.\nThe question is never \"what's the limit?\" It's \"what does this loop look like,\nand which key makes it stop moving?\" Answer that four times and you have\nsomething that holds.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/budgets-that-stop-a-runaway-agent", "title": "The Four Walls: Budgets That Stop a Runaway Agent", "summary": "A runaway agent loop calls the same tool with different arguments every time, so an identical-call guard never fires. Four independent budgets — plus one hard stop — and why each catches exactly what the others structurally cannot.", "date_modified": "2026-07-08T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "Agent Harness", "Agent Engineering", "AI in Production" ] }, { "id": "https://blog.catalex.co/writing-a-good-agent-skill", "content_html": "A skill is a self-contained Markdown directive with a little YAML frontmatter on\ntop, sitting in its own directory. That description makes it sound like\ndocumentation, which is exactly the trap. The interesting part is not what a\nskill contains. It's when the agent pays for it.
\nHere is the sentence we keep in our own skill registry, and it is the whole\ndesign: a skill is a directive the agent reads on demand via a read_skill\ntool, rather than something injected into every prompt — so the agent pays the\ntoken cost only when the skill is actually relevant to the turn.
Read that twice, because every rule below is a consequence of it. On demand is\nnot an implementation detail you can bolt on afterward. It is the property that\nmakes a skill a skill instead of a system prompt with extra steps. And it means\nthe best skill you can write is one that is absent from most of the agent's\nturns — present, correct, and completely unloaded.
\nStructurally there is almost nothing to it:
\n---
name: design-core
description: >-
Read BEFORE designing any visual deliverable — decks, documents, pages.
Carries the non-negotiable design bans and the layout method. Load this
first, then the medium-specific skill.
---
# Core design directive
## Absolute anti-slop bans
- No Times. No Arial.
- No pure black text on pure white.
- No default 1px grey borders.
...
Frontmatter, a body, a directory. The agent's harness knows the file exists and\nknows what the description says. It does not know a word of the body until it\nasks. That gap — the harness knowing the name while deferring the content —\nis the entire budget you have to work with, and four rules follow from spending\nit well.
The agent could read the file with cat. Same bytes into context, no tool\nrequired. Do not do this.
Loading through a dedicated tool keeps the raw directive text out of the\nuser-facing terminal stream. It rides to the model as a tool result; the UI\nshows a compact card. If the agent cats the file instead, the user watches a\nwall of imperatives about font choices scroll through their chat — a\nconversation they were never party to, rendered as if it were output.
The tokens are identical. The product is worse. That asymmetry is worth\ninternalizing, because it's the first place skill design stops being a context\nproblem and starts being an interface problem. Your directive is addressed to\nthe model. Anything that leaks it to the human is a bug, even when it's free.
\nIf bodies load on demand, something has to decide when. That something is the\ndescription field — and it works because the tool that loads skills enumerates\nthe valid skills in its own description. The model sees the full menu of names\nand triggers without loading a single body.
Names in the prompt. Bodies on demand.
\nWe apply the same principle one layer down, to tool schemas: every tool name is\nin the system prompt, but the parameter schemas are fetched only when needed. The\nguidance we give the fetching tool says it plainly — your tools are ALL listed\nby exact name in the system prompt; this is NOT for discovery, it fetches the\nparam schema you don't yet have. Discovery is free. Detail is not. Skills are\nthat idea applied to prose instead of JSON.
\nWhich reframes what your description is for. It is not a summary of the skill.\nNobody is browsing. It is a trigger — the condition under which a model that\nhas never read your body should decide it needs to. Write it as an instruction\nabout when, not a précis of what:
The second one routes. The first one describes.
\nSelection should be boring. Ours is prompt-driven routing through a\nhard-coded table — deliberately not embedding search, not auto-selection. A\npreamble names the deliverable type and the skill to load. Clever selection\nfails in ways you cannot reproduce; a table fails in ways you can read.
The instinct when writing a skill is to make it complete. Resist it. Our six\nshipped skills are deliberately layered: one medium-agnostic core design\ndirective is read before designing anything, and the medium-specific skills\nbuild on top of it with the format rules for exactly one output type. The agent\nreads the core, then the layer.
\n| Skill | Role | Size |
|---|---|---|
| Core design directive | Read before designing anything | ~7.5 KB |
| Presentation | Slide-specific rules on top of the core | ~8 KB |
| Self-contained HTML | Single-file page rules | ~5.6 KB |
| Document (PDF) | Print-oriented format rules | ~3.2 KB |
| Document (DOCX) | Editable-document format rules | ~2.7 KB |
| Spreadsheet (XLSX) | Tabular output rules | ~2.5 KB |
The sizes are the point. Not one of them is a manual. The core carries what is\ntrue of every visual deliverable; each layer carries only what is true of its\nmedium and nothing that the core already said. A spreadsheet task loads roughly\n10 KB total and never sees a word about slide transitions.
\nThe failure this prevents is subtle. A single lumped skill isn't just bigger — it\nforces every task to carry every medium, which means the moment it starts\ncrowding the context window, someone \"fixes\" it by making it vaguer. Layering\nlets you stay specific, because specificity is cheap when it's scoped.
\nSpecificity is also what makes a skill bite. Our core design directive opens with\n\"Absolute anti-slop bans\": no Times, no Arial, no pure black on pure white, no\ndefault 1px grey borders. Concrete prohibitions beat vague encouragement, every\ntime. \"Use tasteful typography\" is a sentence a model can agree with while\nproducing Arial. Skills carry a persona and hard bans because that's what\nsurvives the trip through a generative process. And when prose is the wrong\ninstrument, ship an executable script alongside the markdown — a skill directory\nis not obligated to be only words.
\nOur presentation skill needs a 54 KB theme catalog. It does not contain a 54 KB\ntheme catalog.
\nThe loader appends the catalog's resolved absolute path to the skill text, so the\nagent can json.load it inside run_python without the prompt ever carrying the\ndata — and the skill explicitly instructs the agent not to dump the thing into\nthe chat. A 54 KB asset costs a path.
Generalize this. Any time your skill wants to contain a lookup table, a\nschema dump, a catalog, or a corpus of examples, it wants a path instead. The\nagent has a filesystem and an interpreter. Prose is for judgment; data is for\ndisk. A skill that inlines its data spends the whole context window on a lookup\nthat a single line of Python does better.
\nThe skills we ship inside the product run 2.5–8 KB. The skills our own engineers\nwrite for their internal dev tooling run 25–85 times larger. One internal QA\nskill is 215 KB — far, far past any progressive-disclosure guidance we would give\nanyone, including ourselves.
\nThe detail that makes it instructive rather than embarrassing: that skill has\nan envs/ subdirectory. The author knew the pattern. They applied it to the\nperipheral material and not to the main body.
That's the failure mode, and it's always the same one. Nobody sits down to write\na 215 KB skill. They write a good 8 KB one, then answer a question in it, then\nanother, then paste in the thing that keeps coming up — and each addition is\nlocally correct. The skill grows into a manual because nobody pays the cost of\nloading it at review time. The author reads the diff. The agent reads the\nwhole file, on every task that touches it, forever. The discipline is easy to\nstate and genuinely hard to keep, and pretending otherwise would make the rest of\nthis post less useful.
\nThe pattern that actually holds the line is two-tier. One of our better internal\nskills opens by declaring what it is: it orients and routes, and the deep,\nauthoritative reference is a longer document — the single source of truth. Read\nit before non-trivial work; don't duplicate it here. Eight kilobytes of skill\npointing at twenty-nine kilobytes of reference.
\nSkills should route to depth, not contain it. The skill's job is to know that the\nreference exists, say when it matters, and get out of the way. The reference's\njob is to be right. Conflating those two jobs is how you get 215 KB.
\nRead your skill and ask when it should not load. If you can't answer crisply,\nyour description isn't a trigger and your routing will be guesswork. Then check\nthat nothing in the body would be equally at home in the core, in a sibling\nlayer, or in a file on disk — those are the three things a bloated skill is made\nof.
\nThen apply the test that subsumes all of it:
\nIf your skill would be equally useful pasted into every system prompt, it\nisn't a skill. It's a system prompt you haven't admitted to.
\nThat's not a rhetorical flourish; it's a diagnostic with a fix on either side. If\nthe content genuinely belongs in every turn, move it to the system prompt and\nstop paying the indirection. If it doesn't — if it's for decks, or for one\nlanguage, or for the third Tuesday of the release cycle — then earn the deferral.\nSharpen the trigger, split the layer, push the data to a path.
\nA skill earns its keep by being absent most of the time. Write it so the agent is\nglad to see it, and doesn't have to.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/writing-a-good-agent-skill", "title": "Writing a Skill an Agent Will Actually Read", "summary": "A skill is a directive an agent loads on demand rather than something injected into every prompt. That one design choice dictates how you write it — the four rules that follow, and the discipline that is easy to state and hard to keep.", "date_modified": "2026-07-07T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "Agent Skills", "Agent Engineering", "Agent Harness" ] }, { "id": "https://blog.catalex.co/scheduling-agents-at-scale", "content_html": "\"Run this agent every morning at 9\" sounds like a one-line feature. There's a\ncron expression, there's a job table, there's a thing that reads the table.\nShip it Friday.
\nCron is the easy 5%. The other 95% is not firing twice, not losing a fire, not\nholding a database transaction open across a task that runs for minutes, and\ndeciding whose 9am you actually mean. A cron expression is a trigger. A\nscheduler is everything you build around it.
\n\nThis is a tour of that everything — the design of the scheduling layer under\nCatalEx, and the reasoning behind the parts that look strange until you've been\nbitten.
\nA managed cron trigger fires a tick endpoint every minute. The jobs service\nscans the database for anything due, and dispatches each due job through a\nmanaged task queue. The queue calls back into the owning service's handler,\nwhich does the actual work.
\n\nFour moving parts, three network hops, and every one of them is a place where\nthe same job can fire twice or not at all. Everything below is about closing\nthose gaps.
\nThe cron trigger is configured with zero retries. Deliberately. The comment\nin the code is the whole argument:
\n\n\nThe next tick will re-evaluate any jobs that were due; re-trying a failed\ntick risks double-dispatch against the same window.
\n
This inverts the instinct you've built everywhere else in your system. Down in\nthe leaves, retry is free and correct. At the root of a scheduling stack,\nretry is a duplicate-fire generator. The tick is not a request that must\nsucceed; it's a sample of a clock that will be sampled again in sixty seconds.\nA missed tick costs you a job running one minute late. A retried tick that\npartially succeeded costs you a job that ran twice — and \"ran twice\" for an\nagent means two emails sent, two tickets filed, two rows written into someone's\nCRM.
\nFail-open on the way down. Fail-closed on the way up.
\nZero retries on the tick is a policy, not a mechanism. Below it sit four\nmechanisms, each one assuming the layer above it failed.
\n| # | Mechanism | What it stops |
|---|---|---|
| 0 | Zero retries on the tick | Two ticks evaluating the same due window |
| 1 | FOR UPDATE SKIP LOCKED on the due-job scan | Two dispatcher instances claiming the same row |
| 2 | next_run advanced in the same committed transaction as the execution row | The job still looking due after it's been claimed |
| 3 | Task-name dedup — the enqueue passes execution_id as the queue's task name | Queue redelivery inside the dedup window |
| 4 | Idempotency claim at the handler — insert ON CONFLICT DO NOTHING on execution_id | Everything the first four missed |
Layer 1 is why you can run more than one dispatcher. SELECT … FOR UPDATE SKIP LOCKED on the due-job scan means concurrent dispatcher instances never\ndouble-dispatch the same job — the second instance doesn't block on the locked\nrow, it walks past it and takes the next one. Without SKIP LOCKED you get\neither a convoy (everyone blocks behind row one) or a single-instance\nscheduler, which is a single point of failure with extra steps.
Layer 2 is the one people skip. next_run is advanced in the same\ncommitted transaction that writes the execution row. The moment that commit\nlands, the job is no longer due — before the dispatch has even been\nattempted. The scan can't pick it up again on the next tick, because as far as\nthe query is concerned, the window has already passed.
Layer 3 costs one argument. Every enqueue passes the execution id as the\nqueue's task name. A queue that redelivers the same task name inside its dedup\nwindow drops it. This is free protection you get by naming a thing carefully.
\nLayer 4 is the backstop, and its reasoning is the elegant bit. The handler,\nbefore doing any work at all, inserts a claim row keyed on execution_id with\nON CONFLICT DO NOTHING. If the insert claimed nothing, someone else already\nhas this execution, and the handler returns {\"skipped\": \"duplicate execution\"} and goes home.
Why this is safe. A legitimate retry mints a NEW execution_id, so the SAME\nexecution_id arriving twice is always a duplicate delivery, never a lost retry.
The usual objection to handler-side idempotency is that you'll swallow a real\nretry. You won't — but only because retries are minted upstream as new\nexecutions, so identity and attempt are never conflated. Idempotency keys work\nwhen the key means this delivery, not this task. Get that distinction wrong\nand your dedup layer starts eating work.
\nThese are design rules, not scars. Each one exists because the alternative has\na specific, predictable failure mode.
\nNever hold a transaction across the dispatch. A downstream agent run takes\nminutes. Wrap the dispatch in the transaction that read the due jobs and you're\nholding an idle-in-transaction connection for the duration — long enough to\ntrip the database's idle-in-transaction timeout, at which point the connection\ndies mid-update and you get to find out what your code does about that. Hence\nthe split: read due jobs, commit, then dispatch.
\nCommit before you dispatch — in that order. Dispatch first and the callback\nopens a fresh transaction that cannot see your uncommitted execution row. The\nhandler looks up an execution that doesn't exist, and here's the vicious part:\nit still returns 200. The queue sees success, never retries, and the work\nvanishes with no error anywhere. Silent loss is worse than loud failure, and\nthe ordering is the only thing between you and it.
\nReport failures in-band. A malformed job returns HTTP 200 with\nstatus=\"failed\" in the body. It's tempting to return a 5xx — it feels honest.\nBut the tick's transport layer would read that as a transport error and apply\ntransport retry semantics to a permanently broken job, re-dispatching a payload\nthat will never parse. Status belongs in the body so the tick can apply its own\nretry policy. HTTP status codes describe the conversation, not the work.
Now the part that looks like a bug and isn't.
\nCron parsing is a standard 5-field expression, always UTC. There is no\ntimezone column. Not \"we haven't added one yet\" — the API contract explicitly\nforbids timezone-bearing fields on the wire. The database stores a UTC cron and\nnothing else.
\nAll local↔UTC math lives in the browser, because the browser is the only place\nthat knows the user's local zone. The server never guesses, never infers from a\nheader, never stores a stale zone that was true when the user signed up in a\ncity they've since left. There's one authority for \"your 9am\" and it's the\nmachine sitting in front of the user.
\nThere's no interval primitive either. \"Every N hours\" compiles in the browser\ninto ${minute} */${n} * * *, with the divisors restricted to\n[1,2,3,4,6,8,12] — the clean divisors of 24. That restriction is doing real\nwork: */5 hours doesn't tile 24 evenly, so it silently skips at the midnight\nwraparound. You'd get 0,5,10,15,20 and then a four-hour gap, forever, and\nnobody would notice for weeks. Restricting the input is cheaper than explaining\nthe arithmetic.
Alongside the cron sits a display descriptor — frequency, time_of_day,\ndays_of_week, interval_hours — stored purely so the UI can round-trip\nlosslessly. Rendering a UTC cron through a timezone-naive formatter prints UTC\nnumbers styled as wall-clock time, which is how you end up telling a user their\njob runs at 2pm when it runs at 9am.
\nWeekly schedules roll each weekday independently through UTC, then dedupe. A\nlocal Monday 11pm legitimately lands on Tuesday UTC while the rest of the week\ndoesn't move. The naive version — shift the time, then shift every day by +1 —\nis wrong for exactly the days that cross midnight, which is the only case where\nit matters.
\nThe honest tradeoff: because only a UTC cron is stored, schedules do not\nfollow DST. A 9am local schedule becomes 8am or 10am local when the offset\nchanges. Nothing recomputes crons on DST boundaries. This is a real limitation\nand we'd rather say it plainly than let you discover it in March. The\nalternative isn't free: following DST means storing the user's zone, which means\nrecomputing every affected cron whenever a government changes its rules, and\nowning the ambiguity of schedules that land in the hour that doesn't exist or\nthe hour that happens twice. UTC-only buys a schema with one interpretation.\nThat's the trade, made with open eyes.
\nThe remaining details are the ones that decide whether the thing survives a\nyear.
\nExecution timeout is a per-job timeout_minutes, constrained at the database\nbetween 1 and 15. A stale sweep marks anything past its timeout as failed and\nretries it — which is the only place a retry is minted, and it mints a new\nexecution id, which is what makes layer 4 sound.
Primary keys are UUIDv7 — time-ordered, so inserts land at the end of the index\ninstead of scattering across it. Hot-path indexes are partial: the index on\nnext_run carries WHERE status = 'active', because the dispatcher never asks\nabout inactive jobs and there's no reason to make it page past them.
A duplicate-schedule guard allows one schedule per agent per cron expression,\nenforced at the application layer, returning 409. Distinct agents may share a\ncron — the constraint is against a user double-clicking their way into two\nidentical jobs, not against 9am being popular.
\nOrphan protection runs both ways. Deleting an agent purges its schedules; and if\na run finds its agent already gone, the handler purges the schedules itself.\nOtherwise an orphaned schedule fires every tick forever, doing nothing, and\nnobody notices because nothing is failing.
\nAnd a run that suspends — waiting on a human approval — ends without a terminal\nframe. The execution still counts as completed, and records a status\nbreadcrumb instead. The dispatch did its job; a retry would double-run the\nagent. The scheduler's contract is \"the work was handed off correctly,\" not\n\"the work finished.\"
\nThat's the shape of it. One cron expression at the top, and underneath it four\nlayers of dedup, three rules about where a transaction may not go, a timezone\nmodel with a stated limitation, and a fistful of indexes. The cron was the easy\npart. It's always the easy part.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/scheduling-agents-at-scale", "title": "Cron Is Not a Scheduler: Running Agents on Time at Scale", "summary": "A cron expression is a trigger. A scheduler is everything you build around it — not firing twice, not losing a fire, keeping transactions off the dispatch path, and deciding whose 9am you mean.", "date_modified": "2026-07-06T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "Agent Scheduling", "AI in Production", "Agent Engineering" ] }, { "id": "https://blog.catalex.co/what-agents-can-do", "content_html": "\"What can your agent do?\" is two questions wearing one coat. The first is about\nthe action space: once the agent is awake and thinking, what can it reach,\ncall, read, and change? The second is about the trigger space: what makes it\nwake up in the first place? Teams conflate them constantly, and the conflation\nis expensive, because the two want opposite answers.
\n\nWe build for a deliberate asymmetry. The action space is as wide as we can\nresponsibly make it — a real Python kernel, a shell, roughly a hundred curated\nthird-party integrations, and permission to write loops instead of taking turns.\nThe trigger space is three entries long. Not three for now, pending a roadmap.\nThree because every entry point into an autonomous system is a surface someone\nhas to reason about, and we would rather have three surfaces we understand\ncompletely than thirty we understand approximately.
\nHere is the entire trigger space, as a StrEnum backed by an eight-character\ndatabase column:
class TriggerType(StrEnum):
CHAT = \"chat\"
RUN = \"run\"
SCHEDULE = \"schedule\"
That column width is not an accident of history. Eight characters is a wall.\nAdding a fourth trigger means a migration, which means a conversation, which is\nexactly the friction we want on this particular axis. Cheap to add is the wrong\nproperty for an entry point into a system that takes real actions.
\n| Trigger | Surface | What it means |
|---|---|---|
chat | studio_chat | Interactive conversation with the agent |
run | studio_run | A person clicks \"run now\" |
schedule | scheduled_job | Cron fires |
That's the list. There is no webhook trigger. No email-arrival trigger. No\n\"someone posted in a channel\" trigger. We know those are the first three\nfeatures on most people's wish list, and we know exactly what the absence looks\nlike from outside: a gap.
\nWe'd argue it's a position.
\nThe argument against webhooks isn't that they're hard. They're easy — that's the\nproblem. Every webhook you accept is an entry point where the outside world\ndecides when your agent runs, how often, and with what payload. Each one needs\nits own authentication story, its own replay semantics, its own idempotency\nreasoning, its own answer to \"what happens when this fires four hundred times in\na minute because an upstream system had a bad afternoon.\" That reasoning doesn't\namortize across endpoints. It's per-endpoint, forever.
\nA schedule is a trigger you can reason about. It fires when you said it\nwould. Its concurrency is bounded by its own period. Its payload is whatever the\nagent goes and fetches. And critically — you can poll from a schedule. Nearly\nevery \"I need a webhook\" requirement is really \"I need to know about this\nwithin N minutes,\" and a schedule answers that with a mechanism whose failure\nmodes fit in your head. You trade a little latency for an entry point that\ncannot be triggered by anyone outside your system. That trade is available to\nyou at any time, and it's usually the right one.
\nThe discipline shows up elsewhere too. There is an internal event job type in\nthe platform, and it would have been the natural place to sneak a fourth\ntrigger in. It doesn't start agent runs. It handles post-run work —\nconsolidating memory, scoring KPIs. Work that happens after an execution, on\nthe same rails, never as a way in. The trigger set stayed three because we kept\nit three when it was inconvenient.
Here's the part that makes the narrow set work rather than merely sound\nprincipled.
\n\nA scheduled run is not a special execution mode. It is not batch mode. It is not\na lighter-weight cousin of the interactive path with a few features stubbed out.\nIt goes down the identical orchestration path as a user-initiated run —\ndifferent surface, different trigger_type, same everything else. The intent is\nwritten into the module itself: keep the scheduled run \"on the identical\norchestration path as a user-initiated run.\"
You can see the discipline hold under pressure at the one place you'd expect it\nto bend. When someone manually triggers a schedule — the \"run this now, don't\nwait for 6am\" button — the obvious implementation is a shortcut: skip the\nschedule machinery, fire a direct run. We don't. It creates an execution with\ntrigger_source=\"manual\" and dispatches immediately, leaving the recurring\nschedule intact, and it reaches the agent through the same schedule path. The\nmanual trigger is a nudge to an existing mechanism, not a bypass around it.
The alternative is where divergence bugs live. A separate batch mode starts\nas ninety percent shared code. Then the scheduled path gets a fix the\ninteractive one doesn't. Then someone adds a guard to the interactive path\nbecause that's where the user complaints come from. Six months later you have\ntwo systems with one name, and the failure you're debugging only reproduces at\n4am on Tuesdays. One path with many entry points means a fix lands once, and\nevery trigger inherits it. It also means the trigger set can stay small without\ncosting capability — because narrowing the ways in doesn't narrow what happens\nonce you're in.
\nEverything above is about restraint. None of it applies to the action space,\nwhere the answer is: as much as we can hand it.
\nThe agent has a real computer. Inside the box, a set of tools run locally —\nrun_python, bash, text_editor, query_file, read_file, update_plan,\nload_tools, get_tool_details, fetch_full_result, query_resource,\nuser_input, task_complete, save_memory. These aren't metered as external\noperations, because they aren't external. They're the agent using its own\nmachine.
Above that sit first-party servers: web search, the Brain knowledge and search\nlayer, a platform \"system\" server, and a tool catalog the agent queries to find\nout what else exists. And beyond those, roughly a hundred curated third-party\nintegrations reached through our integration layer — CRM and sales,\ncommunication, finance, documents and storage, calendar and meetings. Wide by\ncategory, curated rather than infinite, but wide.
\nThe default mental model of an agent is a turn-taker. Think, call one tool, read\nthe result, think again. That model is a ceiling, and most of it is\nself-imposed.
\nThe preamble teaches a three-tier ladder instead:
\n1. Answer directly — no tool needed.
2. Call a tool directly — for ONE discrete action.
Don't wrap a one-shot call in code.
3. Write and run code — for orchestration: chaining calls,
looping, fetching in bulk.
The rung that matters is the third, and the guidance that makes it click is\nblunt: \"Calling the same tool across a list is a LOOP — write it once in\ncode.\" An agent asked to touch forty records shouldn't take forty turns. It\nshould write the loop. The Python kernel persists across cells, so state carries\nbetween steps — fetch in one, transform in the next, act in a third, without\nrebuilding the world each time.
\nTier two matters as much, in the other direction. The failure mode of a capable\nagent is wrapping a single API call in eleven lines of Python because code feels\nmore thorough. \"Don't wrap a one-shot call in code\" is the counterweight. The\nladder is a ladder — you're supposed to stop climbing when you've reached the\nrung that fits.
\nAlso worth knowing: large tool results never enter the conversation. The\nagent gets a short preview plus a file reference it queries with SQL. So \"what\nit can do\" includes working over data far larger than a context window — the\nwindow holds the pointer, not the payload.
Widen what an agent can do once it's running. Narrow the number of ways it can\nstart running. Those pull in opposite directions and they should — they're\nprotecting different things.
\nThe action space is bounded by the agent's judgment inside a run you initiated,\nobserved under permissions you set. Getting it wrong costs you one bad run. The\ntrigger space is bounded by nothing except the entry points you built, and every\none you add is a door someone else's system can knock on at a time you didn't\nchoose. Getting that wrong costs you the property that made the whole thing\ngovernable.
\nSo: three triggers, one path, a real computer, and permission to write a loop.\nIf you want an agent to react to something we don't trigger on, poll for it from\na schedule. It's less elegant. It's a mechanism you can hold in your head, and\nafter enough production systems, that stops feeling like a consolation prize.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/what-agents-can-do", "title": "What an Agent Can Actually Do (And When It Does It)", "summary": "An agent's action space and its trigger space are different questions. We made the action space wide — a real computer, ~100 integrations, code-mode orchestration — and the trigger space exactly three entries wide. Here's why that asymmetry is the right one.", "date_modified": "2026-07-05T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "Agent Engineering", "Integrations", "AI in Production" ] }, { "id": "https://blog.catalex.co/creating-an-agent-in-catalex", "content_html": "From the outside, creating an agent looks like filling in a form. You describe\nwhat you want, you look at a preview, you click a button. Thirty seconds, maybe\nless. The interface is doing everything it can to feel like nothing much is\nhappening.
\n\nThat impression is the product working correctly and it is also completely\nwrong. The interesting engineering in an agent-creation path isn't the form —\nit's the handful of seconds after you confirm, where the system decides whether\nyou're getting a working agent or a plausible-looking broken one. Those seconds\nare made of validation, versioning, and a few unglamorous atomicity decisions.
\nThis is a walk through ours, in the order the system experiences it, and an\nargument about what a well-built creation path owes you.
\nThere are three shipped ways to get an agent in CatalEx. You can talk one into\nexistence through a chat wizard — the \"from scratch\" path. You can pick from a\ntemplate catalog of 159 templates across 13 categories, weighted toward the work\npeople actually automate: 41 specialized, 30 marketing, 29 engineering, then a\nlong tail through design, sales, testing, support, finance, and product. Or you\ncan do nothing at all: every account is provisioned an agent at signup.
\nThe three doors converge on the same write path. That's the first design\ndecision worth naming — the wizard is not a privileged client. It ends up\ncalling the same create contract as everything else, which means the guarantees\nbelow are guarantees for all three, not features of one nice UI.
\nThe \"from scratch\" flow is a three-phase state machine:
\ntype Phase = 'chatting' | 'synthesizing' | 'reviewing';
// User turns only. One assistant turn can span several streamed
// messages, so counting messages would let a single exchange
// masquerade as a conversation.
const MIN_TURNS_FOR_CREATE = 3;
In phase one you're talking to a purpose-built system agent whose only job is to\ninterview you. The \"Create Agent\" button doesn't exist until you've taken at\nleast three turns. Not three messages — three user turns, because a single\nassistant reply can arrive as several streamed messages and counting those would\nlet one exchange impersonate a conversation.
\nThe constant is a product opinion wearing a type annotation. You cannot\none-shot an agent definition here, by construction. The reason is not\npaternalism; it's that a one-line description underdetermines an agent. \"Write\nmy release notes\" doesn't say which repository, what tone, who reads them, what\nhappens when a release has no user-facing changes. A model handed that will\ninvent answers, and the invented answers are exactly the ones you'll be\ndebugging in three weeks. Forcing a conversation front-loads the questions to\nthe moment when answering them is cheap.
\nThe interviewer runs under its own budgets: 15 turns, 50,000 tokens, 120 seconds\nof wall clock. An agent whose job is to elicit requirements should not be able to\nelicit them forever. Budgets on a helper agent aren't a safety measure so much as\na design constraint — they force the interview to converge.
\nHit the button and you're in phase two. The synthesizer streams over SSE, and a\nlive checklist tracks it through plan_snapshot frames. The seeded plan is\nexactly five steps: understand request, discover tools, draft instructions, draft\nKPIs, validate definition. You watch them tick.
The second step is more interesting than it sounds. The synthesizer discovers\nwhich tools are available by reading a seeded file on disk — once — rather than\ncalling a discovery tool. Its prompt says so plainly: \"There is no tool_catalog\ntool to call.\"
Put the menu on disk instead of spending a tool call on it. A tool call for a\ncatalog is a round trip, a chunk of context, and a step in the loop, all to\nretrieve something that was static the whole time. If the model needs a fixed\nreference to do its job, hand it the reference. Save the tool calls for the\nthings that actually require going somewhere.
\nFrom there the synthesizer writes into a fixed skeleton — About, Role, Core\nWorkflows, Communication Style, Constraints, Decision-Making, Output Format —\ntargeting 400–800 words across five to eight sections. A skeleton is not a\ncreative restriction. It's what makes the output diffable, reviewable, and\ncomparable across 159 templates and every agent a user writes by hand.
\nPhase three is an editable preview. Nothing has been written yet. You can rewrite\nany of it before it becomes real.
\nHere's the write path.
\n\nFirst, every requested tool key is validated against the live catalog. Not a\ncached list, not the list the synthesizer thought it had. The reason is in the\ncode comment: this guards against model hallucination, because a synthesizer that\nemits a plausible-but-wrong tool key \"would silently produce an agent with 0 tools\nat runtime\". Invalid keys come back as a 400 with the offending keys named.
\nThat sentence is the best single lesson in the whole path, so let it land. A\nhallucinated tool key is not a loud failure. It doesn't throw. It produces an\nagent that looks completely fine in the preview, saves without complaint, appears\nin your list with a green dot, and then does nothing useful when you run it —\nbecause the tool it was told to use resolves to nothing. The failure surfaces\ndays later as \"the agent seems dumb.\"
\nThe general form: anywhere a model's output becomes a key, an ID, or a\nreference into another system, the failure mode is silence. Validate at write\ntime, against the live source of truth, and fail loudly with the specific bad\nvalue. The alternative isn't an error — it's a mystery.
Second, the create contract requires an explicit quota field. There's no\ndefault. A caller cannot create an agent without stating the cap, because — from\nthe code — the service \"never creates without an explicit cap so the invariant\n'every create is cap-checked' holds at the contract layer.\"
\nEncode invariants in the contract, not in a caller's good intentions. A default\nwould work fine right up until the day someone adds a fourth creation path, and\nthen the invariant is a comment in a file nobody opened.
\nThe agent table holds what you'd expect — name, instructions (a DB CHECK caps\nthem at 16,384 characters), description, tags, status (CHECK IN 'active',\n'paused', 'draft'), KPIs as JSONB, the memory and evolution flags, run counters,\ncurrent_version, template_id. The absences are more informative.
There is no model column. Model choice is a service-level concern, not a\nper-agent field. Bake a model into the agent row and every agent becomes a small\nmonument to whatever was best the week it was created, and upgrading the fleet\nbecomes a migration.
\nThere is no schedule column. Schedules live in a separate service, keyed by\nagent id. An agent is a definition of behavior; when it runs is somebody else's\nquestion. Fusing them into one row is how you end up with a scheduler you can't\nchange without touching the agent model.
\nIf the row is inserted, two more things must happen: build version 1 — a snapshot\nof instructions, KPIs, memories, and tools — and pin it. If either step fails, the\nagent row is rolled back and the API returns 503 with a Retry-After. You never\nget a half-created agent, and you never get an agent at version zero whose first\nedit has nothing to diff against.
The 503 is deliberate. Failing the whole create and telling you to try again is\nstrictly better than handing back a live-looking agent with no version history,\nbecause the second one is a debugging problem you inherit silently.
\nAnd now the honest seam. Schedules and tool bindings attached at create time\nare not covered by that guarantee. They're best-effort — attempted, and on\nfailure logged and swallowed. Which means a binding failure can produce a live\nagent that's missing a tool. We know. The v1 pin is load-bearing for correctness,\nso it's atomic; the bindings are recoverable by re-attaching, so they're not. It's\na real seam, chosen rather than overlooked, and worth stating plainly because\nevery atomicity boundary is a decision about which failures you'd rather have.
\nThe agent you get at signup is named Lex. Lex is not a special row. It's an\nordinary agent provisioned from the research-assistant template, subject to the\nsame validation, the same v1 pin, the same everything.
The code never identifies it by name. It matches on template provenance, and the\ncomment explains why: \"What makes it 'built-in' is provenance, so that's what we\nmatch on — never the display name, which the user is free to change.\" When a\nmigration needed to rename the default agent, it only touched rows still carrying\nthe stock name. If you'd renamed yours, it was left alone.
\nMatch on the immutable fact, not the mutable label. Display names are user\ndata. Any logic keyed on one is a bug waiting for the first person who\npersonalizes something.
\nThe provisioning itself is best-effort by design. Every failure path returns 200\nwith provisioned: false and a reason. Signup does not fail because agent\ncreation did — the account is the thing that must exist; the starter agent is a\nconvenience, and convenience never gets to block the front door.
Validate the model's output against the live world before you persist it. Make\nthe invariant a required field, not a default. Draw the atomicity boundary\nconsciously and say where it is. Keep the mutable labels out of your logic.
\nNone of this shows up in the form. That's the point — the whole job is to make\nsure that the thing you clicked \"create\" on is the thing you actually got.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/creating-an-agent-in-catalex", "title": "Creating an Agent: What Happens After You Hit Save", "summary": "Creating an agent looks like filling in a form. The real engineering is in the seconds after you confirm — tool validation against a live catalog, an atomic version-1 pin, and the contract decisions that stop you shipping a broken agent.", "date_modified": "2026-07-04T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "Agent Engineering", "AI in Production" ] }, { "id": "https://blog.catalex.co/vibe-coding-to-spec-driven-to-loop-engineering", "content_html": "Every team that builds seriously with AI walks the same road, usually without\nnoticing. It starts with a chat window and a good feeling, and — if the work\nsurvives contact with production — it ends somewhere far more disciplined. This\nis a field note on that road: the three stages we see teams pass through, what\neach one is genuinely good at, and the specific failure that forces the jump to\nthe next.
\n\nWe are writing it down because the stages are easy to confuse. \"Vibe coding,\"\n\"spec-driven development,\" and \"loop engineering\" get used loosely, as vibes\nthemselves. They are not interchangeable moods. They are three different answers\nto one question — who is responsible for correctness, and how is it checked? —\nand the answer changes as the stakes rise.
\n| Stage | The unit of work | Correctness comes from | Breaks when |
|---|---|---|---|
| Vibe coding | A prompt | The human eyeballing the output | You can't hold the whole system in your head |
| Spec-driven development | A specification | The spec + acceptance tests | The spec is right but the run is flaky |
| Loop engineering | A loop | The system verifying and correcting itself | — (this is where production lives) |
Each row is a reaction to the row above it. Read top to bottom, it is a story\nabout moving responsibility for correctness out of the human's head and into\nan artifact, then into a system.
\nVibe coding is building by feel. You describe what you want in natural language,\nthe model produces something, you look at it, and you react. If it looks right,\nyou keep it. If it looks wrong, you nudge. The loop is fast, tactile, and — this\nis the important part — the only verifier is you, looking at the screen.
\nIt is genuinely great. For a prototype, a throwaway script, a UI you can see\nwith your own eyes, or the first exploratory hour of any project, nothing beats\nit. The feedback is immediate and the cost of being wrong is a re-roll.
\nHere is the shape of it:
\nYou: \"Build a page that lists our open roles from the careers API.\"
Model: <200 lines of code>
You: *scrolls, runs it* \"Close. Make the cards clickable and sort by team.\"
Model: <revised code>
You: *looks* \"Ship it.\"
The reason it works is also the reason it doesn't scale: verification is\nimplicit and human. You are the test suite. That is fine when \"correct\" means\n\"looks right in the browser\" and the whole thing fits on one screen. It falls\napart the moment correctness depends on something you can't see at a glance — a\nconcurrency edge case, a compliance rule, a contract with another service, a\nbehavior that only shows up on the 10,000th request.
\nThe tell that you've outgrown vibe coding: you find yourself re-checking the\nsame things by hand every time you make a change, because you no longer trust\nthat a small edit didn't break something out of view. That manual re-checking\nis a test suite — you just haven't written it down yet.
Spec-driven development moves the source of truth out of the conversation and\ninto a document. Instead of steering the model turn by turn, you write down what\n\"correct\" means — the behavior, the constraints, the acceptance criteria — and\nthe model implements against that. The spec becomes the artifact you review and\nversion; the code becomes a build output of it.
\nThe mental shift is subtle but total. In vibe coding you review code. In\nspec-driven development you review the spec, and you let tests review the code.
\nA spec doesn't have to be heavy. Often it's a short, precise document:
\n## Feature: careers listing
### Behavior
- Fetch open roles from GET /api/roles.
- Show role title, team, location, and an apply link.
- Sort by team (A→Z), then by title (A→Z).
### Constraints
- Roles with `status != \"open\"` MUST NOT be shown.
- If the API is unreachable, render the last cached response and a stale badge.
### Acceptance
- [ ] A closed role in the fixture never appears in the DOM.
- [ ] With the API mocked to 500, the stale badge renders.
- [ ] Sorting is stable and matches the fixture snapshot.
Now correctness lives in two places you can actually point at: the constraints\n(what must be true) and the acceptance tests (how we'll know). The model can\ngenerate the implementation, regenerate it, or a different model can — and the\ntests decide whether the result is acceptable. The human reviews intent, not\nline-by-line syntax.
\nThis is a large step up, and for a great deal of software it is the right and\nfinal stop. It gives you determinism where you need it: the same spec produces a\nresult that either passes the acceptance criteria or doesn't.
\nSo why is there a stage three? Because specs describe a destination, not a\njourney. A spec tells you what a correct result looks like. It says nothing\nabout what to do when the run that's supposed to produce that result is\nnon-deterministic — which is exactly the situation once an AI agent is doing the\nwork at runtime, in the wild, against inputs you didn't write fixtures for.
\nLoop engineering is what you build when the doer is itself an AI system\noperating without you in the chair. The spec is still there — you still need to\nknow what correct means — but now the thing that checks the work, catches the\nmistakes, and tries again has to be part of the system, because you're asleep or\nhandling ten thousand concurrent runs and cannot be the verifier.
\nThe unit of work is no longer a prompt or a spec. It's a loop: generate a\ncandidate, verify it against the spec, and if it fails, feed the failure back in\nand correct. The loop runs until the work passes or a budget is exhausted.
\n\nWritten as a control loop, the idea is almost boring — which is the point:
\ndef run(task, spec, max_attempts=4):
feedback = None
for attempt in range(max_attempts):
candidate = agent.generate(task, spec, prior_feedback=feedback)
report = verify(candidate, spec.criteria) # tests, evals, checks
if report.passed:
return candidate
feedback = report.failures # what went wrong, precisely
escalate(task, feedback) # hand to a human, with context
Everything hard about loop engineering is hidden inside verify and feedback.\nThe loop is trivial. Making the verifier trustworthy is the entire discipline:
r_912 had status closed but appeared in the output at index 3\" tells it\nexactly what to fix. The quality of the feedback string is, in practice, the\nquality of the loop.This is the stage where the earlier two are absorbed rather than discarded. The\nspec supplies the criteria the verifier checks against. The vibe-coding instinct\n— generate fast, react to what you see — becomes the inner generate step,\nexcept the thing reacting is now the verifier instead of your eyes.
Why this is the stage that ships. The industry's open secret is that most AI\nprojects never reach production — not because the models are weak, but because\nthey're wrapped like demos: one shot, no verifier, a human implicitly in the\nloop who quietly isn't there at 3 a.m. Loop engineering is what a demo becomes\nwhen you make the verification explicit and the human optional.
Running agents in production is loop engineering as a full-time job. The\nplatform-level features we talk about — agents that are self-correcting\n(they catch and fix their own mistakes mid-run), self-metriced (they pick and\ntrack the criteria they're judged on), and self-evolving (they get better\nfrom each outcome instead of plateauing on day one) — are all names for parts of\na well-built loop:
\nfail → diagnose → regenerate edge, made reliable.spec.criteria, so the\nverifier isn't frozen at whatever we guessed on day one.None of it is exotic once you see it as one control loop drawn at different\naltitudes. The demo is the inner box. The spec is the criteria. The loop is the\nproduct.
\nYou don't pick a stage as an identity; you pick it per piece of work, honestly.
\nThe mistake is not being at an early stage. Vibe coding a prototype is correct.\nThe mistake is staying at a stage after the work has quietly moved past it —\nshipping vibe-coded code as if it were verified, or shipping a spec as if the run\nthat fulfills it were deterministic. Each stage exists to fix the specific way\nthe previous one lies to you once the stakes go up.
\nKnow which lie you can currently afford. Build the next loop before you need it.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/vibe-coding-to-spec-driven-to-loop-engineering", "title": "From Vibe Coding to Spec-Driven Development to Loop Engineering", "summary": "How building with AI matures in three stages — vibe coding, spec-driven development, and loop engineering — and why each stage exists to fix the failure mode of the one before it.", "date_modified": "2026-07-03T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "Vibe Coding", "Spec-Driven Development", "Loop Engineering", "Agent Engineering", "AI in Production" ] }, { "id": "https://blog.catalex.co/agent-memory-that-compounds", "content_html": "\"Give the agent memory\" is one of those phrases that sounds like a feature and\nis actually an architecture. Everyone agrees agents should remember things\nacross sessions. Almost no one agrees on what that means — and the default\nimplementation, a vector database that everything gets dumped into, produces\nagents that are confidently wrong about their own past.
\n\nThis is a field note on how we think about memory for agents that run in\nproduction: what the pieces are, what each one is for, and the single principle\nthat governs all of them — an agent's memory is only as useful as what it\nrefuses to remember.
\nThe reflexive design is: embed every message, every tool result, every document,\nand store the vectors. At recall time, embed the current query and pull the top\nk nearest neighbors into the context window. This is easy, it demos beautifully,\nand it degrades in a specific, predictable way.
\nThe failure isn't that retrieval \"misses.\" It's that retrieval succeeds at the\nwrong thing. Nearest-neighbor search returns what is semantically similar to\nthe query, not what is relevant to the task. Ask an agent that has processed a\nthousand invoices about \"the Acme invoice\" and it will happily surface eleven\nAcme invoices from three different quarters, none flagged as superseded, and\nsplice them into context with equal authority. The model, being agreeable, will\nreconcile them into a confident answer that is subtly wrong.
\nThe tell: your agent gives a great answer, then gives a contradictory great\nanswer to the same question a day later, and both are traceable to \"something it\nremembered.\" That's not a retrieval bug you can tune away. It's a sign that\nstorage and recall were never separated from judgment about what mattered.
It helps to stop saying \"memory\" and name the pieces. Human cognition is a loose\nanalogy, but a useful one, because each type answers a different question.
\nWorking memory is the current context window: the task, the recent turns,\nthe tool outputs from this run. It is bounded, it is expensive, and it is the\nonly memory the model can actually reason over. Everything else exists to decide\nwhat gets promoted into it.
\nEpisodic memory is \"what happened.\" Specific events, tied to time and\noutcome: on June 3rd we ran the reconciliation for Acme and it failed because\nthe currency field was null; the human corrected it this way. Episodic memory is\nhow an agent learns from its own history instead of repeating it.
\nSemantic memory is \"what is true.\" Distilled facts that outlived the event\nthat produced them: Acme bills in EUR; this customer's fiscal year starts in\nApril; the approval threshold is 5,000. Semantic memory is what episodic memory\nbecomes once a pattern has been seen enough times to trust.
\nProcedural memory is \"how we do this here.\" The learned workflow — the order\nof steps, the checks that matter, the tool that actually works for a given task.\nIt's the difference between an agent that re-derives the process every run and\none that has a way of doing things.
\nThe mistake is storing all four as undifferentiated embeddings. They have\ndifferent write rules, different retrieval rules, and very different lifespans. A\nnull-currency incident is an episode that should decay. \"Acme bills in EUR\" is a\nfact that should persist and be retrieved by entity, not by vector similarity\nto the current sentence.
\nThe most important design choice in an agent's memory is not how it retrieves.\nIt's what it writes down, and that should be a deliberate step, not an automatic\nlog of everything that streamed through the context.
\nA memory-write worth making usually clears three bars:
\nIn practice this means an agent should reflect before it writes: at the end of\na run, summarize what changed, promote the durable facts to semantic memory,\nkeep the specific episode with its outcome, and drop the rest. This is the same\ngenerate-then-verify discipline that governs a good agent loop, pointed inward.
\nOnce memory is typed, retrieval stops being one vector search and becomes a small\nset of targeted ones. Facts about an entity are fetched by that entity. Recent\nepisodes are fetched by recency and relevance, with the outcome attached so the\nmodel can weight a success differently from a failure. Procedures are fetched by\ntask type. Vector similarity is one signal among several — useful for the fuzzy\n\"have we seen anything like this before,\" dangerous as the only lens.
\nThe other half of retrieval is budgeting. Working memory is finite and\nattention is not free; a context window stuffed with fourteen marginally relevant\nmemories reasons worse than one holding the three that matter. Recall should be\nranked and capped, and the cap should be small. More retrieved memory past a\npoint makes agents worse, not better — the signal gets buried and the model\nstarts pattern-matching on the loudest neighbor instead of the most relevant one.
\nHere is the part that feels wrong and is essential: a good memory system spends\nas much design effort on removal as on storage.
\nFacts get superseded — the approval threshold changed, the customer switched\ncurrencies, the API you learned to call was deprecated. If memory only appends,\nevery one of those updates becomes a contradiction the model has to resolve at\nread time, usually by picking one at random. So memories need to be\nupdatable (the new fact replaces the old, with the old one retired, not\ndeleted silently) and decayable (episodes lose weight over time unless\nreinforced). \"Remembers everything\" is not the goal. An agent that cannot forget\na fact that stopped being true is worse than one with no memory at all, because\nit is wrong with history behind it.
\nThis is also where trust and safety live. Memory is an injection surface: content\nan agent reads today can be written into memory and re-executed as instruction\ntomorrow. Typed, attributed, expirable memory is what lets you audit that — to\nanswer \"why did the agent believe this?\" with a specific provenance chain rather\nthan a shrug at a vector store.
\nThe reason to do the harder version is that memory is the mechanism by which an\nagent gets better instead of just staying the same. A stateless agent performs\nthe same on run 10,000 as on run 1 — it re-derives everything, repeats every\navoidable mistake, and never accumulates the specific, hard-won knowledge of\nyour domain that a good human teammate builds over months. An agent with\ntyped, curated, decaying memory does the opposite: each run leaves the next one a\nlittle more competent, because the durable lessons persist and the noise doesn't.
\nAt CatalEx this is what we mean when we say agents are self-evolving and carry\ncontext across sessions. It isn't a bigger vector store. It's the discipline of\ndeciding, every run, what was worth remembering — and what the agent is better\noff forgetting.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/agent-memory-that-compounds", "title": "Memory That Compounds: What \"Persistent\" Actually Means for an Agent", "summary": "Agent memory is not a vector database. A practical model for what an agent should write, retrieve, and forget — and why \"remembers everything\" is a bug, not a feature.", "date_modified": "2026-07-02T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "Agent Memory", "Agent Engineering", "AI in Production" ] }, { "id": "https://blog.catalex.co/the-eval-is-the-product", "content_html": "Every team that has tried to ship an AI feature has hit the same wall. The demo\nworks. A stakeholder asks, \"is it good?\" and the honest answer is a shrug. You\nfeel like it's good. You changed a prompt and it feels a little better, or a\nlittle worse, and you cannot say which, because \"feels\" is the only instrument\nyou have.
\n\nThat wall is the absence of an eval. And the reason so many AI projects stall at\n80% — good enough to demo, never good enough to trust — is almost never the\nmodel. It's that the team never built the one thing that turns \"feels good\" into\na number you can defend: a way to measure whether the system actually works.
\nTraditional software has a comfortable property: correctness is usually binary.\nThe function returns the right value or it doesn't. A test suite is a wall of\ngreen checks, and green means ship.
\nAI systems break that comfort. \"Correct\" becomes a spectrum. A summary can be\naccurate but bury the lede. An agent can complete the task but take a reckless\npath to get there. Two answers can both be right and one can be clearly better.\nThe output space is open-ended, the same input can produce different outputs run\nto run, and the thing you care about — helpfulness, faithfulness, safety — isn't\na return value you can assert on.
\nAn eval is what you build to measure a spectrum instead of a boolean. At its\ncore it is three things: a dataset of representative inputs, a grader\nthat scores each output, and an aggregate score you can track over time.\nThat's it. The engineering is entirely in making each of those three trustworthy.
\nIt's worth being blunt about what the absence of an eval costs, because the costs\nare invisible until they compound.
\nYou can't tell improvement from regression. You tweak a prompt to fix one bad\ncase and silently break five others you never looked at. You have no way to know,\nso you ship it.
\nYou can't compare models. A cheaper model comes out; is it good enough for your\ntask? Without an eval, \"good enough\" is a meeting, not a measurement, and the\nmeeting is won by whoever is most confident.
\nYou can't safely change anything. Every prompt edit, tool change, or model\nupgrade is a gamble on a system whose behavior you can only spot-check. So teams\nstop changing things, and the product freezes at its first acceptable version.
\nAnd you can't close the loop. This is the one that matters most to us: an\nautonomous agent that verifies and corrects its own work is only as honest as the\nverifier inside it. That verifier is an eval, running at runtime. A team that\nnever learned to write a trustworthy grader offline has no chance of embedding a\ntrustworthy one at runtime.
\nAnyone can assemble a dataset. The discipline is in the grader, and graders come\nin a rough hierarchy of cost and fuzziness.
\nProgrammatic checks are the cheap, sharp layer: exact match, JSON-schema\nvalidity, \"did it call the right tool with the right arguments,\" \"does the code\ncompile and pass tests.\" When a task has a checkable ground truth, use it.\nThese are fast, deterministic, and free of the ambiguity that dooms softer\ngraders. Most teams underuse them because they reach for a model before checking\nwhether the property is mechanically verifiable.
\nLLM-as-judge is for the genuinely open-ended: is this summary faithful to the\nsource, is this reply helpful, is this reasoning sound. A second model grades the\nfirst against a rubric. It's powerful and it's a trap in equal measure, because a\nsloppy judge manufactures false confidence at scale. The failure modes are\nwell-documented and consistent: judges prefer longer answers, prefer the first\noption shown, and prefer their own style. A judge is only usable once you have\nevaluated the evaluator — checked its scores against human labels on a sample\nand confirmed it agrees often enough to trust.
\nHuman grading is the ground truth the other two are calibrated against. It's\nslow and expensive, so you spend it deliberately: to build the golden set, to\naudit the automated graders, and to adjudicate the cases the judges disagree on.\nYou don't scale it; you use it to make the things that scale honest.
\nA verifier that rubber-stamps is worse than no verifier. A green eval that\nisn't measuring what you think it is doesn't just fail to help — it actively\nhides regressions behind a number people trust. Before you believe a grader,\nmake it disagree with you at least once on a case where you know the answer.
An eval is not a milestone you pass once. It's a living asset, and it decays in\ntwo directions.
\nIt decays against reality: your dataset was drawn from last quarter's inputs, and\nproduction has moved on. New kinds of requests appear; the eval doesn't cover\nthem; your score stays green while real quality slips. The fix is a flywheel —\nevery genuinely wrong production case that surfaces becomes a new eval example, so\nthe dataset tracks the distribution it's supposed to represent.
\nAnd it decays against gaming: any single metric, optimized hard enough, stops\ncorrelating with the thing it was a proxy for. Optimize a helpfulness judge and\nyou get verbose, hedging answers that read as helpful. The defense is a small\npanel of graders looking at different failure modes — faithfulness, concision,\nsafety, task success — so that gaming one costs you another.
\nThe teams that get this right run evals the way they run tests: in CI, on every\nchange, with the score gating the merge. A prompt edit that drops the eval is a\nregression, full stop, the same as a failing unit test. That single move —\nmaking the number block the merge — is what converts evaluation from a report\nnobody reads into the thing that actually governs the system.
\nPoint the whole picture at an autonomous agent and the stakes get concrete. The\nagent generates a candidate, a verifier scores it against criteria, and on\nfailure it corrects and tries again. That verifier is an eval running in the\nhot path. Its dataset is the live task, its grader is your checks and judges, its\nscore is the gate between \"commit\" and \"try again.\" Every property that makes an\noffline eval trustworthy — real coverage, calibrated graders, resistance to\ngaming — is exactly what makes the runtime loop safe to leave unattended.
\nWhich is why we don't treat evaluation as QA that happens after the product is\nbuilt. The eval is the part of the product that decides whether any of the rest\nof it can be trusted. Build that first, and everything downstream — shipping,\nimproving, running agents without a human in the chair — becomes possible. Skip\nit, and you're left with a great demo and a shrug.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/the-eval-is-the-product", "title": "The Eval Is the Product: Why You Can't Ship What You Can't Measure", "summary": "Evals are not a testing afterthought — they are the artifact that decides whether an AI system can ship, improve, or be trusted. A practical case for treating evaluation as the product.", "date_modified": "2026-07-01T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "Evaluation", "Agent Engineering", "Loop Engineering" ] }, { "id": "https://blog.catalex.co/harness-vs-model", "content_html": "The most expensive misconception in applied AI is that performance lives in the\nmodel. Pick the best model, the thinking goes, and the rest is glue. So teams\nspend their attention on which model to call and treat everything around it — the\ntools, the loop, the context they assemble, the way they check the output — as\nplumbing.
\n\nThe data says the opposite. The scaffolding around the model — what we'll call\nthe harness — routinely moves results as much as swapping the model does,\nand sometimes more. The same weights, handed a better harness, become a\ndifferent product. If you're deciding where to spend the next month of\nengineering, this is the argument for spending it on the harness.
\nThe harness is everything between the user's intent and the model's tokens:
\nNone of that is the model. All of it decides what the model can accomplish.
\nThe cleanest evidence comes from SWE-bench, a benchmark where an agent has to\nresolve real GitHub issues by editing a codebase until the project's tests pass.\nIt's a good measuring stick precisely because the grader is objective — the tests\neither pass or they don't.
\nWatch what happens to the field's score over a couple of years, as both models\nand harnesses improved together:
\nApproximate resolved rates, SWE-bench / SWE-bench Verified. Sources:\nSWE-bench leaderboard; Yang et al., \"SWE-agent\" (2024); public vendor\nreports. Exact figures depend on model version and evaluation date.
A jump from roughly 1% to roughly 67% is not a story about weights getting 60x\nbetter. Model quality improved a great deal in that window — but a large share of\nthe climb came from learning how to wrap the model: giving it a code-navigation\ntool instead of raw file dumps, letting it iterate against the test suite,\nmanaging what stayed in context. The harness matured alongside the model.
\nYou can see the harness's contribution in isolation by holding the model fixed\nand changing only the scaffold. The pattern is consistent across published\nresults: a capable model, asked to emit a patch in one shot, badly underperforms\nthe same model given tools and a loop.
\nRepresentative of the gap reported between minimal and agentic scaffolds on a\nfixed model. Illustrative figures consistent with SWE-agent (2024) and vendor\nagent reports; the point is the shape, not the decimal.
Same weights. Roughly a 40-point spread. The harness is not glue around the\nperformance — a large fraction of the performance is the harness.
\nWhy this happens: a single forward pass has to be right the first time, with\nwhatever context it was handed. A harness converts a one-shot guess into a\nprocess — look, act, observe, check, correct. That turns the model's real\nstrength (reacting well to feedback) into task success, and it's exactly the\nstrength a single shot never gets to use.
There's a second reason to invest in the harness, and it's about time, not just\nscores. Model choice is a decision you re-make every few months — a new frontier\nmodel ships and you re-evaluate. Every dollar of prompt-whispering tuned to one\nmodel's quirks partially evaporates at the next upgrade.
\nHarness investment does the opposite. A good code-navigation tool, a\nwell-designed verification step, a context-assembly strategy that keeps the\nwindow clean — these carry forward onto the next model, and usually get better\nunderneath it, because a stronger model uses a good tool more effectively. The\nharness is the part of your system that accrues value across model generations\ninstead of being reset by them.
\nThis also reframes the \"which model\" debate. On a mature harness, the gap between\nthe top two or three frontier models is often smaller than the gap between a good\nand a mediocre harness on either of them. The model is a component you swap; the\nharness is the product you build.
\nIf the harness is where the leverage is, a few principles follow.
\nMake it model-agnostic. Treat the model as a replaceable component behind an\ninterface. When the next model ships, you want to run your eval suite against it\nin an afternoon, not rewrite your prompts. The teams that move fastest on new\nmodels are the ones whose harness didn't hard-code the old one.
\nInvest in tools and their ergonomics, not just their existence. A tool whose\noutput is a wall of unstructured text costs the model context and attention. The\nsame tool returning a tight, structured result is worth more than a model\nupgrade. Shape tool I/O the way you'd shape an API for a junior engineer who\nreads fast but tires easily.
\nBuild the verification step first, not last. The loop that lets a model check\nand correct its own work is the single highest-leverage piece of the harness — it\nconverts a capable one-shot model into a reliable one. Everything we've written\nabout loop engineering and evals is really about this component of the harness.
\nManage context like a budget. More retrieved text is not more capability past\na point; it's noise that buries the signal. A harness that puts the three right\nthings in the window beats one that puts fourteen plausible ones.
\nAt CatalEx this is the whole thesis of the product. The platform is a harness —\ntools, orchestration, memory, verification, model routing — built so that the\nmodel underneath is a component you can swap as the frontier moves, while the\nscaffolding that actually determines whether an agent works keeps compounding.\nPick your model carefully. Then spend the real effort on everything around it,\nbecause that's where the performance was hiding all along.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/harness-vs-model", "title": "The Harness Is Half the Model: Why Scaffolding Decides Performance", "summary": "The same model can swing 40 points on the same benchmark depending on the harness around it. A data-grounded look at why the scaffolding — tools, loop, context, verification — is as decisive as the model you pick.", "date_modified": "2026-06-30T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "Agent Harness", "Agent Engineering", "AI in Production" ] }, { "id": "https://blog.catalex.co/mcp-vs-cli-for-integration", "content_html": "There's a tidy argument going around: skip the Model Context Protocol, just give\nthe agent a shell and a CLI. No server to run, no schemas to maintain, and — the\nclincher — it saves tokens, because a command is short and a tool definition is\nlong. It's a good argument. It's also only half right, and the missing half is\nwhere most of the cost actually lives.
\n\nThis is a look at the two ways to wire an agent into a third-party system — an\nMCP server versus a command-line tool — with particular attention to the claim\nthat the CLI is the cheaper one. The honest answer is \"sometimes, and not for the\nreason people think.\"
\nMCP is a protocol for exposing tools and data to an agent in a structured\nway. You run a server that advertises a set of tools, each with a typed schema\nfor its inputs and outputs. The agent's runtime loads those definitions, the\nmodel sees them, and it calls them as structured functions with structured\nresults coming back.
\nThe CLI approach skips all of that. You give the agent a shell tool and tell\nit a command exists. It composes a command string, the command runs, and the raw\nstdout comes back as text for the model to read. No schema, no server — the\nintegration surface is \"the agent knows how to use a terminal.\"
\nBoth work. They fail and cost differently, and the difference is almost entirely\nabout tokens and structure.
\nThe pro-CLI token argument focuses on one side of the ledger: tool-definition\noverhead. An MCP server that exposes twenty tools injects twenty schemas into\ncontext on every request, whether or not any of them get used. That's real, and\nit's not small — a directory of chatty MCP servers can burn thousands of tokens\nbefore the model has done anything. A CLI, by contrast, costs almost nothing to\nadvertise: one line saying the command exists.
\nBut there are two sides to the ledger, and the other side is output.
\nThe half that gets ignored: a CLI returns whatever it prints — and most CLIs\nprint for humans, not models. git status, kubectl get, npm install, a\nverbose API client: they emit banners, progress bars, decoration, and pages of\ndetail the model doesn't need. Every character is input tokens on the next\nturn. A single noisy command can cost more tokens than the MCP schema you were\ntrying to avoid.
So \"CLI saves tokens\" collapses into a real question: does your integration make\nfew calls against many tools (definition overhead dominates → CLI looks\ncheaper), or many calls returning verbose output (output dominates → the CLI\ncan be far more expensive)? The answer depends on the workload, not on the\nmechanism.
\nThere's a reason a whole category of tooling exists to sit between an agent and\nits CLIs and strip the output down — filtering, summarizing, and reshaping\ncommand results before they hit the context window. (Our own developer setup\nleans on exactly this kind of proxy; more on that in a follow-up.) The existence\nof that category is the tell: raw CLI output is a token liability that teams\nactively engineer around. The CLI isn't lean by default. It's lean after you do\nthe work MCP's typed outputs give you for free.
\nToken cost is the loudest part of the debate and the least important once you've\nsized it. Four other differences matter more in production.
\nStructure and reliability. MCP hands the model typed arguments and typed\nresults. The CLI hands it a string to compose and a string to parse. Every\nun-typed boundary is a place the model can get the flag wrong, misread a table,\nor hallucinate a field that isn't there. Structured I/O isn't just tidier; it\nremoves whole classes of silent error.
\nDiscoverability. With MCP, the agent can enumerate what's available and how\nto call it. With a CLI, it relies on training-data familiarity or on reading\n--help (more tokens) — and it will confidently invent flags for tools it half-\nremembers. Popular tools it knows well; your bespoke internal CLI it does not.
Composition. This one favors the CLI. Shells pipe, chain, and glue tools\ntogether in ways that are genuinely powerful, and a model that's good at bash can\naccomplish a lot with primitives you never explicitly exposed. MCP tools compose\nonly as far as you designed them to.
\nSecurity and blast radius. A shell is arbitrary code execution by\nconstruction. Handing an autonomous agent a terminal is handing it the ability to\ndo anything the shell can, which means sandboxing, allowlists, and audit become\nyour problem. An MCP tool is a narrow, declared capability — the agent can do\nthat, and only that. For anything touching production or customer data, that\ncontainment is worth a lot.
\nNeither wins categorically. The decision falls out of the workload.
\nReach for MCP when the capability is a durable, high-value operation you'll\ncall often; when the output should be structured for the model rather than for a\nhuman; when you need containment and audit; or when the same capability will be\nreused across many agents. Typed, discoverable, narrow — that's worth the server.
\nReach for the CLI when the tool already exists and is well-known, when you\nneed ad-hoc composition more than a fixed contract, when you're prototyping and\ndon't want to stand up infrastructure, or when the operations are one-off enough\nthat schema maintenance would cost more than it returns. Just budget for the\noutput problem — plan to filter what comes back.
\nAnd often, build both. A common, effective shape is a thin MCP server that\nwraps a CLI: the model gets a typed, discoverable, containable interface, while\nthe server internally shells out and — critically — reshapes the raw output into\na compact, structured result before it returns. You get the CLI's reach and the\nMCP's ergonomics, and you pay the output-token cost once, in code you control,\ninstead of on every model turn.
\nThe metric that settles it: optimize tokens-per-successful-task, not\ntokens-per-call. A CLI that's cheap to invoke but returns noise the model has to\nre-read across five correction turns is more expensive than an MCP call that\ncosts more up front and gets it right the first time. Measure the loop, not the\nline.
\"The CLI saves tokens\" is true in exactly the narrow case where you have many\nrarely-used tools and terse, model-friendly output — and false the moment output\ngets verbose, which for real-world CLIs is most of the time. The tokens you save\non definitions, you often hand right back on results, plus a parsing tax and a\nsecurity surface.
\nSo don't pick the mechanism for its reputation. Size your workload — call\nfrequency, tool count, output verbosity, containment needs — and pick the\ninterface that minimizes tokens-per-successful-task while keeping the blast radius\nsurvivable. Usually that's MCP for the durable core, CLIs for the long tail, and\nan output filter wherever raw command text would otherwise flood the window. The\n\"cheaper\" option is the one you engineered to be cheap, not the one that started\nthat way.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/mcp-vs-cli-for-integration", "title": "MCP vs. CLI for Third-Party Integration: Is the CLI Really Cheaper?", "summary": "Giving an agent a CLI feels leaner than standing up an MCP server — but \"CLI saves tokens\" is only half true. A practical comparison of the two integration styles and what to actually build.", "date_modified": "2026-06-29T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "MCP", "Integrations", "Agent Engineering" ] }, { "id": "https://blog.catalex.co/five-papers-that-shaped-modern-ai", "content_html": "If you want to understand why AI works the way it does today, you don't need to\nread a hundred papers. You need to read about five, in order, and understand what\neach one changed. Everything since is, to a first approximation, a refinement or\na scaling-up of ideas these introduced.
\n\nThis is that list — the five we'd hand a strong engineer joining the field. Not\nthe five \"best\" papers by citation count, but the five that each moved the ground\nunder everyone's feet. Read in sequence, they tell one continuous story: how we\ngot from machine translation to agents.
\nArchitecture → scale → reasoning → alignment → efficient training. Three of\nthe five landed within a few months of each other in early 2022.
Vaswani et al. · June 2017 · arXiv:1706.03762
\nWhy it mattered: it invented the Transformer, the architecture underneath every\nlarge model since.
\nBefore this, sequence models were recurrent — they processed tokens one at a\ntime, which made them slow to train and forgetful over long inputs. The\nTransformer threw out recurrence entirely and replaced it with self-attention:\na mechanism that lets every token look directly at every other token in one step.\nThat single change did two things. It let models learn long-range relationships\nthat recurrence kept losing, and — because attention is parallelizable — it let\ntraining scale across hardware in a way recurrence never could. The paper's title\nwas a swagger, and it earned it. Almost nothing that came after is possible\nwithout it; when people say \"an LLM,\" they mean a very large Transformer.
\nBrown et al. · May 2020 · arXiv:2005.14165
\nWhy it mattered: it showed that scale alone unlocks a new capability —\nlearning from the prompt, with no training.
\nGPT-3 took the Transformer and made it enormous: 175 billion parameters trained on\na large slice of the internet. The headline wasn't the size, though — it was\nin-context learning. Give the model a few examples of a task in the prompt,\nand it performs the task, with no fine-tuning and no weight updates. This\nreframed the entire practice of using models. You stopped training a model per\ntask and started prompting one general model, and \"prompt engineering\" became a\ndiscipline overnight. It's also the paper that made scale the field's central\nbet: make the model bigger, and qualitatively new abilities appear.
\nWei et al. · January 2022 · arXiv:2201.11903
\nWhy it mattered: it showed that asking a model to show its work unlocks\nreasoning it otherwise can't do.
\nLarge models were oddly bad at multi-step problems — arithmetic, logic, word\nproblems — even as they aced other tasks. The fix turned out to be almost\nembarrassingly simple: prompt the model to produce intermediate reasoning steps\nbefore the final answer, and accuracy on hard problems jumps. \"Let's think step\nby step\" became a genuine capability unlock, not a party trick. More deeply, it\nrevealed that a lot of a model's competence is latent — present in the weights\nbut only accessible if you give it room to compute. Every \"reasoning model\" that\nfollowed is a descendant of this observation.
\nOuyang et al. · March 2022 · arXiv:2203.02155
\nWhy it mattered: it turned raw, unruly language models into helpful assistants —\nthe step that made the technology usable by everyone.
\nA raw pretrained model predicts likely text; it does not, by default, do what you\nask. InstructGPT introduced RLHF — reinforcement learning from human\nfeedback — to close that gap: humans rank model outputs, a reward model learns\nthose preferences, and the language model is tuned to produce what humans prefer.\nThe result was startling: a much smaller RLHF-tuned model was preferred over the\ngiant base model, because it was actually trying to be helpful. This is the paper\nthat made the assistant paradigm work, and it's the direct methodological\nancestor of ChatGPT and every chat model since. Alignment stopped being a safety\nfootnote and became the productization step.
\nHoffmann et al. · March 2022 · arXiv:2203.15556
\nWhy it mattered: it corrected how the whole field spends its compute — and\nproved most big models were trained wrong.
\nThe prevailing wisdom after GPT-3 was \"bigger is better,\" so labs raced to add\nparameters. Chinchilla showed that for a fixed compute budget, parameters and\ntraining data should scale roughly together — and that the giant models of the\nday were badly undertrained, starved of data relative to their size. A\nsmaller model trained on more tokens beat a larger one trained on fewer. This\nreshaped training strategy across the industry: data quality and quantity became\nfirst-class, and \"compute-optimal\" entered the vocabulary. It's why the strongest\nmodels aren't simply the biggest ones.
\nRead those five and you have the spine: an architecture that scales (1), the\ndiscovery that scale creates capability (2), a way to summon reasoning from it\n(3), a way to make it helpful (4), and a way to train it efficiently (5).
\nEverything we work on sits one step past the end of this list. Once models could\nreason and follow instructions, the obvious next move was to let them act — to\ncall tools, observe results, and loop — which is the agent paradigm and the line\nof work (ReAct, Toolformer, and what followed) that this blog mostly lives in. If\nyou want the sequel to this reading list, that's where it starts. But start here.\nThese five are the grammar; the rest is composition.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/five-papers-that-shaped-modern-ai", "title": "Five Papers That Shaped Modern AI (Start Here)", "summary": "A short, opinionated reading list — from \"Attention Is All You Need\" to compute-optimal scaling — with a timeline, references, and a paragraph on why each one still matters.", "date_modified": "2026-06-28T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "AI Research", "Engineering Culture" ] }, { "id": "https://blog.catalex.co/managing-context-with-claude", "content_html": "When you build software with an AI agent day to day, the thing you run out of\nfirst isn't intelligence — it's context. The window is finite, attention\ninside it isn't free, and a session that starts sharp gets duller as it fills\nwith the exhaust of the work: verbose command output, whole files read to find\none function, stale documentation, the residue of three tasks ago.
\n\nSo a real part of building CatalEx is context hygiene — keeping the model's\nwindow full of signal and empty of everything else. This is the setup we\nactually run, and how each piece is configured. The theme underneath all of it\nis the same one from our post on harnesses: spend context like a budget, and\nmake every token earn its place.
\nIt's tempting to optimize the wrong number. The instinct is to minimize tokens\nper call — shorter prompts, fewer tools. But the metric that actually matters is\ntokens per successful task. A window bloated with noise makes the model reason\nworse, which causes correction turns, which costs far more tokens than the noise\nyou were trying to trim.
\nEverything below is in service of that number. Some tools cut what enters the\nwindow; some cut what the model has to emit; some replace a large, vague context\nwith a small, precise one. All of them are trying to get the task done in fewer\ntotal tokens by keeping the window clean, not just short.
\nThe single biggest source of context bloat in day-to-day development is command\noutput. git status, npm install, a test run, a container log — these print\nfor humans, in full, and every character lands back in the model's context on the\nnext turn.
RTK (a token-optimizing CLI proxy) intercepts this at the source. It runs as\na PreToolUse hook, so it's transparent — the agent runs a normal command and\nRTK rewrites it through a proxy that strips the output down to what matters\nbefore it ever reaches the window. In our global Claude config it's wired as:
{
\"hooks\": {
\"PreToolUse\": [
{ \"hooks\": [{ \"type\": \"command\", \"command\": \"rtk hook claude\" }] }
]
}
}
The effect is invisible and large: routine dev operations come back 60–90%\nsmaller with no change to how you drive the agent. git status returns the\nsignal, not the decoration. When you genuinely need the raw firehose, there's an\nescape hatch (rtk proxy <cmd>) that bypasses the filtering — but the default is\nlean, which is the right default.
RTK trims what tools put in; caveman trims what the model and you put into\nthe conversation. It's a plugin that switches communication into an\nultra-compressed mode — the tagline is \"why use many token when few do trick\" —\ncutting roughly 75% of conversational tokens while preserving technical accuracy.\nIt's enabled per-repo in .claude/settings.json:
{
\"enabledPlugins\": { \"caveman@caveman\": true },
\"extraKnownMarketplaces\": {
\"caveman\": { \"source\": { \"source\": \"github\", \"repo\": \"JuliusBrussee/caveman\" } }
}
}
The tradeoff is real and worth stating: compression is lossy, and there are\nmoments — subtle design discussion, careful reasoning about a tricky bug — where\nyou want the model spending tokens, not saving them. Caveman is a lever, not a\nlaw. We lean on it for the high-volume, low-nuance stretches of work and ease off\nwhen the thinking gets delicate.
\nA quieter context problem is wrong context. A model's training data has a\ncutoff, so its memory of a fast-moving library is often months stale — and the\nusual fix, pasting in whole documentation pages, floods the window to answer one\nquestion.
\ncontext7 (an Upstash MCP server) solves both ends. It pulls version-specific\ndocumentation and code examples from the source on demand, so the model gets the\ncurrent API for the exact version we're on, scoped to what was asked, instead\nof hallucinating from stale memory or drowning in a doc dump. It's enabled the\nsame way:
\n{ \"enabledPlugins\": { \"context7@claude-plugins-official\": true } }
This is context management by substitution — replacing a large, unreliable\ncontext (training memory, or a pasted doc site) with a small, accurate one.
\nOne of the least obvious context sinks is exploration. To change one function, an\nagent will happily read the whole file it lives in — and the two files it imports\n— just to understand the types. That's a lot of tokens to answer \"what's the\nsignature of this thing.\"
\nWe run language-server plugins (pyright-lsp, typescript-lsp) so the agent has\nprecise, symbol-level intelligence: go-to-definition, hover types, references. It\ncan ask \"what is this symbol\" and get a one-line answer instead of reading three\nfiles to infer it. Across a working session that's an enormous saving, and it\nalso makes the edits more reliable — the model is acting on the actual type, not\na guess assembled from surrounding code.
The last piece is structural. When several pieces of work happen in parallel,\ntheir contexts bleed into each other — and a window carrying the residue of three\nhalf-finished tasks reasons worse on the one in front of it.
\nWe use git worktrees with lifecycle hooks so each task runs in its own isolated\ncheckout, with its environment set up and torn down automatically:
\n{
\"hooks\": {
\"WorktreeCreate\": [
{ \"hooks\": [{ \"type\": \"command\", \"command\": \".claude/hooks/setup-worktree.sh\" }] }
],
\"WorktreeRemove\": [
{ \"hooks\": [{ \"type\": \"command\", \"command\": \".claude/hooks/remove-worktree.sh\" }] }
]
}
}
The context benefit is that each session's window stays scoped to a single task —\nits files, its branch, its environment — instead of accumulating the state of\neverything else in flight. Isolation is a context-management technique as much as\na workflow one.
\nThe pattern across all of these: none is a clever prompt. Each is a piece of\nharness — a hook, a plugin, a proxy, an isolation boundary — that shapes what\nreaches the model. Context management isn't something you do by asking nicely in\nthe prompt; it's something you build into the environment the agent runs in.
Put together, the setup layers cleanly by what it targets:
\nEvery one of them is pointed at the same number — total tokens to get the task\ndone right — and every one is configuration, not willpower. That's the real\nlesson: managing context while building with Claude is the same discipline as\nbuilding a good agent harness, turned around and aimed at your own development\nloop. Set the environment up so the window stays lean, and the model does its\nbest work for longer before the exhaust builds up.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/managing-context-with-claude", "title": "How We Keep Claude's Context Lean While Building CatalEx", "summary": "Context is the scarce resource when you build with an AI agent. The actual setup we run to keep the window lean — RTK, caveman, context7, LSP intel, and worktree isolation — and how each is configured.", "date_modified": "2026-06-27T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "Engineering Culture", "Agent Engineering", "AI in Production" ] }, { "id": "https://blog.catalex.co/write-the-wiki-for-the-llm", "content_html": "Here is a shift that has already happened, whether or not your documentation\nnoticed: a large and growing share of the readers of everything you write are no\nlonger human. They're language models — reading your docs to answer a question,\nyour README to use your library, your marketing page to tell someone what you do.\nAndrej Karpathy has been making this point for a while, and it lands harder every\nmonth: the primary reader of a lot of your writing is now an LLM, and you are\nprobably still writing for a human.
\n\nThis is a look at that idea — sometimes called writing an \"LLM wiki,\" or\nknowledge for machines — and what it actually changes about how you produce docs.\nIt's not a call to stop writing for people. It's the recognition that you now\nhave a second, very different reader, and that reader has been quietly ignored.
\nA human reading your docs skims. They jump to the code block, infer meaning from\na screenshot, tolerate a nav bar and a cookie banner and three paragraphs of\npositioning, and click around until they find the part they need. Human docs are\nbuilt around all of that: visual hierarchy, interactivity, and the reader's\nwillingness to fill gaps.
\nAn LLM reads none of it that way. It ingests text mostly linearly. It can't click\nyour \"see also.\" It can't see the screenshot that contained the actual answer. It\nchokes on the chrome — the navigation, the boilerplate, the styling markup — and\nspends context on all of it. And critically, it will not tell you it got\nconfused; it will confidently produce an answer from whatever it managed to\nparse, gaps included.
\nSo the same page that works beautifully for a person can be nearly useless to a\nmodel: the key fact was in an image, the steps were split across five clicked\npages, and half the tokens were spent on a header that repeats on every route.
\nKarpathy's framing is useful because it's concrete once you take the reader\nseriously. Writing for the model reader means:
\nThe mental test is simple: paste the raw text of your page into a model, ask it\nthe question your user would ask, and see if it can answer from that alone. If it\ncan't, your human-facing polish is hiding the substance from your fastest-growing\naudience.
\nThe clearest instantiation of this idea already exists and is easy to adopt.\nllms.txt — a convention proposed by Jeremy Howard — is a plain-markdown file at\nthe root of your site that gives models a clean, curated map of what you are and\nwhere the real information lives, free of the HTML and navigation they'd\notherwise wade through. It's a wiki entry for your site, written for the machine.
The broader \"LLM wiki\" idea generalizes that: a body of knowledge maintained\nspecifically for model consumption — unambiguous, structured, complete, and\nkept current. Not a copy of your human docs with the styling stripped, but\nknowledge authored with the model as the intended reader. Where Wikipedia\norganized the world's knowledge for people to read, the LLM wiki organizes it for\nmodels to reason over.
\nThis is the same move as the search era, one layer up. For two decades, teams\nlearned to write for Google's crawler — that was SEO. The reader was an index.\nNow the reader is a model that will synthesize an answer and cite whoever it\nunderstood best. Writing for the LLM is the AEO successor to writing for search:\nthe audience changed from an index to a reasoner, and the craft changed with it.
The payoff is direct and it compounds. When a user asks ChatGPT, Claude, or an AI\nsearch engine about your problem space, the answer is assembled from whatever\nsources the model could parse and trust. Documentation written for the model is\ndocumentation the model can actually use — which means you get represented\naccurately in the answer, and cited, instead of paraphrased wrong or skipped for\na competitor whose page was cleaner to read.
\nThere's an internal payoff too. The same properties that make knowledge good for\nan external model make it good for your own agents. An agent that reads your\ninternal docs to do its job has exactly the LLM reader's constraints — linear,\ncontext-limited, unable to click. Knowledge authored for machines is knowledge\nyour own automation can act on. The wiki you write for the world's models is the\nwiki your own agents read too.
\nWe try to practice this, including on the blog you're reading. It ships an\nllms.txt that describes what CatalEx is in plain markdown; a robots.txt that\nexplicitly welcomes AI crawlers rather than blocking them; structured data so a\nmodel can attribute a post correctly; and posts written to be self-contained —\nthe argument is in the prose, not locked in an image. The goal is that when a\nmodel reads any of it, it comes away with an accurate, citable understanding, not\na pile of chrome it has to guess through.
None of this is exotic. It's the old discipline of writing clearly, with one\nupdate to the audience: the most frequent, most literal, least forgiving reader\nof what you publish is now a model. Karpathy's point is really just that — take\nthat reader seriously. Write the wiki for the LLM, and you serve the humans who\nask it questions at the same time.
\nWritten by CatalEx Engineering. We build the AI operating layer for AI-native\ncompanies — one platform to build, deploy, and run AI agents in production. More\nat catalex.co.
", "url": "https://blog.catalex.co/write-the-wiki-for-the-llm", "title": "Write the Wiki for the LLM: Documentation's New Reader", "summary": "Andrej Karpathy's point that your docs' primary reader is now a model, not a human — and what an \"LLM wiki\" actually looks like in practice, from llms.txt to knowledge written for machines.", "date_modified": "2026-06-26T09:00:00.000Z", "author": { "name": "CatalEx Engineering", "url": "https://catalex.co" }, "tags": [ "AI Research", "Engineering Culture" ] } ] }