Hallucination Isn't the Problem. Execution Is.
"AI hallucinates" is a true sentence that has been repeated into harmlessness. It sounds like a quirk — the chatbot got a trivia question wrong, everyone screenshots it, nothing happens.
Now move the same behavior inside a company. A finance agent invents a figure and it lands in a board deck. A support agent invents a refund policy and tells a customer, which in several jurisdictions means you now have that policy. A coding agent invents an API, writes code around it, and the failure surfaces in production at 2 a.m. Same underlying behavior in all four cases. Wildly different consequences, and the difference has nothing to do with the model.
The difference is how much authority sat downstream of the output. In the trivia case, the answer was read by a human who could shrug. In the other three, something in the system took the output and did something with it — put it in a document, said it to a customer, executed it — without anything in between asking whether it was true.
That gap is the thing you control. The hallucination rate is largely not.
Stop optimizing the number that won't reach zero
Teams spend enormous effort trying to make the model never be wrong: better prompts, bigger models, more context, more few-shot examples. It's worth doing, and it works — a good grounding setup can cut fabrication rates dramatically.
It never reaches zero, and the reason is structural rather than a temporary quality gap. A model that generates fluent text is generating the most plausible continuation, and plausible and true are correlated, not identical. The places where they diverge are precisely the places you can't enumerate in advance, because if you could enumerate them you'd have written a check instead.
So a program whose safety argument is "the model is accurate enough" is a program with no safety argument. The rate is a quality metric. It is not a control. The goal is not never be wrong; it's wrong doesn't automatically become executed.
That reframing is the whole post. The rest is where the gap goes.
Ground it, and make the grounding checkable
Grounding — retrieving authoritative content and requiring the answer to come from it — is the highest-leverage accuracy intervention available. But the version that helps is stricter than the version most teams ship.
Requiring citations only works if something verifies them. A model asked to cite will produce citations, including for claims the cited document doesn't make. The check has to be mechanical: does the quoted span actually appear in the retrieved source? Does the number in the answer appear in the number in the document? A string comparison catches a surprising share of confident nonsense, and it costs a millisecond.
The corollary is that the agent needs a way to say I don't have this. An agent that must always produce an answer will always produce one, and the cases where it has nothing to ground against are exactly the cases where it invents. Abstain has to be a first-class outcome that the eval rewards — otherwise every optimization pass trains the model toward confident guessing, because a guess sometimes scores and a refusal never does.
Verify with code wherever code can decide
For a large share of what agents produce in enterprise settings, correctness is mechanically checkable. That's the good news buried under the hallucination discourse.
| Output | Deterministic check |
|---|---|
| A number in a report | Recompute from the source system; compare |
| A generated SQL query | Parse it; validate tables and columns exist; dry-run it |
| Generated code | Compile it, type-check it, run the test suite |
| An API call | Validate against the schema; reject unknown fields and endpoints |
| A cited claim | String-match the quote against the retrieved document |
| A structured extraction | Schema validation, plus range and enum checks |
None of this asks a model whether a model was right. Each one is a program that returns true or false, and each one runs before anything downstream sees the output.
The distinction that matters here: a checker written in code cannot be talked into anything, which makes it a genuine control. A second model reviewing the first model's work is useful — worth doing where nothing deterministic exists — but it's a quality improvement, not a boundary, and it should be labeled as one in your architecture diagrams.
A test the agent wrote is not verification. If the agent generates both the code and the tests that pass it, you've verified self-consistency. Acceptance criteria must come from outside the loop that produced the work — written by a human, or fixed before the task started.
Acceptance tests, not eyeballing
For agents that produce work rather than answers — code, migrations, configs, documents — the useful pattern is the one software already uses: define what "done and correct" means as an executable check, before the agent starts.
The task specification carries its acceptance criteria. The agent runs them itself and iterates until they pass. What arrives at the human is work that has already cleared a bar, along with the evidence that it did. This turns review from "read carefully and hope you spot it" into "confirm the criteria were the right criteria," which is a much easier thing for a person to do well at 4 p.m.
Calibrate the escalation, not the confidence score
"Measure confidence" is standard advice, and the standard implementations are weak. Asking a model how confident it is produces a number that correlates with fluency more than accuracy. Token probabilities measure how sure the model is about the next token, which is not the same as whether the claim is true.
The signals that hold up are structural and cheap:
- No grounding retrieved. Nothing authoritative came back. Highest-value signal available, and it's free.
- Retrieved sources contradict. Two documents disagree; the model picked one silently.
- Citation check failed. The quoted span isn't in the source.
- Deterministic check failed. Compile error, schema violation, recomputed number doesn't match.
- Out of distribution for the task. Unusual entity, unusual amount, a request unlike anything in the eval set.
Any of these fires and the run escalates rather than proceeds. That's the useful form of "measure confidence": not a number in a log, but a specific condition wired to a specific behavior.
Price approval by the action, not by the anxiety
The last control is a human, and it's the one most often applied badly — either everywhere, or nowhere, or wherever someone felt nervous during the design review.
Put the gate where the action is irreversible or its blast radius is wide, and leave it off where the action is recoverable. Reading a dashboard, drafting a document, running a query: recoverable, no gate. Sending to a customer, moving money, deploying, deleting, writing to a system of record: gated, always.
The reason to be disciplined about this isn't efficiency. It's that approval fatigue is real and it destroys the control from the inside. A gate that fires on everything trains people to approve without reading, and then you have the audit trail, the timestamp, the sign-off, and none of the thinking. Every unnecessary approval you remove is buying attention for the ones that remain.
And what the human sees has to be the actual artifact — the recipient, the amount, the diff, the destination — not the agent's summary of what it's about to do. The summary is written by the component whose reliability is the thing in question.
What this costs
Every control here adds latency and reduces how much the agent can do unattended. Deterministic checks fail on outputs that were fine. Abstention means a real question sometimes goes unanswered. Approval gates mean someone has to be there. A system tuned to never execute a hallucination will also refuse things it shouldn't, and that cost lands on the people trying to use it.
There's no configuration that makes this free. What there is: a per-action decision about what happens when the output is wrong, made deliberately, by someone who can answer for it. The teams that do this well can name, for each action their agent can take, what the worst case is and who finds out. The teams that don't can name their model's accuracy on a benchmark.
The design assumption
The model will sometimes be wrong. Not as a temporary condition to be engineered away by the next release — as a standing property of the component you've built on.
Systems that survive that assume it in their architecture: outputs grounded in sources you control, claims checked by code that can't be persuaded, acceptance criteria written before the work, uncertainty escalating instead of proceeding, and human judgment spent where the action can't be undone.
Hallucination is the model's failure mode. Whether it becomes an incident is yours.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.