Hallucination Isn't the Problem. Execution Is.
"AI hallucinates" is a true sentence that has been repeated until it stopped meaning anything. It sounds like a quirk. The chatbot got a trivia question wrong, everyone screenshots it, nothing happens.
Now move the same behavior inside a company.
A finance agent invents a figure and it lands in a board deck. A support agent invents a refund policy and tells a customer, which in several jurisdictions means you now have that policy. A coding agent invents an API, writes code around it, and you find out in production at 2 a.m.
Same underlying behavior in all four cases. Wildly different consequences. And the difference has nothing to do with the model.
What separates them is how much authority sat downstream of the output. In the trivia case a human read it and shrugged. In the other three, something took the output and acted on it, with nothing in between asking whether it was true.
That gap is what you control. The hallucination rate mostly is not.
This post is the production checklist version. First a short section on why models fabricate and what actually goes wrong, because a checklist you do not understand becomes a ritual. Then the checklist itself, organized by stage, with the reasoning behind each item.
Why models fabricate at all
A language model is trained to continue text plausibly. Given everything so far, it produces a likely next token, then another. That objective correlates with truth most of the time, because true text is usually the most plausible continuation, especially for things well represented in training data.
It stops correlating in three situations worth knowing by name.
The answer is not in the model. Your internal API, last quarter's numbers, a policy you wrote in June. The model has no representation of these, but it still has an objective that rewards a fluent continuation. So it produces the shape of a correct answer with invented contents. This is why fabricated content looks so plausible: plausibility is literally what got optimized.
The prompt implies an answer exists. "What is our refund window for enterprise customers?" presupposes there is one. Models are strongly biased toward satisfying the presupposition rather than challenging it.
Nothing rewards abstaining. In most evaluation setups, a guess sometimes scores and a refusal never does. If your evals work that way, every optimization pass pushes the model toward confident guessing. You get the behavior you measured.
Notice that none of the three is a bug you can patch. They are properties of how the thing works, which is why the accuracy number never reaches zero, and why a safety argument that rests on "the model is accurate enough" is not a safety argument at all.
Accuracy is a quality metric. It is not a control.
The reframe
The goal is not never be wrong. The goal is wrong does not automatically become executed.
Everything below is about what goes in that gap.
What actually goes wrong
"Hallucination" is one word covering at least five distinct failures, and they need different checks. Being precise here saves you from building one control and believing you covered all of them.
| Failure | Example | The check that catches it |
|---|---|---|
| Fabricated fact | A revenue number that exists nowhere | Recompute from the source system |
| Unfaithful summary | Source says "declined slightly", summary says "collapsed" | Quote span matching against the source |
| Misattributed citation | Real document cited for a claim it never makes | Verify the quote appears in that document |
| Invented interface | An API endpoint or column that does not exist | Schema validation, compile, dry run |
| Wrong unit or scale | 4.2 million reported as 4.2 thousand | Range checks and unit assertions |
The one teams miss most often is the unfaithful summary, because the citation is real, the document is real, and the sentence sounds careful. Only a comparison against the source text catches it.
The production checklist
Six stages. Each item says what to do and why it is there. Work top to bottom; the early stages are cheaper and catch more.
Stage 1: Ground the output
Retrieve before you generate. Answers to questions about your business come from your systems, not from the model's memory. If nothing was retrieved, that is not a reason to answer anyway.
Make citations verifiable, not decorative. Requiring citations only helps if something checks them. A model asked to cite will produce citations, including for claims the cited document never makes.
def citation_holds(answer_span: str, source_text: str) -> bool:
# Exact or near-exact containment. Cheap, and it catches
# the confident summary that drifted from its source.
return normalize(answer_span) in normalize(source_text)
Give the agent a way to say "I do not have this." Abstention must be a first-class outcome that your eval rewards. Without it, the cases where the agent has nothing to ground against are exactly the cases where it invents.
Show the grounding to the reader. Not a footnote nobody opens. The claim and its source next to each other, so a person can check in two seconds.
Stage 2: Verify with code wherever code can decide
Here is the good news buried under all the hallucination discourse. For a large share of what agents produce in enterprise settings, correctness is mechanically checkable.
| Output | Deterministic check |
|---|---|
| A number in a report | Recompute from the source system and compare |
| A generated SQL query | Parse it, validate tables and columns, dry run it |
| Generated code | Compile it, type check it, run the tests |
| An API call | Validate against the schema, reject unknown fields |
| A cited claim | String match the quote against the retrieved document |
| A structured extraction | Schema validation plus range and enum checks |
| A date or amount | Assert bounds and units before anything consumes it |
None of these ask a model whether a model was right. Each is a program that returns true or false, and each runs before anything downstream sees the output.
A second model reviewing the first model's work is useful where nothing deterministic exists, but label it correctly in your architecture. It is a quality improvement, not a boundary, because it can be talked into things by the same content that fooled the first one.
Stage 3: Define acceptance before the work starts
For agents that produce work rather than answers (code, migrations, configs, documents), use the pattern software already knows. Write down what "done and correct" means as an executable check, before the agent starts.
task: add pagination to the accounts endpoint
acceptance:
- command: pytest tests/api/test_accounts.py
expect: exit_code == 0
- command: openapi-diff baseline.yaml current.yaml
expect: no_breaking_changes
- assertion: response_time_p95_ms < 250
The agent runs these itself and iterates until they pass. What reaches the human is work that already cleared a bar, plus the evidence that it did. Review turns from "read carefully and hope you spot it" into "confirm these were the right criteria," which is a much easier job at 4 p.m.
A test the agent wrote is not verification. If the agent generates both the code and the tests that pass it, you have verified self-consistency. Acceptance criteria have to come from outside the loop that produced the work: written by a person, or fixed before the task began.
Stage 4: Measure the signals that actually predict trouble
"Measure confidence" is standard advice and the standard implementations are weak. Ask a model how confident it is and you get a number that tracks fluency better than accuracy. Token probabilities describe certainty about the next token, which is not a claim about the world.
The signals that hold up are structural, and they are cheap to compute.
That is the useful form of measuring confidence. Not a number in a log, but a specific condition wired to a specific behavior.
Stage 5: Put humans where the action has no undo
Price approval by the action, not by how nervous somebody felt in the design review. Two dimensions decide it: can this be undone, and how far does it reach.
| Narrow reach | Wide reach | |
|---|---|---|
| Reversible | No gate. Log it. | Notify after the fact, easy rollback |
| Irreversible | Approve, show the arguments | Approve, plus a second reviewer |
Reading a dashboard, drafting a document, running a query: recoverable, no gate. Sending to a customer, moving money, deploying, deleting, writing to a system of record: gated, always.
Be disciplined about this for a reason that is not efficiency. Approval fatigue is real and it destroys the control from the inside. A gate that fires on everything trains people to approve without reading, and then you have the audit trail, the timestamp, the sign-off, and none of the thinking. Every unnecessary approval you remove buys attention for the ones that remain.
And what the human sees has to be the actual artifact: the recipient, the amount, the diff, the destination. Not the agent's summary of what it is about to do. That summary is written by the component whose reliability is the question on the table.
Stage 6: Observe and keep score
Log enough to reconstruct a decision. For every action: the prompt version, the retrieved chunk ids, the checks that ran and their results, the escalation signals, and who approved. When something goes wrong, you want to answer "why did it do that" from data rather than from a re-run that may not reproduce.
Track four numbers over time. Groundedness (share of claims with a verified citation), check failure rate, abstention rate, and approval override rate. Abstention deserves attention in both directions: near zero means the agent is guessing, and very high means grounding or retrieval is broken.
Gate releases on a golden set. A fixed set of tasks with known-correct outputs, run in CI on every prompt, tool, retrieval, or model change. Model swaps especially. A same-family point upgrade can move faithfulness in either direction with nothing in the changelog to warn you.
The checklist, condensed
Copy this into your design doc and answer each line for one agent.
Ground
- Answers about the business come from retrieval, never model memory
- Every quoted claim is verified against the source text by code
- The agent can abstain, and evals reward it for abstaining correctly
- Sources are shown next to claims, not buried in a footnote
Verify
- Every mechanically checkable output has a check that runs before use
- Numbers are recomputed from the system of record
- Generated code compiles, type checks, and runs tests
- Structured output passes schema, range, and unit assertions
- Model-as-reviewer is labeled a quality step, not a boundary
Accept
- Every work-producing task carries executable acceptance criteria
- Criteria are authored outside the loop that does the work
- The human reviews evidence, not just the artifact
Escalate
- No grounding retrieved triggers escalation, not an answer
- Contradicting sources surface rather than being silently resolved
- Failed checks stop the run
- Out-of-distribution inputs route to a human
Approve
- Gates are placed by reversibility and blast radius, nothing else
- Approval screens render real arguments, never model prose
- Recoverable actions are ungated on purpose, to protect attention
Observe
- Every action logs prompt version, sources, checks, and approver
- Groundedness, check failures, abstention, and overrides are tracked
- A golden set gates releases, and reruns on every model change
What this costs
Every control here adds latency and shrinks how much the agent can do unattended. Deterministic checks will fail on outputs that were fine. Abstention means a real question sometimes goes unanswered. Approval gates mean somebody has to be there. A system tuned never to execute a hallucination will also refuse things it should not, and that cost lands on the people trying to use it.
There is no configuration that makes this free. What there is: a per-action decision about what happens when the output is wrong, made deliberately, by somebody who can answer for it.
Teams that do this well can name, for every action their agent can take, what the worst case is and who finds out. Teams that do not can name their model's accuracy on a benchmark.
The design assumption
The model will sometimes be wrong. Not as a temporary condition that the next release engineers away, but as a standing property of the component you built on.
Systems that survive assume it in their architecture. Outputs grounded in sources you control. Claims checked by code that cannot be persuaded. Acceptance criteria written before the work. Uncertainty that escalates instead of proceeding. Human judgment spent where the action cannot be undone.
Hallucination is the model's failure mode. Whether it becomes an incident is yours.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.