Knowledge Poisoning: A Correct Answer From Wrong Knowledge
An agent can give a completely correct answer based on completely wrong company knowledge. The reasoning is sound, the citation is real, the document exists — and the number in it has been wrong since March.
This is the failure enterprises underestimate, because it doesn't look like a failure. There's no error, no refusal, no hedge. Retrieval worked. Generation worked. The system did exactly what it was built to do, on top of a corpus nobody was accountable for.
The path there is short. You connect the agent to Drive, Slack, Notion, and Confluence, because that's the pitch — one assistant, all your knowledge. Then somebody uploads a superseded pricing sheet, or an old deck gets re-shared in a channel, or a contractor writes a runbook that was never true. The agent retrieves it, quotes it confidently, and that content has effectively entered your company's answer surface.
When it's accidental, that's knowledge contamination. When somebody does it on purpose — writing plausible false instructions into a document they know the agent retrieves often — it's knowledge poisoning, and it's a genuinely cheap attack. No exploit, no credentials beyond ordinary write access to a wiki page, and the payload looks like documentation.
Why "better RAG" isn't the answer
The instinct is to improve retrieval: better embeddings, a reranker, hybrid search, bigger chunks, smaller chunks. All of that improves your odds of retrieving the most relevant document.
Relevance is not the failing property. The stale pricing sheet is extremely relevant to a pricing question. It's the best match in the corpus by any similarity measure you care to run, and a better retriever finds it faster. Rank quality and truth are orthogonal, and tuning the first will never touch the second.
What's missing isn't retrieval quality. It's everything you'd expect around any other production data source: where did this come from, who's responsible for it, when was it last true, and what happens downstream when it's wrong.
The properties that actually decide trust
Provenance. Every chunk carries its origin — system, document, author, URL, and the moment it was ingested. Not for display. For decisions: you cannot apply a policy to a chunk whose source you don't know at query time.
Permissions that mirror the source. Retrieval must resolve against the asking user's actual access in the system of record, not against a snapshot taken at ingest. Otherwise your knowledge base quietly becomes the thing that launders access — the compensation doc nobody could open is now summarizable by anyone who asks the right question. This one is usually discovered late and is usually expensive to retrofit, because the index was built without the identity plumbing to enforce it.
Freshness. Every document gets an age and, where it matters, an explicit expiry. Pricing, policy, on-call rotations, and org structure have half-lives measured in weeks. Content past its expiry either doesn't get retrieved or gets retrieved carrying its age loudly enough that the model surfaces it: this is from fourteen months ago.
Trust tiers. Not every source deserves equal standing, and the ordering is usually obvious once someone writes it down:
| Tier | Examples | Retrieval treatment |
|---|---|---|
| Authoritative | Systems of record, published policy, approved docs | Preferred; can answer alone |
| Reviewed | Team wikis with named owners | Usable; conflicts lose to authoritative |
| Ambient | Chat threads, meeting notes, drafts | Context only; never sole basis for a factual claim |
| Unverified | Anything user-uploaded or externally sourced | Quarantined until promoted |
The rule that earns the table: a claim's tier bounds what it can be used for. A Slack message can tell the agent where to look. It should not be the source a customer-facing number is quoted from.
Deduplication. Six copies of the same deck across four systems is not six pieces of evidence, but a retriever counting matches will treat it that way. The outdated version usually wins on volume — it's been around longer and has been copied more. Collapse near-duplicates and keep the canonical one, and the fact that a document exists in six places becomes a signal about the corpus rather than an argument to the model.
Deletion in the source is not deletion in the index. The doc gets deleted, the page gets archived, the channel gets cleaned up — and the embedding sits there answering questions for another year. Retractions have to propagate, and the fastest way to find out whether yours do is to delete something and ask the agent about it.
Evaluate retrieval on its own
Most teams evaluate the answer. The answer is a composite of retrieval and generation, and grading it alone means you find out something is wrong without learning which half.
Retrieval deserves its own suite: a fixed set of questions with known-correct source documents, scored on whether the right documents came back and the wrong ones didn't. Run it on every ingest change, every chunking change, every embedding model change. Track the poisoning-shaped cases explicitly — plant a plausible false document in a staging corpus and check whether the authoritative source still wins.
The metric worth watching most is contradiction: how often the top results disagree with each other. Two retrieved documents asserting different prices is not something the model should silently resolve by picking one. It's a corpus defect, and it should surface as one — to the user in the answer, and to whoever owns that content.
Answer the attribution question
The capability that turns all of this from principle into practice is being able to say, precisely: which piece of information caused this decision?
Not "here are some citations" — post-hoc citations can be assembled after the fact, and a model asked to cite will find something plausible to point at. What you want is the retrieval record itself: the chunks that were actually in context for that generation, their sources, their tiers, their ages, and the query that pulled them. Logged, joined to the response, queryable later.
That record is what makes every other control usable. When a wrong answer surfaces, you find the document instead of guessing at the model. When a document turns out to be poisoned, you find every answer it touched instead of issuing a general apology. When someone asks whether the agent has ever quoted the old pricing, that stops being a question about the model's behavior and becomes a query.
It's also the one part that's hard to add later. Provenance and tiers can be backfilled with effort. Retrieval traces you didn't write don't exist.
What we don't have
Trust tiers require a human to assign them, and assignment drifts — a source classified as authoritative in January is a wiki space with three owners and no review process by June. Some of this can be derived from the system of record's own permissions and workflow state; not all of it. It needs a periodic review that someone actually owns.
Freshness is a proxy, not truth. A document written last week can be wrong, and one from two years ago can be perfectly current. Age is the cheapest available signal, and it should not be mistaken for correctness.
And none of this addresses a poisoned document written by someone with legitimate authoring rights to an authoritative source. That's an insider problem, and the controls are the ordinary ones — review on change, ownership, audit trails on the source system. The agent inherits the integrity of the corpus. It cannot exceed it.
The hard half
Connecting an agent to all of your enterprise knowledge is a weekend of OAuth flows. Every vendor demos it, and it works.
Knowing which of that knowledge deserves to be trusted is the actual project. It's unglamorous — provenance plumbing, permission mirroring, expiry policies, dedup, a retrieval eval nobody asks for in the demo — and it's the difference between an agent that sounds authoritative and one that is.
Your agent will answer confidently either way. That's the part it's good at.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.