Knowledge Poisoning: A Framework for What Your Agent Is Allowed to Believe
An agent can give a perfectly correct answer built on completely wrong company knowledge. The reasoning holds up. The citation is real. The document exists. And the number in it has been wrong since March.
This is the failure enterprises underestimate, because it does not look like a failure. No error. No refusal. No hedge. Retrieval worked, generation worked, and the system did exactly what you built it to do, on top of a corpus nobody owns.
The path there is short. You connect the agent to Drive, Slack, Notion, and Confluence, because that is the whole pitch: one assistant, all your knowledge. Then somebody uploads a superseded pricing sheet. Or an old deck gets re-shared in a channel. Or a contractor writes a runbook that was never true in the first place.
The agent retrieves it, quotes it confidently, and that content has effectively joined your company's answer surface.
When it happens by accident, call it knowledge contamination. When somebody does it on purpose, writing plausible false claims into a document they know the agent retrieves often, call it knowledge poisoning. The second one is a remarkably cheap attack. No exploit, no credentials beyond ordinary write access to a wiki page. The payload looks like documentation, because it is documentation.
The rest of this post is a framework you can apply to a corpus you already have. Seven parts, each answering one question about a piece of knowledge before your agent is allowed to act on it. But the framework only makes sense once you can see where the current pipeline checks nothing, so we start there.
First, what retrieval actually does
Five steps, and the important thing is what is absent from them.
Ingest. A connector pulls documents from Drive, Slack, Notion, and the rest.
Chunk. Each document is split into pieces of a few hundred tokens, because whole documents do not fit in a prompt and smaller pieces match more precisely.
Embed. Each chunk goes through an embedding model, which turns text into a vector of numbers. Chunks about similar topics land near each other in that space.
Retrieve. At query time the question gets embedded the same way, and the store returns the k nearest chunks, where k is usually between 5 and 20.
Generate. Those chunks get pasted into the prompt above the question, with an instruction like "answer using the context below."
Now look at that pipeline and ask where truth is checked. Nowhere. Not once.
The embedding model measures whether two pieces of text are about the same thing. It has no opinion about whether either one is correct, current, authorized, or written by somebody who knew what they were talking about. Retrieval is a similarity operation, and similarity is not truth.
That is why this failure is so quiet. Every component did its job.
Why "better RAG" will not save you
The instinct is to improve retrieval. Better embeddings, a reranker, hybrid search, bigger chunks, smaller chunks. All of that improves your odds of retrieving the most relevant document.
Relevance is not the property that is failing. The stale pricing sheet is extremely relevant to a pricing question. It is the best match in your corpus by any similarity measure you care to run, and a better retriever finds it faster.
Rank quality and truth are orthogonal. Tuning the first will never touch the second. What is missing is everything you would expect around any other production data source: where did this come from, who is responsible for it, when was it last true, and what breaks downstream when it is wrong.
That list is the framework.
The framework in one picture
Seven parts. Read them as seven questions asked about a chunk before it is allowed to influence an answer.
Parts one through five run before an answer exists. Parts six and seven tell you whether the first five are working, one in aggregate and one per incident.
| Part | Question it answers | Enforced at | What breaks without it |
|---|---|---|---|
| Provenance | Where did this come from? | Ingest | No other control can be applied |
| Permissions | May this person see it? | Query, as a pre-filter | The index launders access |
| Freshness | Is it still true? | Ingest and query | Confident answers from last year |
| Trust level | What may it be used for? | Query | A chat message sets your pricing |
| Deduplication | One voice or six echoes? | Ingest | The most copied version wins |
| Retrieval evaluation | Did the right source win? | CI | Regressions ship silently |
| Attribution | Which chunk caused this? | Runtime logging | Incidents become guesswork |
Part 1: Provenance
Every chunk carries its origin, and not for display purposes. For decisions. You cannot apply any policy to a chunk whose source you do not know at query time.
{
"chunk_id": "c_8f21",
"text": "Growth plan is $99 per seat per month...",
"source_system": "confluence",
"document_id": "PRICING-2025",
"document_url": "https://wiki.example/PRICING-2025",
"last_modified": "2025-03-11T00:00:00Z",
"ingested_at": "2026-08-01T02:14:00Z",
"tier": "reviewed",
"expires_at": "2026-09-11T00:00:00Z",
"acl": ["group:sales", "group:finance"]
}
Build this first. Every part that follows keys off one of those fields, and backfilling provenance under incident pressure is miserable work.
Part 2: Permissions
Retrieval has to resolve against the asking person's real access in the system of record, not a snapshot taken at ingest time. Otherwise your knowledge base quietly becomes the thing that launders access: the compensation doc nobody could open is now summarizable by anybody who asks the right question.
There is an implementation trap here worth spelling out, because it is the most common way this control gets built wrong.
The tempting approach is to retrieve the top ten chunks and then filter out the ones the user cannot see. That fails in two ways. If eight of the ten are restricted, the user gets an answer built on two chunks with no signal that anything was dropped. And the ranking itself leaks: a thin answer for one person and a rich answer for another is a disclosure about what exists.
Filter before the nearest neighbor search, by passing group memberships into the query, so the search runs only over chunks that person can see.
# Wrong: post-filter, k is k over the whole corpus
hits = store.search(query_vec, k=10)
visible = [h for h in hits if user_can_read(user, h)]
# Right: pre-filter, k is k over the visible corpus
hits = store.search(query_vec, k=10, filter={"acl": {"$in": user.groups}})
Permissions also change after ingest. Somebody leaves a team, a document gets restricted. Re-sync access control lists on a schedule, and for sensitive sources, re-check against the live system at query time.
Part 3: Freshness
Every document gets an age, and where it matters, an explicit expiry. Content past its expiry either does not get retrieved, or comes back carrying its age loudly enough that the model says so in the answer.
| Content type | Suggested expiry | Why |
|---|---|---|
| Pricing, discounts | 30 days | Changes quietly and gets quoted externally |
| Policies, legal | 90 days with owner review | Wrong answers create obligations |
| On-call, org charts | 14 days | People move constantly |
| Runbooks | 180 days | Slower drift, high blast radius when wrong |
| Meeting notes | Never expires, never authoritative | Useful context, poor evidence |
| Architecture docs | 365 days | Slow to change, worth a yearly review |
Age is a proxy, not truth. A document written last week can be wrong and one from two years ago can be perfectly current. Treat it as the cheapest available signal, not as a correctness check.
Part 4: Trust levels
Not every source deserves equal standing, and the ordering is usually obvious once somebody writes it down.
The rule that earns the diagram: a claim's trust level bounds what it can be used for. A Slack message can tell the agent where to look. It should not be the source a customer-facing number gets quoted from.
Enforce the level in the retrieval layer, not in the prompt. When a question is classified as pricing or policy, restrict retrieval to authoritative sources, and when nothing authoritative comes back, say that instead of falling through to whatever ranked next.
Part 5: Deduplication
Six copies of the same deck across four systems is not six pieces of evidence, but a retriever counting matches treats it that way. The outdated version usually wins on volume, because it has been around longer and has been copied more.
Detect near duplicates with a cheap similarity pass, minhash on shingles or cosine similarity above a threshold on the embeddings. Group them, then pick a canonical copy: prefer the most authoritative system, then the most recently modified. Keep the rest out of retrieval but keep the mapping, because "this exists in six places" is a signal worth showing whoever owns the content.
Deletion in the source is not deletion in the index. The doc gets deleted, the page gets archived, the channel gets cleaned up, and the embedding sits there answering questions for another year. Retractions have to propagate through the same pipeline as ingest. The fastest way to find out whether yours do is to delete something and then ask about it.
Part 6: Retrieval evaluation
Most teams grade the answer. The answer is a composite of retrieval and generation, so grading it alone tells you something is wrong without telling you which half.
Give retrieval its own suite. A test case is a question plus the documents that should and should not come back:
- question: "What is the Growth plan price?"
must_retrieve: ["PRICING-CURRENT#growth"]
must_not_retrieve: ["PRICING-2025#growth", "deck-q1-copy#pricing"]
required_tier: authoritative
- question: "What is our refund window?"
must_retrieve: ["POLICY-REFUND"]
must_not_retrieve: ["slack:#support-chatter"]
required_tier: authoritative
Four numbers worth tracking over time:
- Recall at k. Did the correct document come back at all?
- Poisoned document win rate. Plant a plausible false document in a staging corpus and measure how often it beats the authoritative source. This is your poisoning test, and it takes an afternoon to build.
- Contradiction rate. How often do the top results disagree? Two documents asserting different prices is a corpus defect. The model should not quietly resolve it by picking one. Surface it in the answer and route it to the owner.
- Stale hit rate. What share of retrieved chunks are past expiry?
Run the suite on every ingest change, every chunking change, and every embedding model change. Swapping the embedding model reshuffles the entire ranking, so treat it like a database migration rather than a config tweak.
Part 7: Attribution
The part that makes the other six usable in practice: being able to say precisely which piece of information drove a given decision.
Not "here are some citations." Post hoc citations can be assembled after the fact, and a model asked to cite will find something plausible to point at. What you want is the retrieval record itself, logged and joined to the response.
Here is the incident playbook that trace enables. Somebody reports a wrong answer. You look up the response id, read the chunks that were actually in context, and identify the source document. You check whether it is stale, duplicated, mis-tiered, or false. You fix it at the source, purge the chunks, then query the trace log for every other response that used the same document. Those are the people you contact.
Without the trace, every step in that paragraph is guesswork. It is also the part that is hardest to add later: provenance and trust levels can be backfilled with effort, but retrieval traces you never wrote simply do not exist.
Applying the framework to a corpus you already have
You do not need to build all seven at once. Run this sequence instead.
Week one, measure. Pick the ten questions your agent answers most. For each, write down which document should be the source. Then run them and record what actually came back. The gap between those two lists is your real backlog, and it is usually shorter and more boring than people expect.
Week two, provenance and permissions. These two unblock everything else, and permissions is the one with genuine legal exposure. Fix the post-filter bug if you have it.
Week three, freshness and duplicates. Both are mechanical, and between them they resolve most staleness incidents.
Week four, trust levels. This one needs humans to make decisions, so start the conversation early even though the code lands late.
Ongoing, evaluation and attribution. The eval suite pays for itself the first time an embedding upgrade quietly reshuffles your rankings. The trace log pays for itself the first time somebody asks how far a bad document spread.
What the framework does not cover
Trust levels need a human to assign them, and assignment drifts. A source classified as authoritative in January is a wiki space with three owners and no review process by June. Some of it can be derived from the source system's own permissions and workflow state, and some cannot. Somebody has to own the quarterly review.
Freshness is a proxy, as noted. Age correlates with wrongness. It does not measure it.
And none of this addresses a poisoned document written by somebody with legitimate authoring rights to an authoritative source. That is an insider problem, and the controls are the ordinary ones: review on change, named ownership, audit trails in the source system. Your agent inherits the integrity of your corpus. It cannot exceed it.
Connecting an agent to all your enterprise knowledge is a weekend of OAuth flows. Every vendor demos it, and it works. Deciding what that knowledge is allowed to make your agent believe is the real project, and it is the difference between an agent that sounds authoritative and one that is.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.