Also published on Medium on 2026-10-13.
Retrieval-augmented generation has become the default way to build useful AI assistants. Instead of relying only on what a model learned during training, you retrieve relevant documents from your own data and include them in the prompt. Answers get more accurate, more current and more specific to your business.
There’s a security consequence that most teams haven’t fully absorbed: every document in your retrieval corpus is now an input to your model. And most teams don’t treat it like one.
How a poisoned knowledge base works
Here’s the general shape of the problem.
You build an assistant that answers questions using your company’s documents: help articles, internal wikis, support tickets, product specs, maybe some crawled web pages.
Someone manages to get content into that corpus. Depending on your setup, it might come through a support ticket, a shared folder, a customer-uploaded file, a public page you crawl or a wiki anyone in the company can edit.
That content contains text written to steer the model rather than inform the reader, for example instructions that change how the assistant answers questions on a particular topic, or that push users toward a particular action.
Later, a normal user asks a normal question. Your retrieval system finds the planted content because it’s relevant to the question. The model receives it alongside your trusted documents and treats it as context. The answer the user gets is shaped by someone who never touched your application directly, and it arrives with your brand on it.
Why retrieval doesn’t protect you
Retrieval systems are good at one thing: finding text that’s semantically related to a query. They’re not designed to judge whether that text is trustworthy, whether it’s trying to manipulate the model or whether its author had any business being in your corpus.
Relevance and trustworthiness are separate properties. Most RAG pipelines only measure the first.
Defence 1: treat retrieved content as data, not instructions
Start with how you construct prompts. Retrieved content should be clearly separated and labelled as reference material, and your system prompt should tell the model not to follow instructions found inside it.
This doesn’t solve the problem on its own. As with all prompt-level defences, models don’t follow these conventions perfectly. But it raises the bar and costs almost nothing.
Defence 2: scan at ingestion
The cheapest place to catch bad content is when it enters your corpus. When documents are added or updated, inspect them for content that looks like it’s addressing a model rather than a human reader, for hidden or invisible text, and for unusual formatting tricks.
Flag suspicious documents for review instead of silently indexing them. It’s much easier to stop one bad document at the door than to catch it every time it’s retrieved.
Defence 3: inspect the final prompt
Ingestion scanning won’t catch everything, especially with content that changes or comes from live sources. So also inspect the assembled prompt, including all retrieved chunks, right before it goes to the model.
This is a key part of how we designed Koreshield’s gateway. We inspect the whole request, including RAG context, not just the user’s latest message, because that’s where indirect attacks hide.
Defence 4: track provenance
For every chunk you retrieve, you should know which document it came from, who added it, when and through which channel.
Provenance lets you:
- Weight or restrict sources by trust level
- Trace a bad answer back to the document that caused it
- Remove poisoned content quickly and confidently
- Show auditors where your model’s answers come from
Defence 5: tier your sources
Not all content deserves equal trust. A policy document maintained by your legal team is different from a ticket submitted by an anonymous user.
Consider separating sources into trust tiers, and being more careful with low-trust content: stricter inspection, lower retrieval priority, or keeping it out of contexts where the model can take actions.
Defence 6: limit what the model can do with what it reads
Finally, remember that the impact of a poisoned document depends on what the model can do. If your assistant only answers questions, the worst case is a misleading answer. If it can call tools, the worst case is much worse.
Keep tool permissions narrow, especially in flows that include low-trust retrieved content, and require confirmation for consequential actions.
The mindset
The mental shift is simple to state and hard to fully adopt: your knowledge base isn’t a library your model consults. It’s an input channel, and anyone who can write to it can influence your model.
Secure it the way you’d secure any other input: validate what comes in, know where it came from, inspect it before use and limit the damage if something gets through.