Your RAG chatbot indexes internal docs, and some users start seeing data they shouldn't. What's the cause and fix?
Short answer
If retrieval pulls any indexed document regardless of who's asking, the model will faithfully surface data the user shouldn't see — this is an authorization flaw, not hallucination. Enforce the user's document-level permissions at retrieval time so only authorized chunks enter the context. A bigger system prompt is bypassable and doesn't implement access control, temperature is unrelated, and another model has the same gap.
A user asks the chatbot a question and gets back content from a document they were never supposed to access — a salary sheet, another team's roadmap, an HR case file. The instinct is to blame the model. It's the wrong instinct.
Why this is authorization, not hallucination
The model is doing exactly what it was told: it received that document in its context window and summarized it accurately. The failure happened before generation, at the retrieval step. If your retriever runs a similarity search across the entire index and returns the top-k chunks regardless of who is asking, then any sufficiently relevant secret document is fair game. The model can't enforce a permission it was never given.
This is classic broken access control wearing an AI costume. RAG quietly turns your whole document corpus into a queryable surface, and a UI that hides the link to a file does nothing if retrieval still pulls that file's text into the prompt.
The fix: authorize at retrieval
Enforce the requesting user's document-level permissions at query time, so only chunks they are entitled to ever enter the context. Practically:
- Attach ownership/ACL metadata to every chunk at ingestion time.
- Pass the authenticated user's identity and group memberships into the retriever and filter before the similarity search returns results, not after.
- Re-check permissions at answer time for anything sensitive, and re-index or tombstone documents when their ACLs change.
Why the distractors are dangerous
- Raise the temperature / hallucination: temperature changes randomness, not what data is retrievable. The data is real and correctly retrieved — that's the whole problem.
- Longer system prompt: a prompt asking the model to "be careful" is a soft, bypassable guardrail layered on top of already-leaked context. Prompt injection or a clever question defeats it. Access control must be a hard check in code.
- Switch models: every model surfaces whatever you put in its context. A different model inherits the identical retrieval gap.
What interviewers want to hear
That you locate the bug at the retrieval/authorization layer, enforce least-privilege per user, and never rely on the model's prose to substitute for an access-control decision the application should make.
Likely follow-ups
- How would you implement per-user filtering in a vector database that only does similarity search?
- Why is filtering the answer after generation worse than filtering at retrieval?
- How do you handle documents whose permissions change after they're already indexed?