Why it shows up in your week
Someone asks the bot for the return policy. Without RAG it invents a confident paragraph. With RAG it pulls the current PDF chunk, answers from that text, and can show the source. That is the difference between a demo and something support will touch.
In plain language
A query embeds or searches your corpus. Top chunks are attached to the prompt. The model should answer from that context.
RAG is not magic memory. Stale indexes, bad chunking, and missing access controls produce confident wrong answers.
Permissions must be enforced at retrieval time. Otherwise one user can retrieve another customer's file.
Citations or source links let a human verify. Hide the source and you hide the failure mode.
Start with a narrow corpus (one product line, one policy set). Expand after evals pass.
Facts worth keeping
- Pattern
- Retrieve → stuff context → generate
- Typical stores
- Vector index, keyword/hybrid search, document stores
- Fails when
- Stale corpus, bad chunking, missing ACL filters
- Pairs with
- Human-in-the-loop, guardrails, eval sets
- Not the same as
- Fine-tuning model weights on your docs
Not the same as
- Fine-tuningFine-tuning changes model weights. RAG keeps the model fixed and swaps in fresh retrieved context, usually better for frequently changing company knowledge.
- A chatbot with a long system promptPrompts help behaviour. They do not replace retrieval of current documents.
- Training the model on the public webRAG is about your corpus at query time, not general pretraining.
Where it bites
RAG hurts when a demo works on five PDFs and then fails on the real SharePoint mess, or when retrieval returns the wrong customer's data because permissions were ignored.
Trust collapses in one bad answer, and the team disables the bot instead of fixing retrieval.
What to check
- What corpus is indexed, and how often does it refresh?
- Are document permissions enforced at retrieval time?
- Do answers cite or link the retrieved source for human review?
- What happens when retrieval returns nothing useful?
- Which eval questions prove the bot is safe to widen?
Common questions
What is RAG?
Retrieval-augmented generation retrieves relevant documents at query time and provides them as context so a model can generate grounded answers.
How is RAG different from fine-tuning?
Fine-tuning changes model weights. RAG keeps the model fixed and swaps in fresh retrieved context, which is usually better for frequently changing company knowledge.
Do we still need guardrails with RAG?
Yes. Retrieval can return the wrong doc. Guardrails and human review still matter for high-impact answers.
What should you check first for a RAG system?
Corpus quality, refresh cadence, permission filters, chunking strategy, and whether answers expose sources for human verification.
Is RAG enough for EU AI Act concerns?
RAG helps factual grounding. Legal duties may still require transparency, oversight, and risk classification depending on use case.