What RAG does, and what it does not
A language model by itself answers from what it learned in training, which does not include your policies, contracts or product manuals. Retrieval-augmented generation, usually shortened to RAG, adds a step. When someone asks a question, the system searches your documents for the passages most likely to hold the answer and gives those passages to the model, with an instruction to answer only from them and to show its sources.
This makes answers checkable, and it means the assistant knows about a document as soon as that document is indexed. It does not make the assistant right about everything. If the answer is not in the documents, or the wrong passage is retrieved, or two documents contradict each other, the result can still be wrong. Most of the checklist below exists to reduce those three failures.
Teams sometimes ask whether to train a model on their documents instead. For keeping answers current and traceable, retrieval is nearly always the better starting point. Our comparison of RAG and fine-tuning explains the difference.
Start with one group of users and their questions
An assistant for everyone, covering everything, is the hardest version to build and to judge. Pick one group, such as the support team, new starters or the sales desk, and collect the questions they really ask. Thirty to fifty genuine questions, with the correct answers and where those answers live, are worth more than any product demonstration.
- 01
Who will ask, and when in their work?
Why it mattersIt decides where the assistant should appear, whether inside a help desk, a chat tool or an intranet.
- 02
What do they ask today, and whom?
Why it mattersThose questions become your test set, and the people who answer them now become your reviewers.
- 03
What happens when the answer is wrong?
Why it mattersA wrong answer about a lunch menu and a wrong answer about a safety procedure need very different safeguards.
- 04
What should it refuse to answer?
Why it mattersLegal advice, medical advice and anything outside the documents should be declined, and it is better to list those cases at the start.
Prepare the content
The assistant can only be as good as what it reads. Before any technical work, decide which sources are in scope and who owns each one.
| Content problem | What it causes | What to do |
|---|---|---|
| Several versions of one document | Answers quoting a superseded policy | Keep one current version in scope and archive the others |
| Contradictions between documents | Different answers on different days | Name the authoritative source for each topic |
| Scanned pages and images of text | Passages that cannot be found | Convert to text, and check tables with care |
| Knowledge that is not written down | The assistant saying it does not know | Write the missing answers, starting with frequent questions |
| Documents with no owner | Content that goes stale unnoticed | Assign an owner or leave the document out |
A smaller, cleaner set of documents nearly always beats a larger, messier one. Start with the sources your experts already trust.
Respect who is allowed to see what
If the assistant can read a document, it can quote it to anyone who asks, unless the system checks each user's rights. For anything beyond public information, access control has to be part of retrieval itself: the search only returns passages the person asking is already allowed to open in the source system.
Confirm three things with whoever builds the system. Permissions are taken from the source systems, not copied once and forgotten. A change in someone's access takes effect promptly. And questions and answers are logged in a way your security and privacy policies allow. Where personal data is involved, your data protection adviser should review the design.
A worked example
A support team's product knowledge assistant
This example is invented to show the sequence. A software company wants its support agents to get answers from product documentation, release notes and resolved tickets.
- Test setSupport leads collect 60 real questions from the past quarter, each with the correct answer and its source. Twelve turn out to have no written source at all.
- ContentDocumentation and release notes go in. Resolved tickets are left out at first, because many contain customer details and outdated workarounds.
- First testThe assistant answers 38 of the 48 answerable questions correctly with the right source. Most failures trace to two overlapping setup guides that disagree.
- FixThe guides are merged, the twelve missing answers are written, and the test is rerun. The team repeats this cycle instead of adjusting the model.
The assistant is released to agents as a drafting aid: it proposes an answer with sources, and the agent sends it. The test set is rerun after every documentation release, so a drop in quality is seen before agents feel it.
Measure answer quality before and after launch
Impressions from a demo are not evidence. Use the test set to score the system the same way each time something changes: the documents, the retrieval settings or the model.
- 01
Check retrieval first
For each test question, did the system find the passage that contains the answer? If not, no model can answer correctly, and the fix lies in content or search.
- 02
Then check the answer
Is it correct, complete and supported by the passages shown? Have a person who knows the subject judge a sample, even if automated scoring is used for the rest.
- 03
Test the refusals
Ask questions the documents cannot answer. The right behaviour is to say so plainly, not to produce something plausible.
- 04
Keep watching after launch
Let users flag poor answers, review a sample regularly and add the failures to the test set.
Our AI evaluation and monitoring page describes how this is set up as an ongoing practice.
The implementation checklist
RAG implementation checklist
0 of 11 checkedWhat to do next
Collect the questions first. If you can gather forty real questions with their answers and sources, you already hold the most valuable asset of the project, and you will learn how ready your documents are along the way.
To see whether the task is a sound first use of AI at all, try our AI use case assessment. When you want to talk it through, our RAG development page explains how we build these systems, and you can send us your questions list.
Common questions
Do we need to clean up all our documents before starting?
No. Start with the sources your experts already rely on, and leave the doubtful ones out. Testing with real questions will then show precisely which documents cause wrong answers, so cleanup effort goes where it matters. Trying to tidy an entire document estate before starting usually delays the project without improving the first release.
Can the assistant leak confidential documents?
It can reveal anything it is allowed to retrieve, so the design must restrict retrieval to what each user may already see in the source systems. Ask how permissions are taken from those systems, how quickly changes apply and what is logged. For sensitive content, have your security team test it by asking as a user who should not have access.
How do we know the answers are correct?
By testing against questions whose answers you already know, and by having people who know the subject review a sample. Answers should always show their sources, which lets users check them. No system of this kind is correct every time, so decide in advance how much review each kind of answer needs before it reaches a customer.
Which documents work best?
Clear, current, well-structured text with headings: policies, manuals, help articles and procedures. Long tables, scanned pages, slides with little text and documents that contradict each other are harder. Short notes that assume context, such as chat messages, tend to produce weak answers unless the surrounding conversation is included.
Who should look after the assistant once it is live?
Three roles, which may be one person in a small team. Someone owns the content and keeps it current. Someone reviews flagged answers and maintains the test set. Someone technical watches performance and cost. Without those, quality drifts as documents change, and trust is lost faster than it was earned.
