RAG explained: making AI answer from your own documents
RAG stands for retrieval augmented generation and means the model retrieves relevant passages from your own documents before composing an answer, rather than answering from memory. It is the single most important technique for making an assistant usable in production, because every claim can then be traced to a document that actually exists.
RAG, fine-tuning or a longer prompt?
Three ways to make a model know your material, solving different problems. The choice follows how often the source changes and how much of it there is.
| Longer prompt | RAG | Fine-tuning | |
|---|---|---|---|
| Suits | A few pages | Thousands of documents | Tone and format |
| Updating | Edit the prompt | Reindex | Retrain |
| Citations | No | Yes | No |
| Cost per answer | High, everything is sent | Low, only the matches | Low |
| Changes facts | Yes | Yes | Unreliably |
| Remember | Start here if it fits | The default choice | Rarely what you need |
How does RAG work step by step?
Four steps, every time:
- The question is converted into a vector, a list of numbers representing its meaning.
- That vector is compared against your indexed content and the nearest passages are retrieved.
- Those passages go to the model along with the question and an instruction to answer only from them.
- The model composes the answer and cites which passages it was built from.
Why do the answers come out wrong?
When a RAG answer is wrong it is rarely because the model is weak. It is because the wrong passages were retrieved in step two. The model simply never received the evidence. So when answer quality is poor, changing model is the last adjustment, not the first. Look at what was retrieved before you change anything else.
How should the content be chunked?
A document is not indexed whole but in passages. A vector spanning twenty pages becomes so general it matches everything and therefore nothing. But if the split cuts through an argument, the passage loses its meaning when displayed. Practical rules I hold to:
- Follow the structure of the content. Split on headings, not on character counts.
- Keep the heading with each part, so the passage carries its own context.
- Let the parts overlap slightly, so a sentence on the boundary is not lost.
- Index tables separately. They become unintelligible when cut.
What should happen when the assistant does not know?
When nothing sufficiently relevant is retrieved, the answer should be that the information is not available, with a route through to a person. Without that rule the model fills the gap with something plausible, and that is where trust is lost. "I cannot find this" is a good answer. An invented one never is.
How are permissions handled?
In an internal tool the assistant must not surface a document the user is not allowed to read. The check therefore has to happen as passages are retrieved, so they are filtered out before the model sees them. Filtering the answer afterwards is not a control, it is a hope: the material has already passed through the model.
How do you know it got better?
Write down thirty real questions with known correct answers, taken from your own support history. Run them before and after every change. Without that list, every discussion about answer quality becomes a discussion about how it feels, and those discussions never end.
Common questions
Do we have to send our documents to OpenAI or Anthropic?
The retrieved passages go along with the call, yes, but not the whole document collection. Where that is unacceptable there are two routes: vendor agreements where data is not used for training and is processed inside the EU, or a model running in your own environment. The second costs more to operate and gives weaker answers, and that trade-off should be made deliberately rather than assumed.
How much content do you need before RAG is worth it?
Roughly at the point where the material no longer fits in a prompt, so from a few dozen pages upward. Below that a longer system prompt is both simpler and cheaper. Above a few thousand pages, retrieval quality decides everything, and that is where the work belongs.
How often does the index need updating?
When the content changes, and only for what changed. For a site or a document library, a nightly run that indexes whatever has a new timestamp is enough. The cost sits in creating vectors, so reindexing everything every night is waste without benefit.
Can the assistant still make things up?
The risk drops sharply but does not vanish. With retrieval from your documents, a citation in the answer and a rule to decline when the evidence is missing, the remaining errors usually sit in the phrasing rather than the facts. That is why the prompt, the retrieved passages and the answer are all logged, so deviations are found in review rather than by a customer.
More in the same area
Before you build an AI assistant: seven questions to answer
The questions that decide whether the project becomes useful or becomes a demo nobody opens twice.
How to measure whether AI search actually got better
A method for evaluating search quality with numbers instead of opinion, and why hybrid search almost always wins.
