What each one is
Retrieval-augmented generation, or RAG, leaves the model as it is. When a question arrives, the system searches your documents for the relevant passages and gives them to the model along with the question. The model answers from what it was handed, and can point to where each statement came from.
Fine-tuning changes the model itself. You train it further on a set of examples, so that it picks up a pattern, such as how your team writes, how to label a type of document or how to follow a specific output format. The knowledge in those examples is absorbed loosely. The model does not look anything up when it answers.
The two approaches at a glance
| RAG | Fine-tuning | |
|---|---|---|
| What it changes | What the model sees when it answers | How the model behaves |
| Best for | Facts, policies, documents and records | Style, format, tone and narrow classification |
| When information changes | Update the documents, and the next answer reflects it | Retrain, or the model stays as it was |
| Sources | Each answer can cite the passages it used | No sources. The model cannot say where an answer came from |
| Access control | Search can respect who may see which document | Anything trained in is available to every user |
| What you need to start | Your documents, cleaned and indexed | Hundreds to thousands of good examples |
| Main effort | Preparing content and tuning search | Building the example set and evaluating the result |
When each one fits
Choose RAG when
- Answers must come from your documents, not from the model's general knowledge
- Content changes weekly or daily
- Users need to see the source to trust the answer
- Different people may see different documents
- You want a result soon, with the data you already have
Choose fine-tuning when
- Output must follow a strict format every time
- The task is narrow and repeated in volume, such as sorting or labeling
- A house style is hard to describe and easy to show
- Instructions and examples in the prompt have been tried and fall short
- You have many high-quality examples and a way to measure improvement
Before either, try the simplest thing. A clear prompt with a few examples, on a capable model, solves many cases. Add retrieval when the model lacks your facts. Consider fine-tuning when behavior is still inconsistent.
Why company knowledge usually starts with retrieval
Company knowledge has three properties that fine-tuning handles poorly.
- It changes. Prices, policies and procedures are revised, and an answer based on last quarter's version is worse than no answer
- It has owners and permissions. A contract or an HR policy is not for everyone. Retrieval can filter by what the person asking is allowed to read
- It must be verifiable. In legal, financial or safety matters a person needs to check the source before acting. Retrieval can show it
Retrieval does have its own failure. If the search step brings back the wrong passages, the model answers confidently from the wrong material. Much of the quality in a RAG system is decided by how documents are prepared, split and searched, which is why that work deserves at least as much attention as the choice of model.
Using both together
The two are not rivals. They solve different problems and are often combined.
| Setup | How it works | Fits when |
|---|---|---|
| Retrieval only | A general model answers from retrieved passages | Most internal knowledge assistants |
| Retrieval with a tuned model | Retrieval supplies the facts, and a fine-tuned model writes in a required format or voice | Regulated or customer-facing output with strict form |
| Tuned model for a sub-task | A small fine-tuned model sorts or routes requests, and retrieval answers them | High volume, where the sorting step must be fast and cheap |
A worked example
An engineering firm and its project archive
The firm is invented. It wants staff to ask questions of fifteen years of reports, standards and project files.
- First ideaFine-tune a model on the whole archive, so that it knows the firm's work.
- Why it was droppedStandards are revised every year, some project files are confidential to one client team, and engineers must cite the document they relied on.
- What was builtRetrieval over the archive, filtered by each user's project access, with every answer linking to the passages used.
- Where fine-tuning came in laterA small tuned model to label incoming documents by discipline and project, which made the search more precise.
The knowledge stayed in documents, where it can be updated and checked. The tuning was used for a narrow, repetitive job it does well.
Questions to settle before you choose
Decision checklist
0 of 7 checkedWhat to do next
Gather fifty real questions your staff or customers ask, with the documents that answer them. That set will test either approach, and planning an AI pilot with measurable success criteria shows how to use it.
Our AI integration page describes how we build knowledge assistants on your own systems. You can also send us the fifty questions, and we will tell you which approach we would start with.
Common questions
Does fine-tuning teach a model our company's facts?
Only loosely. A fine-tuned model absorbs patterns from its examples, but it does not store documents and cannot quote them reliably. It may state things that sound like your policies and are not. For factual questions about your company, retrieval is the dependable method, because the model answers from text placed in front of it.
Is RAG cheaper than fine-tuning?
It is usually cheaper and faster to start, since no training is involved. Running cost per question is somewhat higher, because retrieved passages are sent to the model each time. Fine-tuning has a cost up front and again whenever the model must be updated. Which is cheaper overall depends on volume and on how often the information changes.
Does RAG stop the model from making things up?
It reduces the problem and does not remove it. When the right passages are found and the model is told to answer only from them, invented answers become much less frequent. When search returns the wrong material, the model may still answer with confidence. Testing on real questions shows how often each happens.
Is our data used to train the provider's models?
That depends on the provider and the plan. Business offerings from the major providers set out whether data is retained and whether it is used for training, and the terms differ between products. Read the terms for the exact service you will use, and ask your development partner to show you where the data goes.
