RAG vs fine-tuning: retrieve the facts, tune the behaviour

For a company knowledge assistant, use retrieval-augmented generation (RAG), and fine-tune only if your evals prove you need to. Fine-tuning changes how a model behaves: its style, its format, a narrow skill. It is an unreliable way to teach a model your facts, which is the whole job of a knowledge assistant. Retrieval hands the model your current documents at answer time, so answers stay up to date and can cite where they came from.

That is the position on our generative AI service page: for most knowledge work, RAG with strong retrieval beats fine-tuning at lower cost and easier updates, and we pick what wins on your evals. Here is the research behind it, where fine-tuning still earns its place, and how to settle the question on your own data rather than in a meeting.

Fine-tuning is further training of an existing model on your own examples. It is good at changing habits: the shape of an answer, the tone, the way a narrow task gets done. It is poor at loading facts the model has not seen before. Two studies tested this directly, one training on raw documents and one on question-and-answer examples.

Ovadia and colleagues (2023) trained models on raw documents (unsupervised fine-tuning) and compared them with RAG on knowledge-intensive tasks. Fine-tuning "offers some improvement", they found, but "RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge." Models "struggle to learn new factual information through unsupervised fine-tuning". Showing them many variations of the same fact could ease the problem.

Gekhman and colleagues (2024) studied supervised fine-tuning on question-and-answer examples. Examples carrying new facts were "learned significantly slower" than ones consistent with what the model already knew. Their conclusion: models "mostly acquire factual knowledge through pre-training, whereas fine-tuning teaches them to use it more efficiently."

A company knowledge assistant answers questions about your policies, products and procedures. It is judged on four things: whether the answer is right, whether it is current, whether you can trace it to a source, and whether you can correct it when the source changes. Retrieval is built for all four.

The original RAG paper made this case. Lewis and colleagues (2020) noted that when knowledge lives only in a model's parameters (the weights learnt in training), its "ability to access and precisely manipulate knowledge is still limited". They added that "providing provenance for their decisions and updating their world knowledge remain open research problems." With a retrieval index, "knowledge can be directly revised and expanded, and accessed knowledge can be inspected and interpreted." The index can be swapped "to update the model without requiring any retraining."

In practice: when your leave policy changes, you re-index one document. With a fine-tuned model, you retrain, re-test and redeploy.

RAG's weakness is retrieval itself. When a RAG system gives a confident wrong answer, the usual cause is that it found the wrong passage, not that the model misread the right one. That is fixable engineering, covered in why your RAG pipeline retrieves the wrong chunks. It is still a better problem to own than facts locked inside a model you cannot inspect.

Fine-tuning earns its place when the model has the right facts and still gets the answer wrong in shape. Three cases come up:

• A strict output format. The assistant must return the same structure every time, such as fields another system reads, and instructions alone keep letting it slip.
• A house voice. Drafts must sound like your organisation and use your terminology, across every team that uses the tool.
• A narrow, repeated skill. Sorting tickets into your own categories, or pulling the same fields from the same kind of document.

This is the side of fine-tuning the research supports. In the LIMA study, Zhou and colleagues (2023) fine-tuned a model on "only 1,000 carefully curated prompts and responses". It learned "to follow specific response formats from only a handful of examples". The authors concluded that "almost all knowledge in large language models is learned during pretraining."

Try prompts before training. We tune voice, terminology and refusal policy through prompts and retrieval scoring, and use fine-tuning where it is warranted. A prompt change ships in a release and, with versioned prompts, rolls back in seconds. A fine-tune is a training run. It carries three costs:

• Curated examples. Someone has to write and check examples of the behaviour you want, and the model learns whatever habits those examples carry.
• The tune belongs to one base model, the model you started from. When that model is retired or a better one appears, the training and the evaluation are repeated. Without a tune, swapping models is a release behind an eval gate.
• Possible side effects. Ovadia and colleagues name "a degree of catastrophic forgetting" as one possible reason fine-tuning trailed retrieval in their tests: training for one thing can disturb what the model did well before.

Some systems need facts from retrieval and behaviour from tuning. One engagement pattern we run, an editorial AI copilot for a digital publisher, pairs voice-tuned drafting with a required citation pipeline and topic-specific grounding. The voice is a behaviour problem. The facts in each piece are a retrieval problem. Neither tool does the other's job.

Fine-tuning can also target how a model uses retrieval. In RAFT, Zhang and colleagues (2024) trained a model on questions paired with retrieved documents, teaching it "to ignore those documents that don't help in answering the question." They report that it "consistently improves the model's performance" in domain-specific RAG on the three datasets they tested. That is tuning for a behaviour, not for storing facts. It suits an assistant whose retrieval returns the right passage among irrelevant ones, and which then answers from the wrong one.

Do not assume the combination wins. In the Ovadia study, using the fine-tuned model inside the RAG pipeline sometimes helped, but the authors note that "this is not consistent." On the task built from entirely new facts, "combining RAG with fine-tuning shows inferior performance compared to RAG alone." Their tuning was unsupervised, so this is not the last word on every method. It is a fair warning that two tools means two things to evaluate.

The studies above ran on research benchmarks, not your documents, and models change underneath everyone. The honest answer to "RAG or fine-tuning" is the one your own questions give you. We build the eval set before we build the system, on your real data, and the same harness then gates every release, including prompt edits. The sequence:

1. Build a golden set. Thirty to fifty real questions is enough to start, each with the passage that should answer it and the checked answer. Pull them from support tickets and search logs: invented questions are phrased more helpfully than real ones.
2. Score a baseline, then RAG. Run the model with prompts alone, then with retrieval. Measure retrieval recall (how often the right passage comes back) on its own. If the right passage never arrives, nothing downstream can fix it.
3. Sort every failure. "Missing or stale fact" points to retrieval: better documents, chunking or ranking. "Right fact, wrong shape" points to behaviour: fix the prompt first, then consider a fine-tune.
4. Fine-tune only against the behaviour failures that remain. Keep the eval questions out of the training data. Score the tuned model on the full set, because a tune that fixes the format and loses facts is a regression.
5. Keep the gate. Whichever setup wins, every later change runs against the same set, and a quality regression blocks the release.

Neither approach takes hallucinations to zero. The controls that sit on top of both, such as required citations and refusal logic, are in how to stop an AI chatbot making up answers. If the assistant will also act in other systems, read the guardrails an AI agent needs before it can act.

Take thirty questions your team has asked, or would ask, the assistant. Run them, and label each wrong answer "missing or stale fact" or "right fact, wrong shape". If most are the first kind, fine-tuning will not help: fix retrieval and the documents behind it. If most are the second, and prompt changes have not closed the gap, you have a case for a fine-tune and a test set to prove it on.