RAG vs Fine Tuning: How to Choose Correctly

RAG vs fine tuning is usually framed as a technical choice and it is really a diagnostic one. Before picking either, name the problem: does the model lack knowledge, or does it behave wrong? Retrieval fixes the first. Tuning fixes the second. Choosing the wrong one produces an expensive project that does not move the failure you started with.

I ask clients one question before this conversation goes further: show me five outputs you are unhappy with. The answer is usually visible in them.

RAG vs fine tuning, the actual difference#

Retrieval augmented generation finds the relevant pieces of your documents at question time and puts them into the prompt. The model then answers from material it can quote. Nothing about the model changes. Update a document and the next answer reflects it immediately.

Fine tuning continues training the model on your examples, adjusting its weights. It learns patterns of behaviour: your output format, your tone, how to classify things the way you classify them. It does not reliably learn facts, and facts learned this way cannot be corrected without training again.

The compressed version: RAG changes what is in front of the model. Fine tuning changes the model.

Diagnose before you build#

That last row matters. Neither technique makes a model better at reasoning. If the task is genuinely hard, decompose it into steps rather than hoping training fixes it.

Why RAG is the default for company knowledge#

Three reasons, and they are practical rather than theoretical.

Facts change. Prices, policies, documentation, staff. Retrieval reflects an edit the moment it is saved. Retraining does not.

Citations. A retrieval system can show which document an answer came from, which is the difference between a tool people trust and a tool they double-check. In regulated work it is not optional.

Permissions. You can filter retrieval by what the user is allowed to see. Weights have no concept of access control, so a fine tuned model that absorbed confidential material will happily repeat it to anyone.

The cost is real: you pay for retrieved context on every call. Claude API pricing covers how to keep that under control with caching and by retrieving three paragraphs rather than attaching the whole manual.

Where fine tuning genuinely earns its cost#

Output format at scale. When every response must be the same structure and prompting gets you to 95%, tuning closes the gap and removes the retry logic.

Voice. A tone that prompts approximate but never nail, across thousands of outputs.

Narrow classification. Your categories, your edge cases, at volume. A small tuned model often beats a large prompted one here, and costs a fraction to run.

Latency and unit cost. If tuning lets you move from a large model to a small one for a high-volume step, the economics can be decisive.

The requirement people underestimate: you need good examples. Hundreds to thousands of them, consistent, representing what you actually want. Assembling that dataset is most of the work and most of the cost, and inconsistent examples produce a model that is confidently inconsistent.

The order to try things in#

1. Improve the prompt and the context. A clear instruction, a worked example, an explicit schema. A day of work, and it resolves a surprising share of projects that had already budgeted for training. Prompt engineering examples and few-shot prompting cover the techniques.

2. Add retrieval. If the failure is knowledge, this is your answer. Start with the simplest version that works: chunk documents sensibly, retrieve a handful, include the source.

3. Constrain the output. Ask for JSON against a schema and validate it. This removes most format complaints without touching a model.

4. Fine tune. Only once you have examples of the behaviour you want, at volume, and a measurable gap that the first three did not close.

Most teams stop at step two and should.

Building the RAG version well#

The naive version disappoints and people blame the technique. Four things fix most of it.

Chunk by meaning, not by character count. A chunk that cuts a table in half retrieves badly.

Keep metadata. Source, date, section, permission scope. You will need all four.

Retrieve fewer, better. Ten mediocre chunks are worse than three relevant ones, and cost more.

Evaluate retrieval separately from generation. When an answer is wrong, you need to know whether the right document was found and ignored, or never found at all. Those are different bugs with different fixes.

Context engineering covers the wider discipline of deciding what goes into the window.

The short version#

Diagnose first: knowledge or behaviour. Use retrieval for facts, because they change and need citing. Use tuning for format, voice and narrow classification at volume, once you have the examples. Try a better prompt before either, and expect to use both eventually.

If you want this designed around your documents and your team trained to maintain it, that is what my AI training packages cover, and the AI automation agency page is where I do this work for clients.