Pattern guide

RAG, fine-tuning or prompting: how to choose

Sorting the requirement into knowledge, behaviour and cost, so the decision survives a finance review.

ways these programmes stall
5
steps in how we would address it
7
questions answered in full
7
In one paragraph

The choice between retrieval, fine-tuning and prompting is a sorting problem rather than a technology preference. Retrieval is for knowledge that has to be current, attributable or permission-bound. Fine-tuning is for behaviour, format and cost: getting a narrower or cheaper model to do a specific job reliably and consistently. Prompting is for everything else, and it is where you start and where you stop as soon as it is good enough. The expensive mistake is using fine-tuning to inject facts, which is the thing it is worst at: a controlled comparison found retrieval consistently outperformed unsupervised fine-tuning for knowledge injection, both for knowledge seen during training and for knowledge that was entirely new.

James Rooney, Founder

Updated

Our approach

We have not delivered this at enterprise scale, and here is exactly what we are drawing on

This is a decision framework rather than a delivery write-up, and the boundary matters. We have built prompting and retrieval systems on client engagements, including the document pipeline and semantic layer delivered on a live insurance engagement. We have not fine-tuned or trained a model for a client. The comparison this guide leans on is other people's published research, cited below. Where the right answer to your question is a specialist model team rather than us, we will say so on the call.

So what should you do about it

Run the prompting baseline before anyone builds anything, and write the number down. It costs a few days, it is the only thing that makes any later comparison meaningful, and in our experience it settles more architecture arguments than any amount of design discussion. If the baseline is close to acceptable, the answer is prompting plus retrieval and the fine-tuning conversation quietly goes away, which saves you a great deal. If it is a long way off, you now have a specific, measured gap to describe to whoever you take it to, including us. Either way you own the evaluation set at the end of it, which is the durable asset in this whole exercise.

Why this matters now

The comparison has been run properly and the result is unfashionable. A controlled study of knowledge injection reported that retrieval consistently outperformed unsupervised fine-tuning, both for knowledge the model had seen in training and for knowledge that was entirely new, and that models struggle to learn new factual information through unsupervised fine-tuning at all. Separately, work on long inputs found that a model's use of its context degrades significantly when the relevant passage sits in the middle of it, which is the quiet limit on the assumption that a large enough context window makes the question disappear. Neither finding is obscure, and neither tends to reach the meeting where the decision is made.

Ask a technical questionanswers from all 12 guides
Ask anything technical about rag, fine-tuning or prompting: how to choose and I will answer from this guide, and tell you first whether this is work we have delivered or an approach we would be taking.

Prefer to talk it through? Ask us on a discovery call →

Diagnosis

Where these programmes actually stall

Not the risks a vendor lists. The ones that stop the work.

01

The question is asked as a technology choice when it is a knowledge-freshness question

Whoever read the most recent article proposes the approach, and the discussion becomes a preference argument nobody can settle with evidence. The question that actually decides it is duller: how often does the correct answer change, does it differ by who is asking, and does anyone have to be able to see where it came from. Answer those three and the architecture is largely determined before anyone has named a technique.

02

Fine-tuning is chosen to teach facts, which is what it is worst at

The intuition is that a model trained on your documents will know your business, and the published comparison says otherwise: retrieval outperformed unsupervised fine-tuning for knowledge injection across the board, and models struggled to learn new factual information that way at all. The cost of learning this yourself is a training run, an evaluation, and a quarter. It is the most common expensive mistake in this decision and it is entirely avoidable by reading one paper.

03

Nobody costed the second training run, or the fifth

Fine-tuning is not a project with an end, it is a standing obligation: curating and maintaining the training data, retraining when the base model is deprecated, re-running the evaluation each time, and holding a version history you can explain to whoever asks how a particular decision was reached. Programmes that costed the first run and not the lifecycle discover the difference when a provider retires the base model on a timetable that is not theirs.

04

Prompting is dismissed as unserious and then quietly does most of the work

Careful prompting with a good retrieval design covers a surprising share of enterprise use cases, and it is dismissed because it does not sound like engineering. The consequence is that nobody establishes the prompting baseline, so no later comparison has anything to be measured against, and the more expensive option cannot be shown to be worth its cost even when it is.

05

Nobody wrote down what would make them change their mind

A decision taken with no stated trigger for revisiting it becomes an identity, and defending it becomes somebody's job. Write down the conditions that would flip the choice, the measurement that would detect them, and the date the decision gets re-examined, at the moment it is taken. In a field where the base models change quarterly, a decision with no expiry is a decision you will be living with long after its reasoning expired.

Prescription

How we would address it

Grounded in The Tenhaw Way and the engagements written up in our case studies.

  1. 01

    Sort the requirement into knowledge, behaviour, format and cost

    Four buckets, and most requirements split across them rather than sitting in one. Knowledge is what the system needs to know that the model does not: retrieval. Behaviour is how it should reason, in what style, following which house conventions: prompting first, fine-tuning if prompting cannot hold it. Format is the shape of the output: schemas and structured output before anything else. Cost is whether a smaller model can be made to do the narrow job: the one place fine-tuning has a clear economic argument. Doing this sort in a room with the business owner takes an afternoon and prevents a quarter of drift.

  2. 02

    Establish the prompting baseline first, and make everything answer to it

    Before any architecture, run the evaluation set against a strong model with careful prompting and no other machinery, and record the number. That baseline is the thing every later option has to beat by enough to justify its cost and its lifecycle. Without it, teams compare a new approach against an impression, and an impression always loses to something that was built with more effort.

  3. 03

    Use retrieval where the answer must be current, attributable or permission-bound

    Three tests, and any one of them points at retrieval. If the correct answer changes on a timescale shorter than your retraining cycle, it must be retrieved. If a reader has to be able to open the source, it must be retrieved. If two people asking the same question should get different answers because they are entitled to see different things, it must be retrieved, because a model's parameters cannot be permission-checked.

    Retrieval and permission-aware access, in full
  4. 04

    Reserve fine-tuning for form and cost, and say exactly what it buys

    Legitimate reasons to fine-tune: a consistent output shape prompting cannot hold reliably, a house style or tone, a narrow classification task with plentiful labelled examples, or the same job done acceptably by a smaller and cheaper model at volume. Each of those is measurable before and after. If the case for a fine-tune is that the model will know more about the company, the case is wrong and the research says so.

  5. 05

    Keep one evaluation set across every option

    The same cases, the same scoring, the same thresholds, whichever approach is being tested. Teams that build a new evaluation for each candidate are comparing three systems on three different exams and then choosing a winner. The evaluation set is also the asset that outlives the decision: it survives model upgrades, supplier changes and the approach itself.

    How to build the evaluation set
  6. 06

    Cost the whole life, not the experiment

    For each option: the build, the per-run inference, retrieval and index refresh where it applies, training and retraining where it applies, human review of exceptions, monitoring, and a full re-evaluation on every model upgrade. Our audits produce an estimated run cost per candidate workflow before anything is committed, because this is the number that decides between designs and it is almost always the one missing from the comparison.

    Building the case a finance function will accept
  7. 07

    Write the decision down with the conditions that would reverse it

    One page: what was chosen, what it was measured against, what it costs to run, what would change the answer, and when it gets re-examined. That page is what stops the same argument being had every quarter by different people, and it is what lets a new engineer understand why the system is shaped as it is without inferring it from the code.

Knowledge is what the system needs to know that the model does not: retrieval.
Provider notes

Where the model provider genuinely matters here

This is one of the places where provider choice is not cosmetic. Whether fine-tuning is offered at all, on which models, at what price, with what hosting and residency position, and how long a fine-tuned model stays supported once the base model moves on, differ substantially and change the arithmetic rather than the aesthetics. So does cost per token, which decides whether a retrieval design that passes a large context is affordable at your volume. We would test a shortlist against your own data rather than against a public benchmark, and we would get the deprecation and support position in writing before anyone commits to a fine-tune.

Where this usually starts

Agentic Proof of Concept

Pick the workflow. Two to four weeks later, look at a working thing. Fixed price · £20k–£55k · 2–4 weeks.

What that engagement covers

RAG, fine-tuning or prompting: how to choose: your questions

Do we need to fine-tune or use RAG?

For getting your own knowledge into the system, retrieval, and the evidence is fairly direct: a controlled comparison of knowledge injection found retrieval consistently outperformed unsupervised fine-tuning, both for knowledge the model had seen in training and for knowledge that was entirely new, and that models struggle to learn new facts through unsupervised fine-tuning at all. Retrieval also gives you three things fine-tuning cannot: content that is current without retraining, answers that cite a source the reader can open, and access control applied per user at query time. Fine-tuning earns its place for behaviour, format and cost, not for facts. Most enterprise systems that work are prompting plus retrieval, and a fine-tune later if the economics call for one.

When is fine-tuning actually the right answer?

Four cases, and they are all about form or money rather than knowledge. When you need an output shape or house style that prompting cannot hold reliably across the long tail. When you have a narrow classification or extraction task with plenty of labelled examples and a stable definition of correct. When volume is high enough that running a smaller, cheaper, fine-tuned model beats a large general one on cost at the same measured quality. And when latency matters enough that a smaller model is the only way to meet the budget. In all four, the case is measurable before you commit, which is the test: if you cannot state the number that would prove it worked, it is not the right answer yet.

Is prompting on its own enough?

More often than the market implies, and you should find out before spending anything else. Careful prompting against a strong model, with structured output and a well-designed retrieval step, covers a large share of enterprise use cases, and it is the only option with no training lifecycle attached. The reason to establish it first is not economy for its own sake: it is that without a prompting baseline you have nothing to measure the expensive options against, so you cannot demonstrate that they were worth their cost even when they were.

Does a bigger context window remove the need for retrieval?

No. Published work on long inputs found performance is highest when the relevant information sits at the beginning or the end of the context and degrades significantly when a model has to use something in the middle, including in models built for long contexts, so a larger window is not the same as reliable attention across it. Cost and latency also scale with what you put in the window, on every call. And a context window solves none of the reasons an enterprise needs retrieval in the first place: permissions per user, freshness without a rebuild, and an answer that cites a source. Retrieval is what keeps the context small, current and defensible.

What does fine-tuning commit us to after the first run?

A lifecycle rather than a deliverable. Training data has to be curated, kept current and governed, because it is now part of how your system behaves. Every base model deprecation forces a retrain on the provider's timetable rather than yours. Every retrain forces a full re-evaluation, so you need the evaluation set anyway. And you carry a version history, because when someone asks why a decision came out as it did in March, the answer involves which model version was live. None of that is a reason not to fine-tune. It is a reason to make sure the case is about form or cost, where the benefit is durable, rather than about facts, where it is not.

How do we choose without running a three-way bake-off?

By sorting the requirement first, which is an afternoon rather than a quarter. Split what you need into knowledge, behaviour, format and cost. Anything in the knowledge bucket that has to be current, attributable or permission-bound is retrieval, and that is settled. Everything else starts at prompting with structured output, because that is the cheapest thing that can work and it establishes the baseline. Only what is left after that, and only where the measured gap is large enough to justify a training lifecycle, is a fine-tuning candidate. A full bake-off is worth running for one decision at most, and usually the sorting has already made it unnecessary.

Can we combine them?

Yes, and most systems that work in production do. A typical shape is careful prompting for the reasoning and the house conventions, retrieval for anything that has to be current or permission-bound, structured output for the shape, and possibly a smaller fine-tuned model handling one high-volume narrow step inside the workflow where the economics justify it. The discipline that makes combining safe is holding the evaluation set constant across every configuration, so you can attribute a change in the score to the thing you changed rather than to the general direction of travel.

Sources

Where the checkable claims came from

Every source below was opened and read before it was attached. Where nothing survived that check, the claim on this page was softened rather than given a plausible-looking link.

  1. 01
    Ovadia, Brief, Mishaeli and Elisha, Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (2023)

    Source of the headline finding: retrieval consistently outperformed unsupervised fine-tuning for knowledge injection, for knowledge seen in training and for entirely new knowledge, and models struggle to learn new factual information through unsupervised fine-tuning.

  2. 02
    Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023)

    The limit on the argument that a large context window replaces retrieval: performance degrades significantly when the relevant information is in the middle of a long input.

  3. 03
    Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)

    The original statement of why an explicit, non-parametric memory is used alongside a model's parametric one.

Keep reading

The closest guides to this one

Nearest first, then the rest. Each one carries the same label: written from delivery, or the method we would bring. All 12 are on the hub.

Our approach

Retrieval, RAG and permission-aware knowledge access

Answering from your own knowledge, without answering from documents the person asking is not allowed to see.

Our approach

MCP, tool calling and integrating agents with your systems

The moment an agent can call your systems, integration stops being plumbing and becomes an access decision.

Our approach

Guardrails, hallucination and accuracy control

You will not stop a language model being wrong. You can decide in advance what happens when it is.

Delivered

Document intelligence to business intelligence

Getting information out of PDFs and into something the business can decide with.

Our approach

Agent evaluation and assurance

Why AI pilots never reach production, and what it takes to get one through the gate.

Our approach

AI governance and regulatory evidence

Building the evidence as a by-product of the work, rather than assembling it under deadline.

All 12 pattern guides, grouped

Talk to us about rag, fine-tuning or prompting: how to choose.

A 30-minute call with James Rooney. We'll tell you honestly which parts of this we have done before and which we would be doing for the first time, and you'll leave with a rough scope either way.

30 minutesWith James personallyNo obligation
See the proof first

Most organisations start with a fixed-price Agent-Readiness Audit · £30k–£90k · 6–8 weeks