RAG, fine-tuning or prompting: how to choose

For getting your own knowledge into the system, retrieval. Tenhaw builds these systems as prompting plus retrieval, including the document pipeline and semantic layer delivered for a client in the London specialty insurance market. A controlled comparison of knowledge injection found that retrieval consistently outperformed unsupervised fine-tuning, both for knowledge the model had seen in training and for knowledge that was entirely new, and that models struggle to learn new facts through unsupervised fine-tuning at all. Retrieval also gives you three things fine-tuning cannot: content that is current without retraining, answers that cite a source the reader can open, and access control applied per user at query time. Fine-tuning earns its place for behaviour, format and cost, not for facts.

Four cases, and they are all about form or money rather than knowledge. When you need an output shape or house style that prompting cannot hold reliably across the long tail. When you have a narrow classification or extraction task with plenty of labelled examples and a stable definition of correct. When volume is high enough that running a smaller, cheaper, fine-tuned model beats a large general one on cost at the same measured quality. And when latency matters enough that a smaller model is the only way to meet the budget. In all four the case is measurable before you commit, and that is the test. If you cannot state the number that would prove it worked, it is not the right answer yet.

More often than the market implies, and you should find out before spending anything else. Careful prompting against a strong model, with structured output and a well-designed retrieval step, covers a large share of enterprise use cases, and it is the only option with no training lifecycle attached. Tenhaw took a London specialty insurance business from PDFs to business intelligence on Azure in a two-week proof of concept, on ground it had circled for roughly a year, with no training run anywhere in it. You establish it first not to save money but because without a baseline you have nothing to measure the expensive options against, so you cannot show they were worth their cost even when they were.

No. Published work on long inputs found performance is highest when the relevant information sits at the beginning or the end of the context and degrades significantly when a model has to use something in the middle, including in models built for long contexts, so a larger window is not the same as reliable attention across it. Cost and latency also scale with what you put in the window, on every call. And a context window solves none of the reasons an enterprise needs retrieval in the first place: permissions per user, freshness without a rebuild, and an answer that cites a source. Retrieval is what keeps the context small, current and defensible.

A lifecycle rather than a deliverable. Training data has to be curated, kept current and governed, because it is now part of how your system behaves. Every base model deprecation forces a retrain on the provider's timetable rather than yours. Every retrain forces a full re-evaluation, so you need the evaluation set anyway. And you carry a version history, because when someone asks why a decision came out as it did in March, the answer involves which model version was live. None of that is a reason not to fine-tune. It is a reason to make sure the case is about form or cost, where the benefit is durable, rather than about facts, where it is not.

By sorting the requirement first, which is an afternoon rather than a quarter. Split what you need into knowledge, behaviour, format and cost. Anything in the knowledge bucket that has to be current, attributable or permission-bound is retrieval, and that is settled. Everything else starts at prompting with structured output, because that is the cheapest thing that can work and it establishes the baseline. Only what is left after that, and only where the measured gap is large enough to justify a training lifecycle, is a fine-tuning candidate. A full bake-off is worth running for one decision at most, and usually the sorting has already made it unnecessary.

Yes, and most systems that work in production do. A typical shape is careful prompting for the reasoning and the house conventions, retrieval for anything that has to be current or permission-bound, structured output for the shape, and possibly a smaller fine-tuned model handling one high-volume narrow step inside the workflow where the economics justify it. The discipline that makes combining safe is holding the evaluation set constant across every configuration, so you can attribute a change in the score to the thing you changed rather than to the general direction of travel.

Almost never, and the cheaper route is worth exhausting first. Output shape belongs in the format bucket, where schemas and structured output come before anything else, because they are the direct way to make a model return the fields you asked for and they cost you an afternoon rather than a training lifecycle. Fine-tuning becomes reasonable only when careful prompting and a schema still cannot hold the shape reliably across the long tail, and that is a measurable claim rather than an impression. Run your evaluation set against the prompted version first and record the number. If a fine-tune then closes the gap, you can show exactly what it bought and what it commits you to afterwards.

Over the life of the system usually yes, though they spend money in different places, so compare lifecycles rather than build quotes. Retrieval spends its money in the running, on per-run inference and retrieval plus the index refresh that keeps content current. Fine-tuning looks like a one-off and is not. You curate and maintain the training data, retrain when the base model is deprecated, and re-run the full evaluation every time. Prompting is cheaper than both, which is why the prompting baseline is worth establishing before either gets funded. Tenhaw publishes the method behind that lifecycle comparison in full, free to adopt without hiring anyone.

Four things, and get the last one in writing. Tenhaw runs that comparison against a client's own data rather than against a public benchmark, and treats the support commitment as part of the price. Whether fine-tuning is offered at all, on which models and at what price, because that differs substantially between providers. Where a fine-tuned model is hosted and what the data residency position is. What cost per token looks like at your volume, since that decides whether a retrieval design passing a large context is affordable anyway. And how long a fine-tuned model stays supported, because once the base model is retired the retrain happens on the provider's timetable, not yours.

It can, and the part it settles is the cost. A Tenhaw audit ends in working prototypes and an estimated run cost per candidate workflow before anything is committed. That number is what decides between designs, and it is almost always the one missing when a team argues retrieval against fine-tuning. The audit also forces the sorting into knowledge, behaviour, format and cost, with the business owner in the room rather than only the engineers. That sort takes an afternoon and the method behind it is published free to copy, so run it yourself first. Buy the audit when the decision has to survive a finance review.

Most likely because it was asked to do the one thing it is worst at. The repair is retrieval work of the kind Tenhaw did on a live engagement for a client in the London specialty insurance market, scoring each extracted value's confidence from its source provenance, model certainty and a search cross-check, none of which a fine-tune can give you. A controlled comparison of knowledge injection found retrieval consistently outperformed unsupervised fine-tuning, for knowledge seen in training and for knowledge that was entirely new. Training on your documents does not reliably put those facts inside the model, and it gives you no citation and no per-user permission check either. Keep the fine-tune, if it is worth keeping, for output shape, house style or cost at volume.

Not on its own, but three questions will tell you what it is worth. What is the prompting baseline, measured on your own evaluation set, that this is meant to beat, and by how much? If nobody has run one, the comparison is against an impression. Is the case about knowledge, or about form and cost, given that a controlled comparison found retrieval consistently outperformed unsupervised fine-tuning for knowledge injection? And how often does the correct answer change, because anything moving faster than your retraining cycle has to be retrieved rather than trained in. Tenhaw builds prompting and retrieval systems, and where the honest answer is a specialist model team we will point you at one.

A few days, and it is the highest-value few days in the whole decision. You need an evaluation set of real cases with agreed scoring, then you run them against a strong model with careful prompting and no other machinery, and you write the result down. That number is what every later option has to beat by enough to justify its cost and its lifecycle. Tenhaw pairs on this rather than running it offstage, which is why, on a two-week proof of concept for a London specialty insurance business, the client's own engineer finished 70% confident they could run the process unaided. You also come out of it owning the evaluation set, which is the durable asset in the whole exercise.

The business owner above all, and whoever can answer for the budget. The sort is not really technical. You are splitting the requirement into knowledge, behaviour, format and cost, and the questions that settle it are how often the correct answer changes, whether it differs depending on who is asking, and whether anyone has to be able to see where an answer came from. Only the business side can answer those, which is why an engineering-only room turns the decision into a preference argument nobody can settle with evidence. Done together it takes an afternoon and prevents a quarter of drift, because the architecture is largely determined once those three answers are on the table.

It sharpens it rather than changing it, and it usually pushes further towards retrieval. Tenhaw's live engagement in the London specialty insurance market is retrieval and semantic-layer work, not a fine-tune. Two of the three tests that point at retrieval are regulatory. If a reader has to be able to open the source behind an answer it has to be retrieved, and if two people asking the same question are entitled to see different things it has to be retrieved, because a model's parameters cannot be permission-checked. Fine-tuning also brings an evidence obligation, because you carry a version history for the day someone asks why a decision came out as it did in March, and the answer involves which model version was live.

The evaluation set carries over, and it is the most durable thing you will build. Held constant across every option rather than rebuilt per candidate, an evaluation set survives model upgrades, supplier changes and the approach itself. Prompts and retrieval design are your work product rather than the provider's, though you re-run the full evaluation on any model change regardless. A fine-tune is the piece that does not travel. When the base model is retired you retrain on the provider's timetable, which is why the support position is worth getting in writing first. Tenhaw writes engagements the same way, with the client owning all work product and the exit date agreed at kickoff.