Fine-Tuning vs. Prompting vs. RAG: Choosing the Right Lever

Ask five teams why they chose to fine-tune a model, and at least two of them will describe a problem fine-tuning was never going to solve. This isn't a competence issue — it's a categorization issue. Fine-tuning, retrieval-augmented generation, and prompting get discussed as if they're competing techniques on a single spectrum of sophistication, with fine-tuning at the serious end and prompting dismissed as the thing you do before you're ready for real engineering. That framing is wrong, and it's expensive when teams act on it.

These are three different tools that fix three different failure modes. The question was never which one is the most advanced. It's which failure mode you're actually looking at, and most of the waste we see comes from skipping that diagnosis and reaching straight for whichever lever felt most like "doing something serious."

Three Tools, Three Different Problems They Actually Solve

Start with what each one is actually built to fix, because the confusion mostly comes from skipping this step. Prompting controls behavior at the moment of the request — it tells the model what to do, how to format the answer, what tone to use, what constraints to respect, and it does this fresh, every time, using only what's in the context window. It's fast to change, fast to test, and costs nothing beyond the inference call you were already making.

Retrieval-augmented generation solves a different problem entirely: it gives the model access to specific information it doesn't reliably have, either because that information postdates its training, is proprietary and was never in the training data to begin with, or changes too frequently to bake into weights that only get updated occasionally. RAG doesn't change how the model behaves. It changes what the model knows at the moment it's answering, by handing it the relevant material alongside the question. Fine-tuning solves neither of those problems well and instead solves a third one: making a specific behavior, style, or skill consistent and reliable at scale without having to re-explain it in every single prompt. It's the right tool when an instruction is complicated enough, or the edge cases numerous enough, that no reasonably sized prompt can reliably capture it, and when the model needs to exhibit that behavior automatically, without a human curating the context every time. Confusing these three — using fine-tuning to inject knowledge, or RAG to fix a formatting problem, or a longer prompt to try to teach a skill that needs hundreds of examples — is where most of the wasted engineering effort in this space comes from.

The Most Common Mistake: Fine-Tuning to Fix a Knowledge Problem

This is the mistake we see most often, and it's expensive in a specific, avoidable way. A team has a large body of internal documents — policies, product documentation, historical records — and wants the model to "know" that material, so they fine-tune on it, treating the fine-tune as something like teaching the model to memorize a textbook. It sort of works, in the sense that the model will often be able to reproduce fragments of that information afterward. But it's a bad tool for the job on nearly every axis that matters in production.

Fine-tuning captures a snapshot. The moment the underlying documents change — and internal documentation always changes — the fine-tuned model is stale, and the fix is another full fine-tuning cycle, not a quick update. It's also much harder to audit: when the model states a fact, there's no clean way to trace that statement back to a specific source document, which is a serious problem the moment anyone downstream asks "where did that come from" for a compliance or accuracy reason. And it's simply the more expensive and slower way to get information into a model's effective knowledge, compared to retrieval, which updates the instant you change what's in the retrieval index and naturally supports citing the exact passage an answer came from. Retrieval is, for the overwhelming majority of "the model needs to know our stuff" problems, the correct default. Fine-tuning for knowledge injection should be treated as a red flag worth a second look, not a first instinct.

The Second Most Common Mistake: Building a RAG Pipeline When a Better Prompt Would've Worked

The opposite mistake is quieter but nearly as common: standing up a full retrieval pipeline — a vector database, a chunking strategy, an embedding model, a retrieval-ranking step — for a task where the model already has everything it needs and the actual problem is that the prompt isn't giving it clear enough instructions. This happens because retrieval has become the default "serious" architecture for anything that feels complicated, so teams reach for it even when the failure they're seeing has nothing to do with missing knowledge.

The tell is usually visible if you look closely at the actual failures. If the model is producing inconsistent formatting, missing steps in a multi-part task, or answering with the wrong tone, none of that is a knowledge gap — retrieval can't fix any of it, because the model already knew everything it needed to know. What it needed was a clearer, better-structured prompt: explicit formatting instructions, a couple of well-chosen examples of the exact output shape you want, explicit constraints on what to include and exclude. Teams that build a retrieval pipeline to fix this kind of failure end up with more infrastructure, more latency, more places for something to break, and a problem that's still unsolved, because the pipeline was never addressing the actual cause. Before reaching for retrieval, it's worth asking plainly whether the task is actually missing information, or whether it's missing clarity — those are different diagnoses with completely different treatments.

When Fine-Tuning Actually Earns Its Cost

None of this makes fine-tuning a bad tool — it makes it a specific one, and there are real situations where it's clearly the right call. High-volume tasks that need a very particular style or format applied with total consistency are a good example: if you're generating the same kind of output thousands of times a day and it absolutely has to sound a specific way every time, baking that consistency into the model's weights is more reliable than hoping a prompt gets followed identically on every single call, especially across a long operating history where prompts tend to drift as people edit them.

Fine-tuning also earns its cost when the goal is distillation: taking a complex, expensive prompt chain running on a large model — one that strings together several reasoning steps or relies on a long, carefully engineered instruction set — and compressing that behavior into a smaller, cheaper, faster model fine-tuned to reproduce it directly. That trade converts an ongoing inference cost into a one-time training cost, which is exactly the right shape of tradeoff at sufficient volume. And it's the right tool for behaviors that are genuinely easier to demonstrate than describe — narrow tasks with enough edge cases that no prompt could enumerate them all, but where a large set of labeled examples can teach the pattern more reliably than any instruction. When the problem is "I know it when I see it, across a thousand variations," fine-tuning is often the only tool built for that shape of problem.

Prompting Is Underrated Because It's Undignified

Part of why teams skip past prompting is that it doesn't feel like engineering. There's no pipeline to diagram, no infrastructure to provision, nothing to put on an architecture slide. It's just careful writing — structuring instructions clearly, providing a few well-chosen examples of the output you want, stating constraints explicitly instead of hoping they're implied, and iterating quickly based on where the model actually fails rather than where you assumed it would. That lack of visible complexity makes it feel like the unserious option, the thing you do before the real work starts.

In practice, disciplined prompting solves a surprisingly large share of the problems teams assume require retrieval or fine-tuning, precisely because a large share of production failures turn out to be instruction-clarity problems wearing a costume. It's also the cheapest and fastest thing to iterate on by a wide margin — a prompt change is a text edit and a test run, not a data pipeline or a training job — which makes it the correct starting point regardless of where you eventually end up. Even tasks that do ultimately need retrieval or fine-tuning benefit from getting the prompt right first, because a sloppy prompt will undermine a well-built retrieval system or a well-executed fine-tune just as easily as it undermines a bare model call.

How GLSRM Thinks About the Decision

At GLSRM, we default to a fairly unglamorous order of operations, and we think most teams would save themselves real money by adopting something similar. We start with prompting, because it's the cheapest possible experiment and it forces you to actually look at how the model fails before reaching for a bigger tool. Most of the time, close attention to the failure mode at this stage tells you exactly what you need next, which is the real value of starting cheap — not that prompting always suffices, but that it's the fastest way to find out what actually will.

If the failure mode is specifically that the model doesn't know something — a fact, a document, something current or proprietary — we add retrieval, because that's precisely the shape of problem retrieval was built to solve, and it stays current in a way no fine-tune can. If the failure mode instead is that the model knows perfectly well what to do and simply won't do it reliably at the volume or consistency we need, or if cost and latency at scale demand a smaller specialized model standing in for a larger general one, that's when fine-tuning earns its place. We rarely start there, and we're skeptical of teams that do, because starting with the most expensive tool usually means skipping the diagnosis that would have told you whether you needed it at all.

The lever a team reaches for first says more about organizational habit than about the actual problem in front of them — comfort with infrastructure, appetite for a training pipeline, a sense that the fancier tool must be the more rigorous choice. None of that is diagnosis. The actual discipline is small and unglamorous: figure out whether the model doesn't know something, won't do something, or simply wasn't asked clearly enough, and only then pick up the tool built for that specific failure. Skipping that question is the expensive part. Everything after it is comparatively easy.