Fine-tuning vs RAG vs prompt engineering: a practical comparison

Fine-tuning vs RAG vs prompt engineering is a choice of which part of the system to change. Prompt engineering changes the instructions. Retrieval-augmented generation (RAG) changes the context the model reads at request time. Fine-tuning changes the model’s weights. Each fixes a different kind of failure, so the right first step is to diagnose why the current system is wrong.

Most comparisons stop at “RAG for knowledge, fine-tuning for behavior.” That is the right instinct, but it isn’t a method. This guide adds the evidence behind it, the cost math that has shifted in the last two years, a decision flowchart and a protocol for testing the options on your own data.

What does each approach actually change?

	Prompt engineering	RAG	Fine-tuning
What changes	Instructions and examples in the prompt	Documents inserted into the prompt per request	Model weights
Model weights	Unchanged	Unchanged	Updated (often a small LoRA adapter)
Time to first result	Minutes to days	Days to weeks	Weeks
Main dependency	A clear task definition	A searchable, well-chunked corpus	Hundreds to thousands of good examples
Updating it	Edit the prompt	Re-index documents	Retrain and re-evaluate

OpenAI’s accuracy optimization guide frames the same split as two axes. Context optimization is for when the model lacks knowledge, has out-of-date knowledge or needs proprietary information, and it “maximizes response accuracy.” LLM optimization is for inconsistent formatting, wrong tone or reasoning that isn’t followed reliably, and it “maximizes consistency of behavior.” Prompting sits at the origin of both, which is why it comes first.

What problems does each one solve?

Prompt engineering: fast iteration and task definition

Prompting is the cheapest experiment and forces you to write down what a correct output looks like. Clear instructions, a schema, a few worked examples and explicit handling of edge cases fix a surprising share of failures.

Its reach has grown. With long context windows, many-shot in-context learning with hundreds or thousands of examples produced significant gains across many tasks, and on low-resource translation it performed close to supervised fine-tuning. The trade-off is that every example is paid for on every request, and inference cost grows linearly with the number of examples.

Prompting runs out when the model ignores instructions under real input variety, when the prompt becomes so long that cost and latency hurt, or when the knowledge simply isn’t in the model.

RAG: knowledge, freshness, citations and access control

RAG, introduced by Lewis et al. (2020), retrieves passages from an index and gives them to the model with the question. It was built for the two problems the authors called open: “providing provenance for their decisions and updating their world knowledge.” In practice RAG is the right tool when:

• Answers depend on documents the model never saw: internal policies, contracts, product catalogs, tickets.
• The facts change. Re-indexing a document is immediate; retraining is not.
• You need citations that a reviewer can check.
• Different users may see different documents. Retrieval can enforce permissions per request; weights cannot.

RAG adds its own failure modes. If retrieval misses the right passage, the model can’t use it. More context isn’t free either: Liu et al. found models use information at the start or end of a long context best and “significantly” worse when it sits in the middle. Measure retrieval quality separately from answer quality.

Fine-tuning: behavior, format, consistency, cost and latency

Fine-tuning trains the model on input-output examples so the behavior is built in rather than described. It is the right tool when:

• Output must follow a strict format or schema every time.
• You need a consistent house style, tone or labeling policy that is hard to describe in words but easy to demonstrate.
• A smaller model must match a larger one on a narrow task, cutting per-request cost and latency.
• Long prompts with many examples are too slow or expensive at your volume.

With LoRA, this is cheaper than many teams assume: Thinking Machines’ “LoRA Without Regret” found LoRA matched full fine-tuning on typical post-training datasets when applied to all layers. The domain-specific model playbook covers the data work, which is where most of the effort goes.

Why is fine-tuning the wrong way to add knowledge?

Because the evidence says so, consistently.

• Ovadia et al. (EMNLP 2024) compared fine-tuning on domain text with RAG across knowledge-intensive tasks: “while unsupervised fine-tuning offers some improvement, RAG consistently outperforms it,” for both previously seen and entirely new knowledge.
• Gekhman et al. (EMNLP 2024) found fine-tuning examples with new facts “are learned significantly slower” than familiar ones and, once learned, “linearly increase the model’s tendency to hallucinate.” Their reading: models acquire facts mostly in pretraining, and fine-tuning teaches them to use what they already know.

So fine-tuning a model on your product manual will not make it a reliable source on that manual, and the attempt can make it more confidently wrong. Put the manual in an index and fine-tune, if at all, for how answers should be framed.

How do the costs and maintenance compare?

Cost driver	Prompt engineering	RAG	Fine-tuning
Upfront	Low: writing and testing prompts	Medium: ingestion, chunking, embeddings, index	Medium to high: data prep, labeling, training, evaluation
Per request	Prompt tokens, which grow with examples	Prompt plus retrieved tokens, plus retrieval calls	Often lowest: short prompt, possibly a smaller model
Ongoing	Re-test on model upgrades	Keep index fresh, monitor retrieval quality	Retrain on new base models or task changes
Main risk	Brittle under input variety	Wrong or missing passages	Stale, overfit or regressed model

Per-request cost is simple arithmetic: tokens per request × requests × price. Long prompts used to be the strongest cost argument for fine-tuning, but prompt caching has changed that. Anthropic’s prompt caching, for example, bills cache reads at 0.1 times the base input price on most models (some newer models are lower), and cache writes at 1.25 times (5-minute lifetime) or 2 times (1-hour lifetime), as of November 2026.

Example (prices assumed for illustration): suppose 1M requests a month share a 6,000-token instruction and example prefix, with 500 tokens of unique input each. At an assumed $1.00 per million input tokens, the prefix costs $6,000 a month uncached. If nearly every request hits the cache at 0.1 times the price, it costs about $600 plus occasional writes at 1.25 times. A fine-tuned model that needs only a 300-token prompt, uncached at the same price, would cost about $300 for that part, saving roughly another $300 a month (the $500 of unique input is the same either way), but it adds training, evaluation and possibly dedicated serving. At this volume, caching alone may be the better answer. At 50M requests a month, or when latency matters, the case for a smaller tuned model gets much stronger. Run the numbers with your own prices, volumes and cache hit rate; reducing LLM costs covers the full method.

How do you decide which to use?

Start from the failure, not the technique. Build a test set and a baseline, try a strong prompt, then let the type of remaining failures choose the next step.

A few rules sharpen the chart:

• Facts the answer depends on, or that change monthly or faster: RAG.
• Answers need citations or per-user permissions: RAG.
• Correct facts are present but the output is in the wrong shape, style or label: improve the prompt, then fine-tune.
• Quality is fine but cost or latency isn’t, at high volume: fine-tune a smaller model, or distill a larger one; see model distillation.
• Only a handful of good examples and no way to get more: stay with prompting and RAG.
• The task itself keeps changing: stay with prompting until it stabilizes; a fine-tuned model freezes today’s definition.

A Survey of Techniques for Maximizing LLM PerformanceJohn Allard and Colin Jarvis, OpenAI · YouTube · 2023

How do you combine RAG and fine-tuning?

The two are additive, not exclusive, and the strongest domain systems often use both: retrieval supplies current facts, and fine-tuning teaches the model what to do with them.

The key is to fine-tune on examples that look like production, retrieved passages included. RAFT (Zhang et al., 2024) trains the model with a question, relevant documents and distractor documents, and teaches it to ignore the distractors and quote the passage that supports the answer, with chain-of-thought reasoning. It improved domain RAG performance on PubMed, HotpotQA and Gorilla. A simpler version works for many enterprise tasks:

1. Run your production retriever over real questions and keep the top-k passages, irrelevant ones included.
2. Write or correct the ideal answer for each, citing the passage it relies on, and include cases where the right answer is “not in the provided documents.”
3. Fine-tune on those full prompts, then evaluate with retrieval switched on.

Combining isn’t automatically better. In OpenAI’s accuracy guide, an Icelandic grammar-correction example scored worse after RAG was added to a fine-tuned GPT-4 model: accuracy dropped four points, from 87 to 83 BLEU, because the retrieved examples “confounded the model by giving it extra noise.” Test each combination; don’t assume.

For multi-step pipelines that chain prompting, retrieval and tuned models, see custom AI workflows.

What are the most common mistakes?

• Fine-tuning to inject facts. It learns them slowly, can raise hallucination and goes stale the day a document changes. Use retrieval.
• Using RAG to fix style or format. Retrieved example documents don’t reliably change how a model writes. Put examples in the prompt or train on them.
• Skipping the strong prompt. Fine-tuning compared against a weak prompt proves nothing. Your baseline is the best prompt you can write in a few days.
• Training on data that doesn’t match production. If production prompts include retrieved passages, training prompts must too.
• Judging retrieval by the final answer only. Measure whether the right passage was retrieved (recall at k) separately, or you will tune the wrong component.
• Freezing a moving task. A tuned model encodes today’s labels and policies. If they change monthly, the retraining cost never stops.
• Deciding from demos. Five good examples in a meeting say nothing about the long tail.

How do you test which approach wins on your data?

Treat it as an experiment with a fixed test set and a fair comparison.

1. Freeze a test set of a few hundred real, representative inputs from production traffic, with reference outputs or a scoring rubric. Split by document or customer so no near-copies appear in any training data. See building an evaluation framework.
2. Define the metric before running anything: exact match or field-level accuracy for extraction, rubric scores for generation, faithfulness to sources for RAG, plus cost per 1,000 requests and p95 latency.
3. Score the variants on the same inputs: current baseline, best prompt, prompt plus RAG, fine-tuned model, fine-tuned model plus RAG.
4. Compare in pairs, with uncertainty. Because every variant sees the same examples, a paired bootstrap gives an interval for the improvement:

import random


def paired_bootstrap(scores_a, scores_b, n_resamples=10_000, seed=0):
    """Mean improvement of B over A, with a 95% interval.

    scores_a and scores_b are per-example scores (for example 1 = correct,
    0 = wrong) for the same test examples, in the same order.
    """
    if len(scores_a) != len(scores_b):
        raise ValueError("score lists must cover the same examples")
    rng = random.Random(seed)
    n = len(scores_a)
    diffs = [b - a for a, b in zip(scores_a, scores_b)]
    means = sorted(
        sum(diffs[rng.randrange(n)] for _ in range(n)) / n
        for _ in range(n_resamples)
    )
    observed = sum(diffs) / n
    low = means[int(0.025 * n_resamples)]
    high = means[int(0.975 * n_resamples) - 1]
    return observed, (low, high)

If the interval for “fine-tuned minus best prompt” includes zero, you haven’t shown that fine-tuning helps. Collect more test cases or keep the simpler system.

5. Check for regressions on behavior you didn’t target: refusals, safety prompts, adjacent tasks.
6. Decide on all three numbers together: quality, cost and latency. A small quality gain that doubles cost may not be worth it; an equal-quality model at a third of the cost may be.

This baseline-first discipline is how Attempts approaches every engagement: measure what you run today, and recommend training only when it wins on your data. If you’d like help running the comparison, tell us about your task.