How to measure whether fine-tuning actually improved your model

To measure whether fine-tuning improved your model, compare it with the strongest alternative you could ship without training, on a held-out test set that played no part in training or tuning decisions, scoring both systems on the same items. Then check that the paired difference is statistically reliable, that general skills and safety did not regress, and that cost and latency still fit.

Most “our fine-tune beat the base model by 20 points” claims fail at least one of those tests. This guide shows how to pass all of them, with the statistics written out and code you can run.

What should the fine-tuned model be compared against?

The untuned base model with a bare prompt is the weakest possible comparison. A fine-tune should earn its place against the options a team would otherwise ship:

Baseline	Why it matters
Current production system (prompt + model)	The thing you are proposing to replace. The decision is relative to it.
Stronger prompt on the same base model	Clear instructions, few-shot examples and output schema often close much of the gap for free.
Retrieval-augmented prompt	If failures come from missing knowledge, retrieval may fix them without training.
Larger or frontier model, prompted	Shows the quality ceiling and what the tuned model is trading for lower cost.

Give each baseline a fair effort: the same development set and a comparable amount of prompt iteration the tuned model’s data got. A tuned model that only beats a neglected prompt has not shown that training was needed. The comparison of fine-tuning, RAG and prompt engineering explains which failures each option fixes.

How do you keep the test set honest?

The test set must be held out from every decision that shapes the tuned model, not only from the training data:

• Training data. Deduplicate the test set against training examples on normalized text and near-duplicates, and split by customer, document or conversation so related items stay on one side.
• Hyperparameters and checkpoint selection. Choose learning rate, epochs and checkpoint on a separate validation set. Picking the best of ten checkpoints by test score inflates the result.
• Prompt and data iteration. If you looked at test failures and added similar examples to the training set, those items are no longer a test.
• Synthetic data generation. If a teacher model generated training data, make sure it never saw test inputs.

Freeze the test set before training, record its content hash, and report every result with that hash. If the test set was compromised, build a fresh one from newer traffic. The evaluation framework guide covers sampling, stratification and leakage in more depth.

Why must comparisons be paired?

Score every system on the same items and analyze the per-item difference, not two separate averages. Some items are hard for every model and some are easy; pairing cancels that shared difficulty out. Evan Miller calls this a “free” variance reduction when the two models’ scores are correlated, which they nearly always are.

For each item i, compute d_i = score_tuned(i) − score_baseline(i). The improvement is the mean of d, and its uncertainty comes from the spread of d. For pass/fail scores, d is −1, 0 or +1, and only the items where the systems disagree carry information.

Two more practices from the same paper:

• Cluster when items are related. Several questions about one document are not independent. Compute errors by cluster or the interval will be too narrow.
• Resample rather than lowering the temperature. At production temperature, generate several outputs per item and average the item’s score. Miller advises against lowering the temperature to make results look stable: it changes the model being measured, and it can shift the variance elsewhere or bias the estimate instead of removing the noise.

How do you test whether the improvement is real?

Two tools cover most cases: a paired bootstrap confidence interval for the size of the gain, and McNemar’s test when outcomes are pass/fail.

Paired bootstrap confidence interval

Resample items with replacement, keeping each item’s baseline and tuned scores together, recompute the mean difference each time, and take the 2.5th and 97.5th percentiles. It works for any per-item metric (pass/fail, rubric score, F1) and makes no assumption about the shape of the distribution, though it does assume items are independent: if several items come from one document or conversation, resample whole clusters instead of single items. If the interval excludes zero, the gain is unlikely to be noise; the interval’s width tells you how precisely you know it.

Bootstrapping Main Ideas!!!Josh Starmer · StatQuest · YouTube · 2021

McNemar’s test for binary outcomes

Arrange paired pass/fail results in a 2×2 table:

	Tuned passes	Tuned fails
Baseline passes	both right	baseline-only (b)
Baseline fails	tuned-only (c)	both wrong

Items both systems get right or wrong say nothing about which is better. McNemar’s test asks whether the discordant counts b and c differ by more than chance: under the null hypothesis each discordant item is equally likely to fall either way, so the exact version is a binomial test of c against b + c. Use the exact version when b + c is small; statsmodels defaults to it.

import numpy as np
from statsmodels.stats.contingency_tables import mcnemar

# One row per test item, same items for both systems. 1 = pass, 0 = fail.
baseline = np.array([1, 0, 1, 1, 0, 1, 0, 1])   # ...load per-item results for the baseline
tuned = np.array([1, 1, 1, 1, 0, 1, 1, 0])      # ...and for the fine-tuned model


def paired_bootstrap_ci(a, b, n_boot=10_000, alpha=0.05, seed=0):
    """CI for mean(b - a), resampling items so each pair stays together."""
    rng = np.random.default_rng(seed)
    diff = b - a
    n = len(diff)
    idx = rng.integers(0, n, size=(n_boot, n))
    boot_means = diff[idx].mean(axis=1)
    lo, hi = np.quantile(boot_means, [alpha / 2, 1 - alpha / 2])
    return diff.mean(), lo, hi


def mcnemar_test(a, b):
    """Exact McNemar test on the discordant pairs of two paired binary outcomes."""
    both = int(np.sum((a == 1) & (b == 1)))
    only_a = int(np.sum((a == 1) & (b == 0)))   # baseline right, tuned wrong
    only_b = int(np.sum((a == 0) & (b == 1)))   # tuned right, baseline wrong
    neither = int(np.sum((a == 0) & (b == 0)))
    table = [[both, only_a], [only_b, neither]]
    result = mcnemar(table, exact=True)          # binomial test on only_a vs only_b
    return only_a, only_b, result.pvalue


delta, lo, hi = paired_bootstrap_ci(baseline, tuned)
only_a, only_b, p = mcnemar_test(baseline, tuned)
print(f"improvement {delta:+.1%} (95% CI {lo:+.1%} to {hi:+.1%})")
print(f"baseline-only wins {only_a}, tuned-only wins {only_b}, McNemar p = {p:.3f}")

Report the interval first and the p-value second: “+4.8 points (95% CI +2.1 to +7.4)” tells a decision-maker far more than “p = 0.001”. When you compare several candidates or many slices, expect some false positives by chance; pre-register the primary comparison and treat the rest as exploratory, or correct for multiple comparisons. Dror et al. found significance testing in NLP papers “often ignored or misused” and give a protocol for choosing a test.

How many test examples do you need?

Decide this before training. Card et al. found underpowered comparisons common in NLP; on several GLUE tasks, test sets were too small for most comparisons against the state of the art to be adequately powered. An underpowered test makes real gains look like noise and makes lucky noise look like gains.

For paired pass/fail outcomes, let p_gain be the share of items only the tuned model gets right and p_loss the share only the baseline gets right. The net gain is δ = p_gain − p_loss, and the variance of the per-item difference is (p_gain + p_loss) − δ². At a 5% significance level and 80% power, the number of items needed is about:

n ≈ (1.96 + 0.84)² × [(p_gain + p_loss) − δ²] / δ²

This is the paired form of the sample-size formula in Miller’s paper. His worked example, which assumes a per-item difference variance of 1/9, needs about 969 items to detect a 3-point difference.

from math import ceil
from statistics import NormalDist


def items_needed(p_gain, p_loss, alpha=0.05, power=0.8):
    """Paired items needed to detect a net gain of (p_gain - p_loss).

    p_gain: share of items the new model gets right and the baseline gets wrong.
    p_loss: share the baseline gets right and the new model gets wrong.
    Uses n = (z_{alpha/2} + z_beta)^2 * Var(d) / delta^2 with d in {-1, 0, +1}.
    """
    z = NormalDist().inv_cdf
    delta = p_gain - p_loss
    var_d = (p_gain + p_loss) - delta ** 2
    return ceil((z(1 - alpha / 2) + z(power)) ** 2 * var_d / delta ** 2)


def minimum_detectable_effect(n, discordant_rate, alpha=0.05, power=0.8):
    """Approximate smallest net gain detectable with n paired items (small-effect approximation)."""
    z = NormalDist().inv_cdf
    return (z(1 - alpha / 2) + z(power)) * (discordant_rate / n) ** 0.5


print(items_needed(p_gain=0.08, p_loss=0.03))           # 5-point net gain, 11% of items flip
print(f"{minimum_detectable_effect(500, 0.11):.1%}")    # MDE with 500 items

Example: suppose a pilot on the validation set suggests 8% of items flip in the tuned model’s favor and 3% flip against it. Detecting that 5-point net gain needs 338 items: (2.80)² × (0.11 − 0.0025) / 0.05² ≈ 337.5, rounded up. The figure rests on several assumptions: a two-sided 5% significance level and 80% power; independent items (no clusters); the normal approximation, using the variance under the expected gain; and pilot flip rates that are close to the truth, which a small pilot rarely guarantees, so add a margin. The exact McNemar test is slightly conservative: computed exactly, 338 items give about 76% power, and roughly 370 are needed for 80%. Turned around, with 500 items and 11% discordance, the minimum detectable effect is about 4.2 points; a real 2-point gain would usually be missed. More discordance (models that disagree more) needs more items; a smaller gain needs many more, since n scales with 1/δ².

If the test set is too small for the gain that matters to the business, grow it, or accept that the decision rests on a weaker signal and say so.

How do you stop the judge from deciding the result?

For open-ended outputs you will often use an LLM judge, and judges have documented biases. Zheng et al. (2023) identified position, verbosity and self-enhancement bias. Wang et al. (2023) showed that with ChatGPT as the judge, simply reordering responses let Vicuna-13B beat ChatGPT on 66 of 80 queries. A tuned model that writes longer answers can “win” for the wrong reason.

Controls that belong in every fine-tuning comparison:

• Blind labels. The judge sees “Response 1” and “Response 2”, never model names or anything that identifies the source.
• Randomize and swap order. Judge each pair in both orders and count a win only if it survives the swap.
• Control for length. Report average output length per system and check whether wins track length; length-controlled AlpacaEval shows how to adjust for it statistically.
• Use a judge from a different family than the tuned model and, if the training data was distilled from a teacher, different from the teacher too.
• Calibrate against humans on a sample of the same pairs and report agreement before trusting the judge’s verdicts.

import random
from typing import Callable

# judge(prompt, first, second) returns "first", "second" or "tie"; it never sees model names.
Judge = Callable[[str, str, str], str]


def blind_pairwise(prompt: str, baseline_out: str, tuned_out: str, judge: Judge,
                   rng: random.Random) -> str:
    """Judge both orders in a random sequence; count a win only if it survives the swap."""
    orders = [("baseline", "tuned"), ("tuned", "baseline")]
    rng.shuffle(orders)
    outputs = {"baseline": baseline_out, "tuned": tuned_out}
    winners = []
    for first, second in orders:
        verdict = judge(prompt, outputs[first], outputs[second])
        winners.append({"first": first, "second": second}.get(verdict, "tie"))
    return winners[0] if winners[0] == winners[1] else "tie"

The per-item outcomes (tuned win, baseline win, tie) feed straight into the paired bootstrap above, coded as +1, −1 and 0.

Did fine-tuning break anything else?

A fine-tune is trained to be good at one distribution. What happens outside it has to be measured, not assumed.

General capabilities. Luo et al. observed catastrophic forgetting of domain knowledge, reasoning and reading comprehension during continual instruction tuning in models from 1B to 7B parameters. The method matters: Biderman et al. found LoRA “learns less and forgets less” than full fine-tuning, preserving more of the base model’s performance outside the target domain, though in their standard low-rank configurations on code and math it also underperformed full fine-tuning on the target. Thinking Machines’ LoRA Without Regret reports that LoRA applied to all layers, including the MLP layers, matches full fine-tuning on typical post-training datasets. The LoRA and QLoRA tutorial covers those settings.

Safety behavior. Qi et al. showed that fine-tuning an aligned model can weaken its safety guardrails: 10 adversarial training examples were enough to bypass GPT-3.5 Turbo’s guardrails, and fine-tuning on ordinary benign datasets also degraded safety, though to a lesser extent. A tuned model needs its own safety evaluation even if the base model passed one.

A practical regression suite has three parts:

1. Adjacent tasks from your own product: neighboring intents, other languages, other document types the model will also see.
2. Public capability checks relevant to your use, run identically on base and tuned models with a tool such as lm-evaluation-harness, which can load a LoRA adapter via peft= in --model_args. Here a public benchmark is fine, because you are measuring change, not absolute rank.
3. Safety and policy prompts: refusals that must still happen, instructions that must still be followed, and prompt-injection cases from your domain.

Set a non-inferiority margin for each: for example, “no more than 1 point worse, with the upper bound of the 95% interval for the drop below 2 points.”

Do cost and latency count as part of the result?

Yes. Often they are the reason to fine-tune: a smaller tuned model that matches a larger prompted model can be cheaper and faster per request. Measure them on the same test items, under the serving setup you will actually use:

• Cost per successful task, not per token: (input tokens × input price + output tokens × output price) / pass rate, plus the training and hosting cost amortized over expected volume. A cheaper model that fails more often may cost more per success once retries and human fixes are counted.
• Latency at p50 and p95, measured at realistic concurrency.
• Prompt length. Fine-tuning often lets you drop long instructions and few-shot examples, which shrinks input tokens on every request.

Example: suppose the current system costs $4.00 per 1,000 requests at an 86% pass rate, about $4.65 per 1,000 successes, and a tuned smaller model costs $1.20 per 1,000 requests at 91%, about $1.32 per 1,000 successes. Whether that saving pays for training, evaluation and hosting depends on your volume; the guide to reducing LLM costs works through that math.

What does a shipping decision rule look like?

Write the rule before you see the numbers, agree it with the business owner, and apply it mechanically. This removes the temptation to find a slice where the new model wins.

decision_rule: ticket-triage-finetune-v2
test_set: triage-test-2026-10 (sha256 recorded in run log), 820 items, never used in training or tuning
compare_against: production (current prompt + model) and best prompted baseline
primary_metric:
  name: category_accuracy
  ship_if: paired 95% bootstrap CI lower bound > +2.0 points vs BOTH baselines
guardrails:  # all must hold
  - schema_validity >= 99.5%
  - no slice with >= 50 items has a paired CI upper bound below -2.0 points
  - adjacent-task suite: drop <= 1.0 point, with the 95% CI upper bound of the drop < 2.0 points
  - safety suite: refusal rate on must-refuse prompts >= production
  - judge calibration on 150 human-labelled pairs reported before judge results are used
operations:
  - p95 latency <= 1.2 s at expected peak concurrency
  - cost per 1,000 successful tasks <= 70% of production
otherwise: do not ship; record the result and the failure analysis

What does a good results table look like?

The table below uses illustrative, hypothetical numbers to show the format, not results from any real system. Every row is scored on the same 820 frozen test items; intervals are paired against production.

System	Accuracy (95% CI vs prod)	Safety suite	p95 latency	Cost / 1k successes
Production prompt + model	86.1% (reference)	100%	1.6 s	$4.65
Improved prompt, same model	88.0% (+0.6 to +3.2)	100%	1.7 s	$4.90
Improved prompt + retrieval	88.4% (+0.9 to +3.7)	100%	2.1 s	$5.60
Fine-tuned smaller model (LoRA)	91.0% (+3.4 to +6.4)	99.4%	0.7 s	$1.32

Read against the decision rule above, and assuming its paired interval against the prompt-plus-retrieval baseline also clears +2.0 points (that comparison is not shown in the table), this hypothetical tuned model passes the accuracy, latency and cost gates but fails the safety guardrail, so it does not ship yet; the next step is to inspect the safety failures and retrain or add safety data. That is the point of a rule written in advance.

This is the discipline Attempts applies to every tuned model: measure it against the client’s current prompt and model on held-out data, and recommend against shipping, or against training at all, when the evidence does not support it.

Checklist before you call it an improvement

• The tuned model beat the current system and a fairly tuned prompt or retrieval baseline.
• The test set was frozen before training and never used for any tuning decision.
• All systems were scored on the same items, with paired intervals reported.
• The test set was large enough, by a power calculation done in advance, to detect the gain that matters.
• LLM-judge results were blinded, order-swapped, length-checked and calibrated against humans.
• General-capability and safety regressions were measured, with margins set in advance.
• Cost per successful task and p95 latency were measured on the real serving setup.
• The decision followed a rule written before the results were seen.