SFT vs DPO vs RL: understanding modern post-training methods

SFT vs DPO vs RL comes down to the training signal. Supervised fine-tuning (SFT) imitates demonstrations. Direct Preference Optimization (DPO) learns from pairs where one response is preferred over another. Reinforcement learning (RL) samples from the model and pushes up whatever earns reward, whether from a learned reward model (RLHF with PPO) or from a program that checks the answer (RLVR, usually with GRPO).

This article gives the exact objectives, the data each method needs, the compute and stability costs, the ways each fails, and a decision table. For the wider picture (data, LoRA, evaluation, when to train at all), start with the complete guide to LLM post-training.

How does supervised fine-tuning work?

SFT is maximum-likelihood training on demonstrations. Given a prompt x and target completion y, it minimizes the token-level negative log-likelihood:

L_SFT(θ) = − Σ_t log π_θ(y_t | x, y_<t)

Two details matter in practice:

• Loss on completions only. The prompt tokens are masked so the model is not trained to reproduce user inputs or system prompts. In Hugging Face TRL, prompt-completion datasets compute loss on the completion by default (completion_only_loss), and conversational datasets can restrict loss to assistant turns with assistant_only_loss=True, according to the TRL SFT docs.
• The model learns everything in the target. Its tone, its length, its errors. SFT has no notion of “this answer is worse than that one”; every demonstration is treated as correct.

Data: prompt and ideal response pairs, from experts, logs of good human work, or a stronger teacher model (distillation). Compute: one forward and backward pass per example, no sampling, no extra models. Stability: high; loss curves are easy to read.

SFT is where nearly every pipeline starts, and for narrow tasks such as extraction, classification, routing and fixed-format generation it is often where it should end. The LoRA and QLoRA tutorial shows a complete run.

How does RLHF with a reward model and PPO work?

InstructGPT established the classic three-step recipe: SFT on demonstrations, then a reward model trained on human rankings of model outputs, then reinforcement learning against that reward model.

Step 1: train a reward model. Labelers compare responses to the same prompt. A reward model r_φ(x, y) is trained so the preferred response y_w scores above the rejected y_l, using the Bradley-Terry loss:

L_RM = − log σ( r_φ(x, y_w) − r_φ(x, y_l) )

Step 2: optimize the policy. The policy maximizes reward while a KL penalty keeps it close to a frozen reference (usually the SFT model):

max_θ E_{y ~ π_θ(·|x)} [ r_φ(x, y) ] − β · KL( π_θ(·|x) ‖ π_ref(·|x) )

PPO optimizes this with a clipped policy-gradient objective. It needs per-token advantage estimates, which come from a separately trained value model (critic). A PPO run therefore keeps four networks in play: the policy, the reference, the reward model and the value model, and it generates fresh samples from the policy every iteration.

Why it worked: InstructGPT’s human raters preferred its 1.3B model over the 175B GPT-3. Why teams avoid it: it is expensive (online generation plus four models), has many interacting hyperparameters, and is sensitive to reward model quality.

How does DPO remove the reward model?

DPO starts from the same KL-regularized objective. That objective has a known closed-form optimum:

π*(y|x) = (1/Z(x)) · π_ref(y|x) · exp( r(x,y) / β )

Rearranged, any reward can be written in terms of its optimal policy:

r(x,y) = β · log( π*(y|x) / π_ref(y|x) ) + β · log Z(x)

Substituting this into the Bradley-Terry preference model, the intractable partition function Z(x) cancels because both responses share the same prompt. Writing the trainable policy π_θ in place of π* and maximizing the likelihood of the observed preferences leaves a loss on the policy directly:

L_DPO = − E [ log σ( β · [ log(π_θ(y_w|x)/π_ref(y_w|x)) − log(π_θ(y_l|x)/π_ref(y_l|x)) ] ) ]

The policy is the reward model, hence the paper’s subtitle. The quantity β · log(π_θ(y|x)/π_ref(y|x)) is the implicit reward, which TRL logs as rewards/chosen and rewards/rejected. In code the whole objective is a few lines:

import torch.nn.functional as F


def dpo_loss(policy_chosen_logps, policy_rejected_logps,
             ref_chosen_logps, ref_rejected_logps, beta=0.1):
    """Each input is the summed log-probability of a completion, shape (batch,)."""
    chosen_logratio = policy_chosen_logps - ref_chosen_logps
    rejected_logratio = policy_rejected_logps - ref_rejected_logps
    logits = beta * (chosen_logratio - rejected_logratio)
    loss = -F.logsigmoid(logits).mean()
    # Implicit rewards, useful for logging: beta * log(pi / pi_ref)
    chosen_reward = (beta * chosen_logratio).detach()
    rejected_reward = (beta * rejected_logratio).detach()
    return loss, chosen_reward, rejected_reward

What DPO removes: the reward model, the value model and sampling during training. Reference log-probabilities can even be precomputed once. What it keeps: the reference model and β, which controls how far the policy may drift (TRL’s default is 0.1). What it changes: DPO is offline. It learns only from the pairs you give it, so it cannot discover better responses than the ones in your data, and pairs generated by a very different model are off-distribution for your policy.

The DPO paper reported that it exceeded PPO-based RLHF at controlling sentiment and matched or improved response quality on summarization and single-turn dialogue, while being “substantially simpler to implement and train.”

DPO variants worth knowing

Method	What changes vs DPO	Data	Reference model
IPO	Squared loss regressing the log-ratio margin toward a fixed target, instead of log-sigmoid	Pairs	Yes
KTO	Prospect-theory utility on single responses labeled desirable or undesirable	Unpaired thumbs up/down	Yes
ORPO	SFT loss on the chosen response plus λ times a log-sigmoid loss on the log odds ratio of chosen vs rejected (odds from length-normalized likelihoods); one stage, no SFT warm-up	Pairs	No
SimPO	Implicit reward is the length-normalized average log-probability, plus a target margin	Pairs	No

Why the variants exist: IPO’s authors showed that when preferences in the data are deterministic (one response always wins), DPO’s optimum pushes the losing response’s probability to zero irrespective of the KL coefficient (β here, τ in the IPO paper), so “the strength of the KL-regularisation becomes weaker and weaker the more deterministic the preferences.” With finite data that is easy to hit, and overfitting follows. KTO matters because thumbs-up/down feedback is far easier to collect than pairs; its authors report it “matches or exceeds” preference-based methods from 1B to 30B. SimPO targets length: it reported beating DPO by up to 6.4 points on AlpacaEval 2 without making outputs much longer. In TRL most of these are a loss_type switch on DPOConfig (for example "ipo", or "sigmoid_norm" for SimPO-style normalization); KTO has its own trainer.

Direct Preference Optimization (DPO) explained: Bradley-Terry model, log probabilities, mathUmar Jamil · YouTube · 2024

How does RL with verifiable rewards and GRPO work?

When a program can check the answer, you do not need a learned reward model. Tulu 3 named this Reinforcement Learning with Verifiable Rewards (RLVR) and used it as the final stage after SFT and DPO. Typical verifiers: exact match on a math answer, unit tests on generated code, JSON schema validation, field-level accuracy against labeled extractions.

The algorithm most associated with RLVR is Group Relative Policy Optimization, introduced in DeepSeekMath. In the paper’s words, “GRPO foregoes the critic model, instead estimating the baseline from group scores.” For each prompt it samples G completions, scores them, and normalizes:

Â_i = ( r_i − mean(r_1..r_G) ) / std(r_1..r_G)

With outcome rewards, every token in completion i gets the same advantage Â_i. The policy update is PPO’s clipped objective with these advantages, averaged over each completion’s tokens and then over the group, and the KL to the reference is added directly to the loss rather than folded into the reward. What GRPO removes vs PPO: the value model and its training, and the per-token advantage estimation that depends on it. What it keeps: the policy, online sampling, the clipped ratio, a reward source and (if the KL term is on) the reference model. DeepSeekMath sampled 64 outputs per question.

import torch


def group_relative_advantages(rewards: torch.Tensor, eps: float = 1e-4) -> torch.Tensor:
    """rewards: shape (num_prompts, group_size), one scalar per sampled completion."""
    mean = rewards.mean(dim=1, keepdim=True)
    std = rewards.std(dim=1, keepdim=True)
    return (rewards - mean) / (std + eps)


# Example: two prompts, four samples each, binary correctness reward.
rewards = torch.tensor([[1.0, 0.0, 0.0, 1.0],
                        [1.0, 1.0, 1.0, 1.0]])
print(group_relative_advantages(rewards))
# Row 1: correct samples get positive advantage, wrong ones negative.
# Row 2: every sample is correct, so every advantage is 0 and the prompt
# contributes no learning signal.

The second row is the practical lesson: prompts the model always solves, or never solves, teach nothing. RLVR datasets need difficulty in the band where the model succeeds some of the time.

DeepSeek-R1 made the approach famous. R1-Zero applied GRPO directly to a base model with rule-based accuracy and format rewards, avoiding neural reward models for reasoning tasks because they “are susceptible to reward hacking during large-scale reinforcement learning.” Reasoning behaviors emerged, but outputs had poor readability and mixed languages, so the released R1 used four stages: SFT on a few thousand cold-start chain-of-thought examples; reasoning-focused RL with an added language-consistency reward; rejection sampling from that checkpoint plus non-reasoning data for a second SFT round (about 800,000 samples); and a final RL stage for all scenarios, with rule-based rewards on reasoning prompts and learned helpfulness and harmlessness reward models on general prompts.

Implementations have moved past the original formulation. Dr. GRPO identified an optimization bias in GRPO’s per-response length normalization that “artificially increases response length,” most of all for incorrect outputs, and argued that dividing by the group standard deviation adds a question-difficulty bias. As of TRL v1.17, GRPOTrainer defaults to loss_type="dapo" (token losses normalized by the number of active tokens in the batch) and beta=0.0 (no KL term, so no reference model is loaded), while still scaling rewards by the group standard deviation unless you set scale_rewards, per the TRL GRPO docs. Check which variant your library runs before comparing results with a paper.

A verifiable reward for a structured-extraction task, wired into TRL with LoRA:

import json

from datasets import load_dataset
from peft import LoraConfig
from trl import GRPOConfig, GRPOTrainer


def field_accuracy_reward(completions, expected, **kwargs):
    """Verifiable reward: share of expected JSON fields the model got exactly right.

    `expected` is a dataset column holding the target JSON as a string; TRL passes
    every non-prompt column to reward functions as a keyword argument.
    """
    rewards = []
    for completion, target_json in zip(completions, expected):
        text = completion[0]["content"]  # conversational dataset format
        target = json.loads(target_json)
        try:
            pred = json.loads(text)
        except json.JSONDecodeError:
            rewards.append(0.0)  # invalid JSON earns nothing
            continue
        if not isinstance(pred, dict):
            rewards.append(0.0)
            continue
        hits = sum(pred.get(key) == value for key, value in target.items())
        rewards.append(hits / max(len(target), 1))
    return rewards


# Each row: {"prompt": [{"role": "user", "content": "..."}], "expected": "{...}"}
dataset = load_dataset("json", data_files="extraction_train.jsonl", split="train")

trainer = GRPOTrainer(
    model="Qwen/Qwen3-0.6B",  # in practice, start from your SFT checkpoint
    reward_funcs=field_accuracy_reward,
    args=GRPOConfig(
        output_dir="grpo-extraction",
        num_generations=8,          # group size G
        per_device_train_batch_size=8,
        max_completion_length=256,
        learning_rate=1e-5,         # LoRA typically wants ~10x the full-FT rate
        beta=0.0,                   # TRL default: no KL term, no reference model
    ),
    train_dataset=dataset,
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear",
                           task_type="CAUSAL_LM"),
)
trainer.train()

LoRA is a reasonable fit for RL: Thinking Machines’ LoRA Without Regret found that for policy-gradient RL, LoRA matched full fine-tuning even at rank 1, because each episode carries little information.

[GRPO Explained] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language ModelsYannic Kilcher · YouTube · 2025

SFT vs DPO vs RL: what data, compute and stability does each need?

	SFT	DPO family	PPO (RLHF)	GRPO (RLVR)
Training signal	Target completions	Chosen/rejected pairs (or binary labels for KTO)	Learned reward model	Program-checked reward
Models in memory	Policy	Policy + reference (none for ORPO/SimPO)	Policy, reference, reward, value	Policy (+ reference if KL on)
Samples during training	No	No	Yes, every step	Yes, G per prompt
Relative cost	Lowest	Low	Highest	High (generation-bound)
Stability	High	High, can overfit pairs	Lowest, many hyperparameters	Medium, depends on reward and difficulty

Online methods are dominated by generation time, which is why RL trainers integrate fast inference engines such as vLLM. RL compute is also harder to plan: Khatri et al. ran over 400,000 GPU-hours of experiments to fit predictable compute-performance curves for RL and found that many recipe choices change efficiency rather than the final ceiling.

What are the failure modes of each method?

SFT: imitation without judgment. The model copies whatever is in the targets, including mistakes, inconsistency and verbosity. Training on a narrow set also causes forgetting of unrelated behavior. Fix it in the data.

Reward hacking. Any learned reward is a proxy. Gao et al. used a “gold” reward model to show that optimizing a proxy reward model eventually lowers true quality, in line with Goodhart’s law, with RL and best-of-n following different curves. Verifiable rewards narrow the gap but do not close it: a test suite with weak coverage, or a checker that accepts malformed output, will be exploited. Read samples, not only reward curves.

Length bias. Singhal et al. found that much of RLHF’s reward improvement came from longer responses, and that a reward based on length alone recovered most of RLHF’s gains over SFT. DPO inherits the bias from preference data in which longer answers tend to win; SimPO’s length normalization and length-controlled evaluation exist because of this. GRPO’s original normalization added its own length bias, as Dr. GRPO showed.

Mode collapse and lost diversity. Kirk et al. found RLHF “generalises better than SFT to new inputs” but “significantly reduces output diversity.” For classification that is fine. For brainstorming, creative writing, or generating synthetic data, it is a real cost.

DPO-specific: overfitting and falling chosen likelihood. With near-deterministic preferences the KL anchor weakens (see IPO above). TRL’s documentation notes that DPO in practice typically lowers the likelihood of rejected responses rather than raising that of chosen ones, so monitor logps/chosen as well as the reward margin.

RL-specific: no signal and instability. All-correct or all-wrong groups give zero advantage; overly hard data stalls training. Rising reward with flat held-out accuracy is the classic sign of a hacked reward.

Which method should you use?

Your situation	Use	Why
You can write or collect correct answers	SFT	Cheapest, most stable, directly teaches the target
Correctness is a judgment call; experts can compare two answers	DPO (or IPO) after SFT	Uses comparisons without a reward model
You only have thumbs-up/down feedback from production	KTO after SFT	Does not need paired responses
A program can check the output	GRPO-style RLVR after SFT	Optimizes the real metric; the model explores
You need a small model to match a large one	Distillation (SFT on teacher outputs), then optional RL	DeepSeek-R1 found distillation beat direct RL at 32B
You need reward-model RL at scale with online exploration	PPO with a reward model	Most flexible, most expensive

The distillation row has strong evidence behind it: the DeepSeek-R1 authors found that a 32B model distilled from R1 “performs significantly better” than the same Qwen2.5-32B base trained with large-scale RL (72.6 vs 47.0 pass@1 on AIME 2024). It is one comparison at one scale, so treat it as strong evidence rather than a law.

Why is the sequence usually SFT first, then preference optimization or RL?

Each later stage assumes the earlier one worked. DPO’s math assumes the reference policy assigns reasonable probability to the responses in your pairs, which an SFT model trained on similar data does and a raw base model often does not. RL needs the model to succeed sometimes; otherwise every group’s reward is identical and the gradient is zero. SFT puts the model in the region where those signals become informative.

The open recipes reflect this. Tulu 3 runs SFT, DPO, then RLVR. DeepSeek-R1 found that pure RL from a base model was possible but produced hard-to-read output, and added cold-start SFT. A practical enterprise sequence:

1. SFT on clean demonstrations until held-out metrics plateau.
2. Measure the remaining gap against your baseline with an honest evaluation.
3. If failures are about preference (tone, helpfulness, choosing between valid answers), collect pairs and run DPO or a variant.
4. If failures are about correctness that a program can check, build the verifier and run RLVR.
5. Re-run the regression suite after every stage, because each one can undo gains from the last.

Most enterprise tasks are won or lost at step 1, which is why Attempts currently offers supervised fine-tuning with LoRA only. If a gap remains that only preference data or a verifier can close, measure it before committing to the larger engineering effort of DPO or RL. If preference pairs or verifiable tasks need to be generated, the synthetic data guide covers how to do it without poisoning your evaluation.