The complete guide to LLM post-training in 2026

LLM post-training is the set of training stages applied after a model has been pretrained on raw text: supervised fine-tuning on demonstrations, preference optimization such as RLHF or DPO, reinforcement learning against checkable rewards, and distillation from a stronger model. Pretraining gives a model knowledge; post-training decides how it behaves, what format it answers in, and how well it performs one job.

This guide covers what each stage does, what data it needs, how parameter-efficient methods change the economics, and, most importantly, how to decide whether post-training is worth doing at all for your workload.

What is the difference between pre-training and post-training?

Pre-training is next-token prediction over trillions of tokens of mostly unlabeled text. It is where a model acquires vocabulary, facts, coding patterns and general reasoning ability. It costs millions of dollars and is done by a handful of labs.

Post-training starts from that checkpoint and uses far smaller, targeted datasets. (Continued pretraining on raw domain text sits between the two; it is closer to pretraining in data volume and, as discussed below, is where LoRA is weakest.) A base model that only continues text becomes an assistant that follows instructions, refuses unsafe requests, emits valid JSON, or classifies insurance claims the way your adjusters do.

The most useful mental model comes from the LIMA paper: its authors fine-tuned a 65B LLaMA model on just 1,000 curated examples, with no reinforcement learning, and concluded that “almost all knowledge in large language models is learned during pretraining.” Post-training mostly teaches a model which of its existing capabilities to use and how to present them.

That has a practical consequence. If your problem is that the model does not know something (this quarter’s pricing, a new regulation, your internal policies), post-training is usually the wrong tool. Retrieval is. If your problem is that the model knows enough but behaves inconsistently, ignores your format or is too expensive at the size you need, post-training is a candidate. The fine-tuning vs RAG vs prompt engineering comparison goes deeper on that split.

Deep Dive into LLMs like ChatGPTAndrej Karpathy · YouTube · 2025

What does the modern LLM post-training stack look like?

Open recipes published in the last two years converge on a similar sequence. Tulu 3, a fully open post-training recipe built on Llama 3.1 base models, uses supervised fine-tuning, then Direct Preference Optimization, then a stage it named Reinforcement Learning with Verifiable Rewards (RLVR). DeepSeek-R1 used four stages: SFT on a few thousand “cold-start” long chain-of-thought examples, reasoning-focused reinforcement learning, a second SFT round on about 800,000 samples (reasoning data built by rejection sampling from the RL checkpoint plus general non-reasoning data), and a final RL stage covering all scenarios, with rule-based rewards for reasoning and reward models for helpfulness and harmlessness. The same 800,000 samples were then used to distill smaller Qwen and Llama models.

Here is what each stage does, briefly. The SFT vs DPO vs RL deep dive covers the objectives and failure modes in detail.

Supervised fine-tuning (instruction tuning)

SFT trains the model to reproduce target responses given prompts, with the usual cross-entropy loss, normally computed only on the response tokens. The data is demonstrations: a prompt and the answer you want. It is the cheapest, most stable and most widely used stage, and for narrow enterprise tasks (extraction, classification, routing, fixed-format generation) it is often the only stage you need.

Preference optimization: RLHF and DPO

When “correct” is hard to write down but easy to judge, you collect comparisons instead: for one prompt, which of two responses is better. InstructGPT popularized the three-step recipe of SFT, a reward model trained on human rankings, and reinforcement learning (PPO) against that reward model. Its headline result: human raters preferred outputs from a 1.3B InstructGPT model over the 175B GPT-3, despite 100 times fewer parameters.

Direct Preference Optimization showed that the same KL-constrained objective can be optimized with a simple classification loss on preference pairs, with no separately trained reward model and no sampling during training. DPO and its relatives (IPO, KTO, ORPO, SimPO) are now a common way to use preference data, and DPO is the preference stage in open recipes such as Tulu 3.

Reinforcement learning with verifiable rewards and GRPO

For tasks where a program can check the answer (a math result, unit tests passing, a JSON object validating against a schema), you can skip human preferences and reward the model directly. DeepSeekMath introduced Group Relative Policy Optimization (GRPO), a PPO variant that drops the separate value (critic) model and instead estimates the baseline from the scores of a group of answers sampled for the same prompt. For reasoning tasks, DeepSeek-R1 used rule-based accuracy and format rewards and avoided neural reward models, which its authors observed “are susceptible to reward hacking during large-scale reinforcement learning.” RL against verifiable rewards with GRPO-family algorithms is behind many recent open reasoning models.

Distillation

Distillation trains a smaller student on outputs from a larger teacher. In practice for LLMs this usually means SFT on teacher-generated responses. The DeepSeek-R1 authors found their distilled 32B model “performs significantly better” than the same Qwen2.5-32B base trained with large-scale RL directly (72.6 vs 47.0 pass@1 on AIME 2024). For small models, distilling from a strong teacher is usually more efficient than running RL yourself. A newer variant, on-policy distillation, samples from the student and has the teacher score every token; Thinking Machines (2025) cites Qwen3 report figures of 74.4% on AIME 2024 at about 1,800 GPU-hours for on-policy distillation versus 67.6% at 17,920 GPU-hours for RL, both starting from a similar SFT checkpoint.

Should you use LoRA or full fine-tuning?

Full fine-tuning updates every weight. Parameter-efficient fine-tuning (PEFT) freezes the base model and trains a small number of new parameters. LoRA, the dominant method, adds trainable low-rank matrices next to existing weight matrices; the original paper reported about 10,000 times fewer trainable parameters than full fine-tuning GPT-3 175B, and no added inference latency once merged. QLoRA goes further by keeping the frozen base weights in 4-bit NF4 and training LoRA adapters in higher precision on top, which made fine-tuning a 65B model possible on a single 48GB GPU.

The quality question has a better answer than it did two years ago:

• Thinking Machines’ LoRA Without Regret (2025) found LoRA matched full fine-tuning on typical instruction-tuning and reasoning datasets, provided it is applied to all layers, especially the MLP layers. Attention-only LoRA underperformed. The best LoRA learning rate was consistently about 10 times the full fine-tuning rate. For policy-gradient RL, even rank 1 matched full fine-tuning.
• LoRA falls behind when the dataset carries more information than the adapter can hold. Biderman et al. found it “substantially underperforms” full fine-tuning on large code and math training runs, including 20B-token continued pretraining, but also that it “forgets less” of the base model’s other abilities.

	LoRA / QLoRA	Full fine-tuning
Trainable parameters	Small fraction of the model	All
GPU memory	Much lower; QLoRA lower still	Highest (weights, gradients, optimizer state)
Quality on typical post-training data	Matches full FT when applied to all layers	Reference
Large-scale continued pretraining	Falls behind	Better
Forgetting of base abilities	Less	More
Serving	Many adapters can share one base model	One full copy per variant

The serving column matters more than people expect. Inference servers such as vLLM can load several LoRA adapters on one base model and route requests to them by name, so ten task-specific variants do not mean ten GPU deployments. The LoRA and QLoRA tutorial has working training and serving code.

Why does data quality matter more than quantity?

LIMA’s 1,000 examples are the famous data point, but the principle shows up everywhere. A post-training dataset is a specification. Every inconsistent label, outdated policy or sloppy answer teaches the model that inconsistency is acceptable. For a narrow task, the questions that decide data quality are:

1. Is it representative? Sample from real production inputs, including the ugly ones: truncated emails, mixed languages, scanned documents, edge cases your current system fails on.
2. Are the targets right? Have domain experts review a sample. If two experts disagree on 20% of labels, you cannot reliably measure accuracy much above their agreement rate, and the model learns the disagreement too.
3. Is it consistent? One output schema, one tone, one policy for ambiguous cases. Write the policy down before labeling.
4. Is it deduplicated and separated from evaluation data? Near-duplicates inflate apparent accuracy and leak test answers into training.
5. Is it legally usable? Check consent, data processing terms and any licence terms on model-generated outputs.

Where synthetic data fits

When real examples are scarce, you can generate them: have a strong model write prompts and responses, rewrite seed examples into harder variants, or sample many answers and keep only those that pass a checker (rejection sampling). Synthetic data powers most open recipes today, including Tulu 3 and the R1 distillations. It also brings risks: copying a teacher’s style without its accuracy, losing rare cases, contaminating your evaluation set, and violating provider terms. The synthetic data guide covers generation and filtering with a working pipeline.

How do you know post-training worked?

Evaluation is not a final step; it is the gate every stage passes through. Build it before you train.

• A frozen held-out set drawn from the same distribution as production, never used for training or prompt tuning. A few hundred well-labeled examples is a reasonable start for a narrow task.
• A baseline to beat: your current prompt on your current model, measured on the same set with the same scorer. A fine-tuned model that scores 91% means nothing until you know the baseline scores 89% or 70%.
• Task metrics that map to business outcomes: exact-match field accuracy for extraction, schema-validity rate for structured output, pass rate for code, precision and recall at your operating threshold for classification. Use LLM judges only where no deterministic metric exists, and calibrate them against human labels.
• Regression checks: a small suite of adjacent behaviors you must not lose, such as refusal behavior, following instructions outside the training format, multilingual inputs and long inputs.
• Operational metrics: latency at your real input length, cost per 1,000 requests, failure and timeout rates.

How to measure fine-tuning improvement covers statistical significance, judge calibration and how to report results honestly.

What can go wrong: forgetting and regressions

Training on a narrow distribution moves the model toward it and away from everything else. This catastrophic forgetting shows up as a model that aces your extraction task but now fails at simple follow-up questions, ignores new instructions or answers in the training format when asked something else.

Safety behavior can erode too. Qi et al. showed that fine-tuning on as few as 10 adversarial examples broke a commercial model’s guardrails, and, more relevant to enterprise teams, that fine-tuning on ordinary benign datasets also degraded safety alignment, though less severely.

Mitigations, roughly in order of cost:

1. Use LoRA rather than full fine-tuning (it forgets less) and avoid unnecessary epochs.
2. Mix in a slice of general instruction data or examples of the behaviors you want to keep.
3. Keep the training format as close as possible to how the model will be called in production.
4. Run the regression suite on every checkpoint, not just the final one, and pick the checkpoint on held-out results rather than training loss.

When is post-training worth it for an enterprise?

Most teams should not start with training. Work through this order, measuring at each step:

1. Prompting. A clear instruction, a schema and a handful of examples on a capable model. This is the baseline.
2. Retrieval. If failures come from missing or changing knowledge, add retrieval before touching weights.
3. Workflow design. Decomposition, validation and retries often fix reliability problems more cheaply than training.
4. Post-training. Consider it when a measured gap remains and at least one of the conditions below holds.

Signal	Points toward post-training	Points away from it
Task stability	Same task and format for months	Requirements change weekly
Failure type	Format drift, inconsistent judgment, style	Missing or stale facts
Volume	High, steady request volume	A few thousand calls a month
Data	Hundreds to thousands of trusted examples	No labels and no experts to make them
Latency or cost pressure	Need a smaller model to match a larger one	Current cost and latency are acceptable
Control	Must run open weights in your own environment	No hosting or data-residency constraint

Cost is a common motivation, so do the arithmetic before training. Hypothetical example (every price and volume below is an assumption for illustration, not a quote): suppose 2 million requests a month at 1,500 input and 200 output tokens. Monthly token cost is requests × (input_tokens × input_price + output_tokens × output_price). If a frontier API costs $2 per million input tokens and $8 per million output tokens, that is 2M × (1,500 × $2 + 200 × $8) / 1M = $9,200 a month. If a fine-tuned 8B model served at an assumed effective $0.20 per million tokens (input and output combined) matches quality, the token bill is 2M × 1,700 × $0.20 / 1M = $680, but you must add the cost of serving capacity, data preparation, evaluation and periodic retraining. At 20,000 requests a month the same comparison is $92 against about $7, a saving of roughly $85 a month, which will not repay the work; at that volume a dedicated deployment would also cost more than the per-token rate implies. The cost reduction guide walks through the full model.

One 2026 development makes this decision more pressing for teams on closed models: OpenAI is winding down self-serve fine-tuning. Since May 7, 2026, organizations that had never run fine-tuning cannot create jobs; since July 2, 2026, neither can organizations with no fine-tuned-model inference in the previous 60 days; and active existing customers can create jobs only until January 6, 2027 (earlier where the base model shuts down first). Existing fine-tuned models keep serving until their base model is deprecated. Teams that relied on that platform need a migration plan, and open-weight models are one of the options.

What does a realistic end-to-end post-training project look like?

For a narrow enterprise task, a disciplined project looks like this:

1. Define the task and the metric. One sentence on what the model must do, and a metric a domain expert agrees with. Write the labeling policy for ambiguous cases.
2. Build the evaluation set first. Sample a few hundred real inputs, have experts label them, freeze them.
3. Measure the baseline. Your best prompt on your current model, and on one or two candidate open-weight models. Many projects end here, correctly, because prompting is good enough.
4. Assemble training data. Real examples first, synthetic augmentation for gaps, deduplicated and decontaminated against the evaluation set. Keep provenance for every record.
5. Train SFT with LoRA. Start with a modest rank on all linear layers, two or three epochs, and a learning-rate sweep. Save checkpoints.
6. Evaluate every checkpoint on the held-out set and the regression suite. Compare to the baseline with confidence intervals, not single numbers.
7. Iterate on data, not hyperparameters. Read the failures. Most improvements come from fixing labels, adding missing case types and removing contradictory examples.
8. Add preference optimization or RL only if a measured gap remains and you have the signal for it: pairs for DPO, a reliable checker for RLVR.
9. Deploy behind the same interface your application already calls, with request logging, a shadow or canary period, and a rollback path to the baseline.
10. Monitor and refresh. Track the production metric, sample outputs for review, and retrain when input distributions shift.

This is the process Attempts runs for clients: diagnosing the task, shaping the data, measuring a LoRA fine-tune against the client’s current baseline and recommending against training when prompting or retrieval is enough. If you want a second opinion on whether your workload is a fit, describe it here.