Synthetic data: how to generate training data for better models

Synthetic data for fine-tuning is training data produced by a language model instead of written by people: generated prompts, responses from a stronger teacher model, or many sampled answers filtered down to the ones a checker accepts. Done well, it fills coverage gaps cheaply and lets a small model learn from a large one. Done carelessly, it teaches confident mistakes and inflates your evaluation.

This guide covers the generation methods that have held up in research, how to verify and filter what you generate, the real risks (model collapse, imitation without accuracy, contamination, licences, privacy), how to mix synthetic with real data, and a working pipeline you can adapt. For where synthetic data sits in the overall training process, see the complete guide to LLM post-training.

When should you use synthetic training data?

Synthetic data earns its place in four situations:

1. Coverage gaps. You have 400 real examples, but only six involve refunds in a foreign currency. Generate more of the rare case.
2. Distillation. A large model already does the task well, and you want a smaller, cheaper model to match it. The large model’s outputs become training targets.
3. Verifiable tasks. When a program can check correctness (code, math, schema-valid extraction), you can generate many candidates and keep only correct ones.
4. Preference data. Generating two responses and having a judge or verifier pick the better one is a common way to build preference datasets for DPO (see SFT vs DPO vs RL).

It is the wrong tool when you have no real examples at all (you have nothing to anchor or evaluate against), when the teacher cannot do the task reliably either, or when the real problem is missing knowledge, which retrieval handles better.

How do you generate synthetic data?

Self-Instruct: bootstrap from a small seed set

Self-Instruct had a model generate new task instructions, inputs and outputs from a small pool of seed tasks, filtered out invalid or near-duplicate generations, and fine-tuned on the rest. Applied to the original GPT-3, it reported a 33% absolute improvement on Super-NaturalInstructions and came within 5% of InstructGPT-001 in human evaluation. The core pattern (seed examples in the prompt, ask for something new, filter similar outputs) is still the default starting point.

Evol-Instruct: make examples harder on purpose

Self-generated instructions drift toward easy, generic requests. Evol-Instruct, used to train WizardLM, rewrites seed instructions “step by step into more complex instructions”: adding constraints, requiring more reasoning steps, deepening the topic. Use the same idea for enterprise data: take a real ticket and ask for a variant with two issues, an ambiguous request, a missing account number or a hostile tone.

Persona and taxonomy seeding: buy diversity deliberately

Sampling the same prompt at high temperature produces surface variation, not real diversity. Two methods condition each generation on something different:

• Personas. Persona Hub built 1 billion personas from web data and conditioned each generation on one of them to push outputs toward different perspectives. In practice you need dozens to hundreds of personas that match your real users (a small-business owner on mobile, a procurement officer quoting contract clauses, a non-native speaker).
• Taxonomies. GLAN generated instruction data from a structured taxonomy of fields, subjects and concepts rather than from seed examples. For an enterprise task, the taxonomy is your own: product lines × issue types × customer segments × languages.

Distillation from a stronger teacher

The simplest generation method is to run your real prompts through a strong model and train on its answers. The risk is learning the teacher’s style without its substance. Gudibande et al. fine-tuned models on ChatGPT outputs and found crowd raters judged them competitive, while targeted evaluations showed they closed “little to none of the gap” on tasks the imitation data did not cover. They were “adept at mimicking ChatGPT’s style but not its factuality.” Orca improved on plain imitation by training on the teacher’s step-by-step explanation traces. The lesson for enterprises: distill on inputs drawn from your real distribution, and evaluate on tasks the distillation data does not cover. Model distillation covers the economics.

Rejection sampling: generate many, keep the correct ones

For tasks with a checker, sample several candidates per prompt and keep only those that pass. DeepSeek-R1 built SFT data this way from an RL checkpoint: “For each prompt, we sample multiple responses and retain only the correct ones.” Its distilled small models were then fine-tuned on 800,000 samples curated with R1. Rejection sampling is the most reliable way to make synthetic data correct rather than merely plausible, so build a verifier whenever the task allows one.

Best of 2024: Synthetic Data / Smol Models, Loubna Ben Allal, HuggingFace [LS Live! @ NeurIPS 2024]Loubna Ben Allal (Hugging Face) · Latent Space · YouTube · 2024

How do you verify and filter synthetic data?

Generation is cheap; filtering is where quality comes from. Apply these in order, cheapest first.

Filter	What it catches	How
Programmatic checks	Invalid JSON, out-of-schema labels, wrong answers, failing code	Parsers, schema validators, unit tests, answer checkers
Heuristics	Truncation, refusals, boilerplate, length outliers	Length bounds, refusal-phrase lists, language ID
LLM judge	Plausible but wrong or off-policy answers	Rubric-based grading, calibrated against human labels
Deduplication	Repeats and near-repeats	Exact hashing, n-gram or MinHash similarity, embedding similarity
Decontamination	Overlap with evaluation data	n-gram and embedding overlap against the frozen eval set
PII screening	Personal data in seeds or outputs	Pattern matching plus a dedicated detector such as Presidio

LLM judges have known biases. Zheng et al. found GPT-4 as a judge reached over 80% agreement with human preferences, about the level of agreement between humans, but also documented position bias, verbosity bias and self-enhancement bias (favoring outputs from itself or similar models). Do not use the same model as generator and sole judge, randomize answer order in pairwise comparisons, and measure judge agreement against a few hundred human labels before trusting it.

Deduplication matters more than it looks. Lee et al. found that deduplicating training data made models emit memorized text ten times less often and reach equal or better accuracy in fewer steps. Synthetic pipelines are especially prone to near-duplicates, because a generator asked for “a new example” keeps returning variations of its favorite one.

Decontamination protects your evaluation. If a generated training example is a paraphrase of a test item, your measured gain is partly fake. Tulu 3 decontaminated its open training datasets against benchmarks for this reason. For your own task, the “benchmark” is your frozen held-out set. The evaluation framework guide covers how to build that set.

What is model collapse, and should you worry about it?

Shumailov et al. (Nature, 2024) found that “indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear,” and named this model collapse. They demonstrated it in language models, variational autoencoders and Gaussian mixture models. In their language-model experiments, the first OPT-125m model was fine-tuned on wikitext2, and each later generation was fine-tuned on text the previous generation’s model generated as continuations of wikitext2 sequences. With no original data kept, performance degraded across generations; when each generation also sampled a random 10% of the original data, the authors report “only minor degradation,” though degradation was still present.

Gerstgrasser et al. sharpened the point: collapse follows when synthetic data replaces real data each generation, while accumulating synthetic data alongside the original real data avoided it across language models, diffusion models and VAEs. In their theoretical analysis, which uses linear models, accumulated data gives test error a finite upper bound independent of the number of iterations.

For a fine-tuning project, the practical reading is:

• Model collapse is a risk of recursive training on your own or other models’ outputs with real data discarded. One round of carefully filtered synthetic augmentation on top of real data is a different situation.
• The mechanism, lost tails, is still relevant in one round: generated data underrepresents rare cases. That is exactly where enterprise tasks fail. Generate rare cases deliberately, and verify them.
• Never retrain repeatedly on your deployed model’s own outputs without fresh real data and human review.

Can you legally train on model outputs?

This is a contract question first and a copyright question second. Read the terms that apply to the teacher you use, on the date you use it.

• Commercial APIs often restrict using outputs to build competing models. Anthropic’s commercial terms (effective June 17, 2025) say customers may not “access the Services to build a competing product or service, including to train competing AI models,” except as expressly approved by Anthropic. Other providers have their own wording; check whether your use case counts.
• Open-weight licences vary. The DeepSeek-R1 model card states the code and weights are MIT-licensed and that the series allows derivative works “including, but not limited to, distillation for training other LLMs.” The distilled R1 models are different: they keep their base models’ licences (Apache 2.0 for the Qwen2.5-based ones, Llama licences for the Llama-based ones). The Llama 3.1 licence allows using its outputs “to create, train, fine tune, or otherwise improve an AI model,” but any such model that is distributed or made available must include “Llama” at the beginning of its name, and licensees with more than 700 million monthly active users at the Llama 3.1 release date must request a separate licence from Meta.
• Record provenance. For every synthetic record, store the teacher model and version, the prompt template and the date. If a licence question arises later, you can remove affected records and retrain.

This is not legal advice; involve counsel for anything you will distribute or sell.

How do you handle privacy and PII in synthetic data?

“Synthetic” does not mean “anonymous.” Three ways personal data gets in:

1. Seeds. If real customer records are pasted into generation prompts, the generator can copy names, account numbers and addresses into outputs.
2. The teacher. Sending real records to a third-party API is a data transfer governed by your data processing agreement and privacy obligations. Using an open-weight teacher you host yourself avoids that transfer.
3. Memorization. Whatever enters training data can come back out of the fine-tuned model.

Scrub seeds before generation, instruct the generator to invent identifiers, and scan outputs. Tools such as Presidio detect and de-identify common PII types, but its own documentation warns “there is no guarantee that Presidio will find all sensitive information,” so pair automated detection with sampling and human review for regulated data.

How should you mix synthetic and real data?

There is no published ratio that transfers across tasks, so treat the mix as a hyperparameter you measure:

1. Keep all of your real data. It is the anchor and, per the collapse research, the safeguard.
2. Build the evaluation set from real data only, frozen before generation.
3. Train three or four variants: real only, and real plus synthetic at increasing amounts.
4. Pick the variant that wins on real held-out data, and check the per-slice results (the rare cases you generated for) as well as the average.
5. If synthetic data helps the slices it targeted but hurts the average, reduce it or improve its filtering. If it never beats real-only, drop it.

Tag every record with its source (real, synthetic, teacher name) so slices and removals are trivial later.

A practical synthetic data pipeline

The script below implements the core loop for an example task, classifying support tickets into a JSON label: generate new tickets seeded by real ones and conditioned on a persona, label them with a teacher, verify the label programmatically, screen for obvious PII, deduplicate, decontaminate against the frozen evaluation set, and write chat-format JSONL with provenance. The teacher is any OpenAI-compatible endpoint, such as an open-weight model served with vllm serve.

"""Minimal synthetic-data pipeline: generate -> verify -> dedupe -> decontaminate.

Task (example): classify support tickets into JSON {"category": ..., "priority": ...}.
The teacher is any OpenAI-compatible endpoint, for example an open-weight model
served locally with `vllm serve`. Check the teacher's licence before training on it.
"""
import hashlib
import json
import os
import random
import re

from openai import OpenAI

CATEGORIES = {"billing", "bug", "account_access", "feature_request", "other"}
PRIORITIES = {"low", "medium", "high"}
TEACHER = os.environ["TEACHER_MODEL"]
client = OpenAI(base_url=os.environ["TEACHER_BASE_URL"],
                api_key=os.environ.get("TEACHER_API_KEY", "EMPTY"))


def load_jsonl(path):
    with open(path) as f:
        return [json.loads(line) for line in f if line.strip()]


def chat(prompt, temperature):
    resp = client.chat.completions.create(
        model=TEACHER,
        messages=[{"role": "user", "content": prompt}],
        temperature=temperature,
    )
    return resp.choices[0].message.content.strip()


# 1. Generate: new inputs seeded by real examples and a persona for diversity.
def generate_ticket(seeds, persona):
    examples = "\n---\n".join(s["ticket"] for s in random.sample(seeds, k=3))
    return chat(
        f"Here are real support tickets:\n{examples}\n---\n"
        f"Write one new, realistic ticket from this customer: {persona}. "
        "Vary length, tone and detail. Use invented names and no real contact "
        "details. Output only the ticket text.",
        temperature=1.0,
    )


def label_ticket(ticket):
    return chat(
        "Classify the ticket. Reply with JSON only: "
        f'{{"category": one of {sorted(CATEGORIES)}, '
        f'"priority": one of {sorted(PRIORITIES)}}}\n\nTicket:\n{ticket}',
        temperature=0.0,
    )


# 2. Verify: reject anything a program can prove is wrong.
PII = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+|\+?(?:\d[\s().-]?){9,15}\d")  # emails, phone and card numbers


def verify(ticket, label_text):
    if not 20 <= len(ticket) <= 2000 or PII.search(ticket):
        return None
    try:
        label = json.loads(label_text)
    except json.JSONDecodeError:
        return None
    if not isinstance(label, dict):
        return None
    if label.get("category") not in CATEGORIES or label.get("priority") not in PRIORITIES:
        return None
    return label


# 3. Dedupe: exact hash, then near-duplicates by word 5-gram Jaccard similarity.
def shingles(text, n=5):
    words = re.findall(r"\w+", text.lower())
    return {" ".join(words[i:i + n]) for i in range(max(len(words) - n + 1, 1))}


def jaccard(a, b):
    return len(a & b) / max(len(a | b), 1)


def dedupe(records, threshold=0.7):
    kept, seen_hashes, kept_shingles = [], set(), []
    for rec in records:
        digest = hashlib.sha256(rec["ticket"].strip().lower().encode()).hexdigest()
        sh = shingles(rec["ticket"])
        if digest in seen_hashes or any(jaccard(sh, k) >= threshold for k in kept_shingles):
            continue  # O(n^2): fine for thousands; use MinHash LSH beyond that
        seen_hashes.add(digest)
        kept_shingles.append(sh)
        kept.append(rec)
    return kept


# 4. Decontaminate: drop anything that shares long n-grams with the eval set.
def decontaminate(records, eval_texts, n=8):
    eval_grams = set().union(*(shingles(t, n) for t in eval_texts))
    return [r for r in records if not (shingles(r["ticket"], n) & eval_grams)]


def main():
    seeds = load_jsonl("seed_tickets.jsonl")      # real, consented examples
    personas = load_jsonl("personas.jsonl")       # e.g. {"persona": "..."}
    eval_set = load_jsonl("eval_frozen.jsonl")    # never used for training
    out = []
    for i in range(int(os.environ.get("N_CANDIDATES", "500"))):
        persona = random.choice(personas)["persona"]
        ticket = generate_ticket(seeds, persona)
        label = verify(ticket, label_ticket(ticket))
        if label is None:
            continue
        out.append({"ticket": ticket, "label": label,
                    "provenance": {"teacher": TEACHER, "persona": persona, "i": i}})
    before = len(out)
    out = decontaminate(dedupe(out), [e["ticket"] for e in eval_set])
    print(f"kept {len(out)} of {before} verified candidates")
    with open("synthetic_train.jsonl", "w") as f:
        for rec in out:
            messages = [
                {"role": "user", "content": rec["ticket"]},
                {"role": "assistant", "content": json.dumps(rec["label"])},
            ]
            f.write(json.dumps({"messages": messages, "provenance": rec["provenance"]}) + "\n")


if __name__ == "__main__":
    main()

What to change for production:

• Scale the dedupe. The pairwise Jaccard check is quadratic. Past a few thousand records, use MinHash with locality-sensitive hashing (for example the datasketch library) or embedding similarity.
• Add semantic decontamination. Long n-gram overlap catches copies, not paraphrases. Add an embedding-similarity check against evaluation items with a conservative threshold.
• Strengthen PII detection. The regex is a crude first screen for emails and long digit strings; run a dedicated detector and sample outputs for review.
• Add a calibrated judge for label quality where no programmatic check exists, and have a domain expert review a random sample of every batch.
• Track the yield. If most candidates fail verification, fix the prompt or the teacher before generating more.

Then train on real plus synthetic, and compare it with a real-only model on the real held-out set. That comparison, not the size of the synthetic set, tells you whether the pipeline worked.