How to build domain-specific models from proprietary data

A domain-specific LLM is a general model adapted to perform one job in your field better than a general model can, using data only you have. In practice that means fine-tuning an open-weight model on examples drawn from your proprietary records, keeping changing facts in retrieval, and proving the result beats what you run today on a held-out test set.

The hard parts are not the training run. They are choosing the right task, getting the data into a clean, lawful, leak-free shape, and refusing to ship a model that hasn’t earned it. This playbook walks through each step in order.

What makes a good first task for a domain-specific model?

“A model that knows our business” is not a task. A good first task has three properties:

1. Narrow. One input type, one output type. “Turn a support ticket into a category, priority and summary in this JSON schema” is narrow. “Answer any question about our products” is not.
2. High-volume. Thousands to millions of calls a month. Volume repays the fixed cost of data work and evaluation, and it gives you a steady stream of real inputs for testing.
3. Measurable. You can say, for each output, whether it is right, or at least score it against a rubric that two reviewers apply consistently.

Three hypothetical examples, used throughout this guide:

• Example: an education publisher wants practice questions generated from a textbook section, tagged to its curriculum standards and returned in a fixed JSON schema. Editors have approved and rejected thousands of such items over the years.
• Example: a hiring platform wants job postings parsed into structured fields (title, seniority, skills, location, salary range) to power search. It has millions of postings and a few thousand human-corrected parses.
• Example: a research lab wants abstracts screened for a systematic review: include or exclude, with a one-line reason tied to the review’s criteria. It has past reviews with expert decisions.

Each is narrow, repeats at volume and has a ground truth. Note what the hiring example deliberately avoids: ranking or screening candidates. Systems used for recruitment or selection, including analysing applications and assessing candidates, are listed as high-risk under Annex III of the EU AI Act. If your task touches decisions about people, involve legal counsel before you collect a single example and test outcomes across groups.

How do you audit proprietary data before training on it?

Treat the audit as a gate, not paperwork. Answer four questions for every data source.

Do you have the right to use it for training? Check customer contracts, terms of service, privacy notices and consent records. Data collected “to provide the service” is not automatically licensed for model training, and data licensed from third parties often prohibits it. If you plan to generate synthetic examples with another model, check that model’s licence too: the Gemma Terms of Use covering Gemma 3 and earlier, for example, treat models trained on Gemma outputs or by distillation from Gemma as derivatives. The synthetic data guide covers this in depth.

What personal and confidential data does it contain? Language models can memorize and reproduce training text. Carlini et al. extracted hundreds of verbatim sequences from GPT-2’s training data, including names, phone numbers and email addresses, some of which appeared in only one training document. Remove or mask direct identifiers (names, emails, phone numbers, account and ID numbers, addresses) and consider quasi-identifiers such as rare job titles combined with small towns. NIST SP 800-188 is a practical reference on de-identification techniques and governance, and it warns that “not all tools that merely mask personal information provide sufficient functionality.” Regulated data, such as health records, has its own legal standards; follow them.

Who is allowed to see it? A model trained on data from many customers can surface one customer’s content to another. If your product is multi-tenant, either train only on data every user of the model may see, or train per tenant.

Is it representative? Compare the training pool with current production traffic: input lengths, languages, categories, edge cases. Five-year-old records may reflect old products, old policies and old formats.

How should you prepare and label the training data?

Supervised fine-tuning needs input-output pairs in the same shape the model will see in production. The most common format is JSON Lines with chat messages:

{"messages": [{"role": "system", "content": "Extract posting fields as JSON."}, {"role": "user", "content": "Senior Data Engineer, Austin TX (hybrid). 6+ years Spark and Airflow. $150k-$180k."}, {"role": "assistant", "content": "{\"title\": \"Data Engineer\", \"seniority\": \"senior\", \"skills\": [\"Spark\", \"Airflow\"], \"location\": \"Austin, TX\", \"work_mode\": \"hybrid\", \"salary_min\": 150000, \"salary_max\": 180000}"}], "metadata": {"document_id": "posting-48213"}}

Keep a metadata record per example (source document, customer, date) in the file or a parallel manifest. You will need it for splitting, auditing and deletion requests. Strip it before training if your trainer would otherwise read it.

Labeling rules that save weeks later:

• Write a labeling guide with definitions and worked edge cases, and revise it as disagreements surface.
• Measure agreement. Have two people label the same sample. If they disagree often, the task definition is the problem, and no model will fix it.
• Prefer corrected outputs to raw history. Logged human outputs contain shortcuts, typos and policy drift. In the education example, use editor-approved items only, and treat rejected items as a separate signal for later preference work.
• Match production exactly. Same system prompt, same input formatting, same retrieved context if the production system uses retrieval.
• Deduplicate. Repeated text increases memorization: Lee et al. found deduplicated models emit memorized text about ten times less often, and that train-test overlap affected over 4% of the validation sets of standard datasets.

How do you split data so the test set doesn’t leak?

Random row-level splits are the most common way domain models look better in the lab than in production. If three examples come from the same textbook chapter, the same customer account or the same template, and one lands in test while two land in training, the test is partly a memory check.

Split by the unit that would be new in production:

• By group: document, customer, author or template. The scikit-learn guidance is explicit: to know whether a model generalizes to unseen groups, “all the samples in the validation fold come from groups that are not represented at all in the paired training fold.”
• By time: train on older data, test on the most recent period. This catches drift in products, policies and language. The research lab should test on a review run after every review in its training set.
• Then check for copies across groups, because the same text often appears under different IDs.

import hashlib
import json
import re

import numpy as np
from sklearn.model_selection import GroupShuffleSplit

with open("examples.jsonl") as f:
    records = [json.loads(line) for line in f]

# Group by the source document, not by row: every example cut from
# the same document lands on the same side of the split.
groups = np.array([r["metadata"]["document_id"] for r in records])

# test_size is a share of groups (documents), not of rows.
splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(splitter.split(np.zeros(len(records)), groups=groups))
assert not set(groups[train_idx]) & set(groups[test_idx])


def fingerprint(record):
    """Hash of the normalized user turns, to catch copies across groups."""
    text = " ".join(m["content"] for m in record["messages"] if m["role"] == "user")
    text = re.sub(r"\s+", " ", text.lower()).strip()
    return hashlib.sha256(text.encode()).hexdigest()


train_prints = {fingerprint(records[i]) for i in train_idx}
leaks = [int(i) for i in test_idx if fingerprint(records[i]) in train_prints]
print(f"{len(leaks)} of {len(test_idx)} test examples repeat a training input")

Exact hashing catches only identical text. For near-duplicates (templated postings that differ by a city name), add a fuzzy method such as MinHash or embedding similarity with a threshold you review by hand. Carve a validation set out of the training side the same way, use it for every tuning decision, and freeze the test set: look at it only to make the final call.

Should you use continued pre-training, fine-tuning or RAG for domain knowledge?

This is where many domain projects go wrong. Each method changes something different.

Approach	What it changes	Use it for	What the evidence says
Continued pre-training	The model’s general distribution over domain text	Specialized vocabulary and style, given a large unlabeled corpus	Gains on in-domain tasks; needs far more data and compute; risk of forgetting
Supervised fine-tuning	How the model maps inputs to outputs	Format, labels, style, task skill, using provided context	Poor at adding new facts; unfamiliar facts can raise hallucination
Retrieval (RAG)	What the model sees at inference	Facts, fresh information, citations, per-user access	Consistently beat fine-tuning for knowledge injection

The evidence on knowledge is consistent:

• Ovadia et al. (EMNLP 2024) compared unsupervised fine-tuning on domain text with retrieval across knowledge-intensive tasks and found “RAG consistently outperforms it,” for both knowledge seen in pretraining and entirely new facts. They also observed that models “struggle to learn new factual information” this way, and that exposing a model to numerous variations of each fact during training may help.
• Gekhman et al. (EMNLP 2024) found fine-tuning examples that introduce new knowledge “are learned significantly slower” than familiar ones, and that as they are eventually learned, they “linearly increase the model’s tendency to hallucinate.” Their conclusion: models mostly acquire facts in pretraining, and fine-tuning teaches them to use what they know.
• Continued pre-training can help with domain language. Gururangan et al. (2020) showed a second phase of in-domain pretraining improved downstream tasks across biomedical, computer science, news and review domains. But at LLM scale it is expensive, and Biderman et al. (2024) found that in standard low-rank settings LoRA “substantially underperforms” full fine-tuning on code and math, including continued pretraining on 20B tokens, while forgetting less of the base model’s other skills.

The practical recipe for most enterprise tasks is retrieval for facts plus supervised fine-tuning for behavior, including fine-tuning the model to use retrieved passages well. In the research-lab example, the review criteria and current literature go in retrieval; the fine-tuning teaches the include/exclude judgment and the one-line reason format. Reserve continued pre-training for cases where you have a very large corpus in a language or notation the base model handles badly, and have shown that fine-tuning and retrieval are not enough. Fine-tuning vs RAG vs prompt engineering compares the options in more detail.

BloombergGPT: How We Built a 50 Billion Parameter Financial Language ModelDavid Rosenberg (Bloomberg) · TMLS · YouTube · 2023

How do you choose a base model?

Pick the smallest model that can reach your quality bar, then check the practical constraints:

• Licence. Confirm commercial use, attribution, naming and any model-as-a-service clauses for the exact version. The open-weight vs proprietary comparison has a verified licence table.
• Size and latency. Your p95 latency budget and expected traffic bound the model size you can serve affordably.
• Context length. Long documents, plus retrieved passages, plus the system prompt must fit with headroom.
• Language and notation. Check performance on your languages, and count tokens per typical document: a tokenizer that splits your domain terms into many pieces costs more and can perform worse.
• Instruct vs base. With a few thousand examples, starting from an instruction-tuned checkpoint is usually simpler, because it already follows chat formats.

For the training method, LoRA is the standard starting point. Thinking Machines’ “LoRA Without Regret” found LoRA matched full fine-tuning on typical post-training dataset sizes when applied to all weight matrices, especially the MLP layers, with an optimal learning rate about 10 times the full fine-tuning rate. LoRA adapters are also small, which makes versioning, rollback and per-tenant models practical. See the LoRA and QLoRA tutorial for code.

Why measure a baseline before training, and how do you iterate?

Before any training, score the strongest cheap alternatives on the frozen test set:

1. What you run today, prompt and model unchanged.
2. A better prompt with clear instructions and a handful of examples.
3. That prompt plus retrieval, if the task depends on documents.

If option 2 or 3 meets your bar, you may not need a domain model at all. If they fall short, you now know by how much and on which cases. The fine-tuned model must beat the best of them on the same test set, on the same metric, with regressions on related behavior counted as failures. Measuring fine-tuning improvement covers the statistics.

Then iterate on data, not hyperparameters:

• Read the failures on the validation set and cluster them (wrong field, wrong label, refused, format break).
• Fix the cause: unclear labeling rules, missing cases, contradictory examples.
• Add targeted examples for the largest clusters, retrain and re-score on validation.
• Touch the test set only for the final decision.

Who owns the weights, and what governance does a domain model need?

A model trained on your proprietary data is a derived asset of that data. Govern it like one.

• Ownership and portability. With an open-weight base, you can keep the fine-tuned weights or adapter, within the base licence’s terms. With hosted fine-tuning of a proprietary model, the tuned model typically lives only on the vendor’s platform and follows its lifecycle: OpenAI’s deprecations page states that fine-tuned models follow the shutdown schedule of their base models, and new fine-tuning jobs end for existing customers on January 6, 2027. Whoever trains for you, get export rights and deletion terms in writing.
• Lineage. Record which dataset version, base model, configuration and code produced each model version, and which source records each dataset contains. When a customer withdraws consent or asks for deletion, you need to know which models are affected; small adapters make retraining without that data feasible.
• Access control. Restrict who can download weights, since they may encode training data. Apply the same classification as the source data.
• Release checks. Before each release, run the held-out evaluation, safety and refusal tests, and extraction-style probes for memorized personal data.
• Documentation. Keep a short model card: intended use, out-of-scope uses, training data summary, evaluation results and known failure modes.

How do you deploy and monitor a domain-specific model?

Deploy behind an authenticated, private endpoint, and roll out gradually:

• Shadow first. Run the new model alongside the current system on live traffic without serving its outputs, and compare.
• Canary. Serve a small share of traffic, watch the metrics, then expand.
• Keep the baseline warm for instant rollback.

If you serve several domain adapters on one base model, open-source servers support this directly. vLLM, for example, loads multiple LoRA adapters with --enable-lora and --lora-modules:

# BASE_MODEL is the Hugging Face ID or local path of the base the adapters were trained on.
vllm serve "$BASE_MODEL" \
  --enable-lora \
  --lora-modules postings=/models/adapters/postings-v3 screening=/models/adapters/screening-v2 \
  --max-loras 2 \
  --max-lora-rank 32

Clients then select an adapter by name in the request’s model field. Set --max-lora-rank to the largest rank among your adapters, and confirm in vLLM’s supported-models list that LoRA works for your base architecture.

Monitor four things in production:

1. Validity rate: share of outputs that parse and pass schema or business-rule checks.
2. Sampled quality: a weekly human review of a random sample, scored with the same rubric as the test set.
3. Input drift: changes in length, language, category mix or new templates compared with training data.
4. Fresh evaluation: each month, label a new sample of recent traffic and re-score both the current model and the baseline. When the gap shrinks, refresh the data and retrain.

This end-to-end loop, from task choice through measured deployment, is what Attempts runs for clients on open-weight models, with the baseline comparison as the release gate. Whether you build it in-house or not, the order matters: task, data rights, clean splits, baseline, then training. To discuss a specific use case, start here.