How to fine-tune an open-weight LLM with LoRA and QLoRA

To fine-tune an open-weight LLM with LoRA, you freeze the pretrained weights and train two small matrices per linear layer whose product is a low-rank update. QLoRA trains the same adapters while the frozen base sits in 4-bit precision, so it needs far less GPU memory. This tutorial walks through data, training, evaluation and serving with current TRL, PEFT and vLLM.

Before you start, make sure fine-tuning is the right tool. If the model lacks facts, retrieval usually beats training. If a better prompt closes the gap, take the prompt. Fine-tuning vs RAG vs prompt engineering covers that decision. LoRA pays off when you need a consistent behavior, format or skill on a narrow task, and you can measure it.

What do LoRA and QLoRA actually change?

A linear layer computes y = Wx. LoRA (Hu et al., 2021) keeps W frozen and learns an update ΔW = (α/r)·BA. B is d × r, A is r × k, and the rank r is much smaller than d or k. Only A and B receive gradients, so optimizer state shrinks with them. PEFT initializes B to zero, which means training starts exactly at the base model.

Three settings control the adapter:

• Rank (r) sets capacity. The trainable parameter count grows linearly with it.
• Alpha (lora_alpha) scales the update by α/r. According to LoRA Without Regret, that 1/r factor makes the best learning rate nearly independent of rank. The authors used α = 32 and could not improve on PEFT’s default initialization.
• Target modules set which layers get adapters. This setting matters more than most older tutorials admit.

Much LoRA advice written before 2025 says to adapt only the attention projections. In Thinking Machines’ 2025 experiments, attention-only LoRA underperformed MLP-only LoRA even at matched parameter count (rank 256 attention-only against rank 128 MLP-only). Adding attention on top of the MLPs gave no extra benefit. In PEFT, target_modules="all-linear" covers every linear layer except the output head. The same study found LoRA matched full fine-tuning’s sample efficiency and final performance on small-to-medium instruction and reasoning datasets. It fell behind only when the dataset exceeded the adapter’s capacity, and in some settings at large batch sizes.

QLoRA (Dettmers et al., 2023) adds three memory techniques to LoRA:

1. 4-bit NormalFloat (NF4) for the frozen base weights. NF4 is a data type designed for normally distributed weights.
2. Double quantization, which quantizes the quantization constants too. The Transformers docs put the extra saving at about 0.4 bits per parameter.
3. Paged optimizers, which handle memory spikes during training.

The adapters still train in higher precision, and the matrix multiplications run in your compute dtype (bf16 here). QLoRA does not shrink or improve the adapter. It lowers the memory the frozen base needs, and each step runs slower because weights are dequantized on the fly. The paper’s headline result was finetuning a 65B model on a single 48 GB GPU.

LoRA: Low-Rank Adaptation of Large Language Models - Explained visually + PyTorch code from scratchUmar Jamil · YouTube · 2023

How much GPU memory do LoRA and QLoRA need?

Memory has four parts: frozen weights, trainable parameters with their gradients and optimizer state, activations, and framework overhead. The first two are simple arithmetic. Here they are for the model this tutorial uses.

Example (estimates, not measurements). Qwen3-4B-Instruct-2507 has 4.0B parameters, 36 layers, hidden size 2,560 and tied embeddings over a 151,936-token vocabulary (from its model card and config.json). The figures below come from PEFT’s parameter count for this architecture plus bytes-per-parameter arithmetic.

Component	LoRA (bf16 base)	QLoRA (NF4 base)
Frozen weights	~8 GB (2 bytes × 4.0B)	~2.5–3 GB (linear layers at ~0.5 bytes, embeddings stay 16-bit)
Adapter, rank 32, all linear layers	66M params	66M params
Adapter weights, grads, Adam states (fp32, 16 bytes/param)	~1.1 GB	~0.7 GB with 8-bit paged Adam
Activations	grow with batch × sequence length; gradient checkpointing cuts them sharply	same

For comparison, mixed-precision full fine-tuning with Adam needs about 16 bytes per parameter before activations, according to the ZeRO paper. That is roughly 64 GB for the same 4B model. Rank 16 halves the adapter row (33M parameters) and rank 256 multiplies it by eight (528M).

Use QLoRA when the bf16 weights don’t fit, or barely fit, on your card. If LoRA fits, it is the faster choice.

How should you format the training data?

Use one JSON object per line with a messages list in the chat format the model was trained on. The last message is the target assistant reply:

{"messages": [{"role": "system", "content": "Classify the support ticket. Reply with JSON only."}, {"role": "user", "content": "I was charged twice for order 88412."}, {"role": "assistant", "content": "{\"category\": \"billing\", \"priority\": \"p2\"}"}]}

Most of the gains come from the data. The QLoRA authors found data quality far more important than size: a 9,000-example dataset (OASST1) beat a 450,000-example one (FLAN v2, subsampled) on chatbot performance. In practice:

• Draw examples from real traffic and cover the input variety you see in production, including the awkward cases.
• Write the assistant turns exactly as you want the model to answer. The model learns inconsistent formatting just as faithfully as consistent formatting.
• Deduplicate before splitting, so near-identical prompts can’t appear in both train and test.
• Freeze a test set and never train on it. It is the only honest before-and-after comparison you will get.

If you need to generate or augment examples, read synthetic data for fine-tuning first. Generated data has its own failure modes.

This script validates, deduplicates and splits data/all.jsonl with a fixed seed:

import hashlib
import json
import random
from pathlib import Path

SOURCE = Path("data/all.jsonl")
OUT = Path("data")
SEED = 42


def key(example):
    # Deduplicate on the user turns so near-identical prompts cannot leak across splits.
    user_text = "\n".join(m["content"] for m in example["messages"] if m["role"] == "user")
    return hashlib.sha256(user_text.strip().lower().encode()).hexdigest()


def valid(example):
    msgs = example.get("messages", [])
    return (
        len(msgs) >= 2
        and msgs[-1]["role"] == "assistant"
        and all(m["role"] in {"system", "user", "assistant"} and m["content"].strip() for m in msgs)
    )


rows, seen = [], set()
with SOURCE.open() as f:
    for line in f:
        ex = json.loads(line)
        if not valid(ex):
            continue
        k = key(ex)
        if k in seen:
            continue
        seen.add(k)
        rows.append(ex)

random.Random(SEED).shuffle(rows)
n = len(rows)
n_test = max(100, n // 10)
n_val = max(50, n // 20)
splits = {
    "test": rows[:n_test],
    "val": rows[n_test:n_test + n_val],
    "train": rows[n_test + n_val:],
}
for name, items in splits.items():
    with (OUT / f"{name}.jsonl").open("w") as f:
        for ex in items:
            f.write(json.dumps(ex, ensure_ascii=False) + "\n")
    print(name, len(items))

Train on the answers, not the prompts

By default, a conversational dataset is trained on every token, so the model also learns to reproduce your system prompt and user messages. Set assistant_only_loss=True to compute the loss only on assistant turns. It needs a chat template with {% generation %} markers. TRL ships patched templates for common families, including Qwen3-Instruct-2507, and swaps them in automatically (TRL chat templates). For a model outside that list, you have to add the markers to its template yourself.

If your data is in prompt/completion form instead ({"prompt": [...], "completion": [...]}), TRL computes the loss on the completion only by default (completion_only_loss).

Check token lengths before you pick max_length

SFTConfig.max_length defaults to 1,024 tokens, and anything longer is truncated. If truncation cuts off the whole assistant turn, TRL drops the example as fully masked. Check the length distribution first:

import json

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B-Instruct-2507")
lengths = []
with open("data/train.jsonl") as f:
    for line in f:
        msgs = json.loads(line)["messages"]
        ids = tok.apply_chat_template(msgs, tokenize=True, return_dict=True)["input_ids"]
        lengths.append(len(ids))

lengths.sort()
for q in (0.5, 0.9, 0.99, 1.0):
    print(f"p{int(q * 100)}: {lengths[min(len(lengths) - 1, int(q * len(lengths)))]} tokens")

In Transformers v5, apply_chat_template(..., tokenize=True) returns a dict-like encoding by default. Take input_ids from it rather than calling len() on the result.

Step 1: install a current stack

This tutorial uses Qwen/Qwen3-4B-Instruct-2507. It is Apache 2.0, text-only, non-thinking, and has a standard dense architecture that TRL, PEFT, bitsandbytes and vLLM all support, and it fits one 24 GB GPU. Newer options exist, such as Qwen3.5-4B, a vision-language model with a hybrid Gated DeltaNet architecture, and Gemma 4 12B, which is multimodal. Both can be fine-tuned, but their model classes and target modules need extra care, which would clutter a first run. The same script works for larger text models: change the model id and recheck the memory math.

python -m pip install "torch>=2.6" "transformers>=5.22" "trl>=1.17.1" "peft>=0.21" \
  "datasets>=4.7" "accelerate>=1.11" "bitsandbytes>=0.45.2"

These minimums are the versions this tutorial’s code was checked against in November 2026: trl 1.17.1, peft 0.21.2, transformers 5.22.2 and bitsandbytes 0.50.2. The training script was executed end to end on a tiny test model, but the full 4B training run was not executed. bitsandbytes also has Intel and CPU backends, but this tutorial assumes an NVIDIA GPU: NF4 and the 8-bit optimizers need Pascal or newer, and bf16=True needs Ampere or newer.

Step 2: write the training script

import argparse

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoTokenizer, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

MODEL_ID = "Qwen/Qwen3-4B-Instruct-2507"

parser = argparse.ArgumentParser()
parser.add_argument("--model", default=MODEL_ID)
parser.add_argument("--qlora", action="store_true", help="load the base model in 4-bit NF4")
parser.add_argument("--rank", type=int, default=32)
parser.add_argument("--lr", type=float, default=2e-4)
parser.add_argument("--epochs", type=float, default=2)
parser.add_argument("--output-dir", default="out/qwen3-4b-lora")
args = parser.parse_args()

data = load_dataset(
    "json",
    data_files={"train": "data/train.jsonl", "validation": "data/val.jsonl"},
)
tokenizer = AutoTokenizer.from_pretrained(args.model)

quantization_config = None
if args.qlora:
    quantization_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",          # 4-bit NormalFloat from the QLoRA paper
        bnb_4bit_use_double_quant=True,     # also quantize the quantization constants
        bnb_4bit_compute_dtype=torch.bfloat16,
    )

peft_config = LoraConfig(
    r=args.rank,
    lora_alpha=32,
    lora_dropout=0.0,
    target_modules="all-linear",  # every linear layer except the output head
    task_type="CAUSAL_LM",
)

training_args = SFTConfig(
    output_dir=args.output_dir,
    model_init_kwargs={"dtype": torch.bfloat16},
    # Data and loss
    assistant_only_loss=True,   # loss on assistant tokens only
    max_length=2048,            # check your token-length distribution first
    packing=False,
    # Optimization
    learning_rate=args.lr,
    lr_scheduler_type="cosine",
    warmup_steps=0.03,          # a float below 1 is a fraction of total steps
    num_train_epochs=args.epochs,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,   # effective batch 16
    gradient_checkpointing=True,
    optim="paged_adamw_8bit" if args.qlora else "adamw_torch_fused",
    bf16=True,
    # Evaluation and checkpoints
    eval_strategy="steps",
    eval_steps=50,
    save_strategy="steps",
    save_steps=50,
    save_total_limit=3,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
    logging_steps=10,
    report_to="none",
    seed=42,
)

trainer = SFTTrainer(
    model=args.model,
    args=training_args,
    train_dataset=data["train"],
    eval_dataset=data["validation"],
    processing_class=tokenizer,
    peft_config=peft_config,
    quantization_config=quantization_config,
)
trainer.model.print_trainable_parameters()
trainer.train()
trainer.save_model(args.output_dir)  # saves the adapter and tokenizer, not the base model

Run python train.py for LoRA, or python train.py --qlora for QLoRA. Several of these settings are there to override defaults that would otherwise hurt you:

• model_init_kwargs={"dtype": torch.bfloat16}. When you pass a model id as a string, SFTTrainer loads it in float32 unless you set a dtype (SFT Trainer docs). That doubles the weight memory.
• quantization_config passed to the trainer. Current TRL SFTTrainer accepts it directly and loads the quantized model for you. Combine it with peft_config for QLoRA.
• target_modules="all-linear". This applies the 2025 evidence. For Qwen3 it adapts q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj and down_proj in all 36 layers.
• packing=False. Packing improves throughput, but leave it off until the unpacked run works and you have confirmed your loss mask behaves as expected with it enabled.
• load_best_model_at_end on eval_loss. The final adapter comes from the best validation checkpoint, not the last step.

How do you choose LoRA hyperparameters?

Start from evidence-based defaults, then sweep the one parameter that matters most: the learning rate.

Setting	Starting point	Why
Learning rate	2e-4, then try 1e-4 and 4e-4	The best LoRA rate was about 10x full fine-tuning (fitted ratio 9.8 across 14 models on Tulu3; nearer 15x for very short runs). TRL documents 2e-4 for LoRA SFT.
Rank	16–64 for a narrow task	Low ranks run out of capacity on large, diverse datasets; raise the rank if the model underfits.
Alpha	32, fixed	With α/r scaling, the best learning rate barely moves between ranks 4 and 512.
Effective batch size	8–32	LoRA paid a larger penalty than full fine-tuning at large batch sizes in some settings, and higher rank didn’t fix it.
Epochs	1–3	Small datasets need more passes, but watch validation loss.
Target modules	all linear layers	Attention-only adapters underperform.

Signs of overfitting:

• Validation loss bottoms out and climbs while training loss keeps falling.
• Outputs start copying phrasings from training examples word for word.
• The model gets worse at things you didn’t train, such as following a different instruction format or answering general questions.

Keep the best checkpoint, not the last. Run a small regression set of general prompts alongside your task test set.

If training loss barely moves, check the learning rate and the loss mask first. If loss falls quickly to near zero, suspect duplicates, leakage, or a trivially easy target.

Step 3: evaluate the base model and the adapter on the same test set

Training loss measures how well the model fits your examples. It does not tell you whether the model is better at the task. Score both models on the frozen test set with the metric your application cares about, using the same prompts, the same decoding settings and the same scorer.

The simplest way is to serve both models from one vLLM process, as in the next section, and call each by name. This scorer checks a JSON-output task field by field:

import json
import sys

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="local-test")
MODEL = sys.argv[1]  # "Qwen/Qwen3-4B-Instruct-2507" for the baseline, "tickets" for the adapter
FIELDS = ("category", "priority")


def score(pred_text, gold_text):
    try:
        pred = json.loads(pred_text)
    except json.JSONDecodeError:
        return {"valid_json": 0, **{f: 0 for f in FIELDS}}
    gold = json.loads(gold_text)
    return {"valid_json": 1, **{f: int(pred.get(f) == gold.get(f)) for f in FIELDS}}


totals, n = {}, 0
with open("data/test.jsonl") as f:
    for line in f:
        ex = json.loads(line)
        prompt, gold = ex["messages"][:-1], ex["messages"][-1]["content"]
        reply = client.chat.completions.create(
            model=MODEL, messages=prompt, temperature=0, max_tokens=256
        ).choices[0].message.content
        for k, v in score(reply, gold).items():
            totals[k] = totals.get(k, 0) + v
        n += 1

print(MODEL, {k: round(v / n, 3) for k, v in totals.items()}, f"n={n}")

Run python evaluate.py Qwen/Qwen3-4B-Instruct-2507, then python evaluate.py tickets.

For a real decision, the baseline is whatever you run in production today, which may be a larger model with a carefully tuned prompt, not the untuned 4B model. On a test set of a few hundred examples, small differences fall within noise. How to measure whether fine-tuning actually improved your model covers confidence intervals and regression checks. For open-ended outputs that need rubric or judge-based scoring, see how to build a custom evaluation framework.

Step 4: should you merge the adapter or serve it?

	Merge into the base	Serve the adapter
Deployment	One standalone model	Base model plus small adapter files
Inference overhead	None, because the update is folded into W	Small per-request adapter compute
Many tasks on one base	One full copy per task	One base, many adapters
Rollback	Swap whole model weights	Swap or unload an adapter
Quantize for serving	Merge first, then quantize the merged model	Adapter runs on the served base

To merge, load the base in 16-bit, even if you trained with QLoRA, apply the adapter, and call merge_and_unload(). It returns a new model rather than modifying in place (PEFT reference).

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "Qwen/Qwen3-4B-Instruct-2507"
ADAPTER = "out/qwen3-4b-lora"
OUT = "out/qwen3-4b-tickets-merged"

# Merge into the 16-bit base, even if you trained with QLoRA.
base = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, ADAPTER)
model = model.merge_and_unload()   # returns a new model; not in place
model.save_pretrained(OUT)
AutoTokenizer.from_pretrained(BASE).save_pretrained(OUT)

A QLoRA adapter was trained against dequantized NF4 weights, so merging it into bf16 weights gives a slightly different model from the one you validated. Re-run the test set on whatever artifact you actually deploy.

Step 5: serve the adapter with vLLM

vLLM marks LoRA as supported for the Qwen3ForCausalLM architecture, and the model card asks for vllm>=0.8.5. Start the server with the adapter registered under a name:

vllm serve Qwen/Qwen3-4B-Instruct-2507 \
  --enable-lora \
  --lora-modules tickets=./out/qwen3-4b-lora \
  --max-lora-rank 32 \
  --max-model-len 8192

Set --max-lora-rank to the highest rank among the adapters you plan to load. vLLM’s docs warn that setting it far higher than needed wastes memory. The JSON form --lora-modules '{"name": "tickets", "path": "./out/qwen3-4b-lora", "base_model_name": "Qwen/Qwen3-4B-Instruct-2507"}' also records the base model. Clients select the adapter through the model field of the OpenAI-compatible API:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "tickets", "messages": [{"role": "user", "content": "I was charged twice for order 88412."}], "temperature": 0}'

Requests with "model": "Qwen/Qwen3-4B-Instruct-2507" still hit the untouched base, which is what makes the side-by-side evaluation in step 3 cheap.

vLLM can also load adapters at runtime through /v1/load_lora_adapter when VLLM_ALLOW_RUNTIME_LORA_UPDATING=True. Its documentation warns that this carries security risks and should not be used in production outside an isolated, fully trusted environment.

A serving endpoint also needs authentication, rate limits, logging that respects your data policy, and a rollback path. If you’d rather not run that stack yourself, Attempts runs this workflow on a client’s own data: it diagnoses the task, shapes the data, measures the tuned model against the current baseline, and deploys the approved model behind a private, authenticated API.

Troubleshooting LoRA and QLoRA fine-tuning

CUDA out of memory

• Lower per_device_train_batch_size and raise gradient_accumulation_steps to keep the same effective batch.
• Keep gradient_checkpointing=True (the TRL default) and confirm dtype is bf16, not the float32 default.
• Reduce max_length once you know your length distribution.
• Switch to --qlora. As a last resort, lower the rank.
• If memory spikes on a few long examples, filter outliers or use the paged 8-bit optimizer.

Chat template mismatch

The symptoms are outputs that never stop, outputs that repeat role markers, or a model that does well in training and badly in serving.

• Train and serve with the same template.
• Ship the base model’s tokenizer with a merged model.
• For base (non-instruct) checkpoints, set eos_token so it matches the template’s end-of-turn token, as the TRL docs describe for Qwen models.
• If assistant_only_loss=True raises an error about generation markers, your model’s template isn’t in TRL’s bundled list. Add the markers or switch to prompt/completion data.

Loss doesn’t decrease

• Check that the learning rate is in the LoRA range (around 1e-4 to 4e-4), not a full fine-tuning rate like 2e-5.
• Confirm that examples aren’t being dropped as fully masked. TRL filters them out in a dataset-preparation step labeled “Dropping fully masked examples”, so compare the dataset size before and after.
• Make sure the labels aren’t empty because of truncation.
• Check that you didn’t target too few modules.
• Decode one batch with the mask applied and read it. Many “the model won’t learn” bugs are data bugs.

Adapter doesn’t load in vLLM

• The adapter rank is larger than --max-lora-rank.
• The server is running a different base model from the one the adapter was trained on.
• The path is wrong, or adapter_config.json is missing.
• You trained modules the server can’t apply as LoRA, such as embeddings added through modules_to_save.

To confirm the adapter itself is sound, load it with PeftModel.from_pretrained locally and generate once.

Once the basic run works, the next steps are better data and better evaluation, not more hyperparameters. The post-training guide explains when supervised fine-tuning stops being enough and preference or reinforcement methods are worth their cost.