How to Fine-Tune an LLM: When It Pays Off, LoRA and QLoRA

Fine-tune an LLM when you need to change how a model behaves: a fixed output format, a house style, a narrow classification or extraction task, or the same quality from a smaller, cheaper and faster model. Do not fine-tune to add knowledge; facts that change or must be cited belong in a retrieval layer. With LoRA or QLoRA the training run itself is cheap, often hours on one GPU, so the real work is the dataset, an honest comparison against a well-prompted baseline, and operating the adapter after launch.

Key takeaways

• Try a strong prompt with examples, then structured outputs, then retrieval, before fine-tuning. Each step is cheaper to change than a trained model.
• Fine-tuning changes behavior, not knowledge. Use RAG for facts.
• A few hundred consistent, well-covered examples beat thousands of noisy ones. Split train and test by time or source, never by random row.
• LoRA and QLoRA make training affordable on a single GPU. Chat template mismatches and unmasked prompt tokens are the classic silent bugs.
• Ship only if the tuned model beats the best prompted baseline on task metrics, cost or latency. Then version the adapter, base model, data and evaluation together.

Fine-tune, prompt or retrieve?

Most requests to "fine-tune on our data" turn out to need retrieval or a better prompt. Use this table before you collect a single training example.

Structured outputs deserve a special mention. Many model APIs and inference servers can now constrain generation to a JSON Schema. That removes the most common historical reason to fine-tune, format compliance, without training anything.

The order we follow is simple: prompt with examples, constrain the output, add retrieval, then fine-tune, then distill to a smaller model if volume justifies it. Each step only happens when the evaluation shows the previous one has run out of room.

Kinds of fine-tuning

Where you train matters as much as how.

Build the dataset

Data quality decides the result. The training run is the easy part.

Where examples come from

• Production logs with human corrections. The best source: real inputs, outputs that a person fixed or approved.
• Expert-written examples for cases that are rare in logs but important.
• A larger model's outputs, reviewed. This is distillation. Review is not optional: unreviewed synthetic data bakes the teacher's mistakes into the student.

Format

Use the chat format your training library and inference server both understand, one JSON object per line, with the same system prompt you will use in production.

{"messages": [{"role": "system", "content": "Classify IT helpdesk tickets. Reply with JSON: category, priority, needs_human."}, {"role": "user", "content": "VPN drops every 30 minutes since this morning's update. I have to log in again each time."}, {"role": "assistant", "content": "{\"category\": \"network_vpn\", \"priority\": \"medium\", \"needs_human\": false}"}]}
{"messages": [{"role": "system", "content": "Classify IT helpdesk tickets. Reply with JSON: category, priority, needs_human."}, {"role": "user", "content": "Someone in finance got an email asking them to approve a wire transfer from the CEO's personal address."}, {"role": "assistant", "content": "{\"category\": \"security_incident\", \"priority\": \"high\", \"needs_human\": true}"}]}

Rules for clean data

• Consistency over cleverness. If two annotators would label the same input differently, the model learns the disagreement. Write labeling guidelines and resolve conflicts before training.
• Cover the hard cases. Ambiguous inputs, multi-category items, malformed inputs and "this needs a human" cases. A model never shown when to escalate will not escalate.
• Balance, within reason. If one class is most of the data, the model will over-predict it. Downsample the majority or upsample rare classes, and measure per-class results.
• Scrub sensitive data. Models can memorize training examples. Remove or mask personal data, secrets and anything you would not want reproduced verbatim.
• Split by time or source, not by random row. Near-duplicate tickets from the same customer or week leak across a random split and inflate test scores. Hold out the most recent period, or whole customers, and freeze that test set.

A validation script catches most data problems before they cost a training run:

"""validate_data.py: catch data problems before they cost a training run."""
import hashlib
import json
import sys
from collections import Counter
 
ALLOWED_ROLES = ("system", "user", "assistant")
seen, labels, problems = set(), Counter(), []
system_prompts = Counter()
 
for n, line in enumerate(open(sys.argv[1], encoding="utf-8"), start=1):
    example = json.loads(line)
    msgs = example["messages"]
    roles = [m["role"] for m in msgs]
    if any(r not in ALLOWED_ROLES for r in roles) or roles[-1] != "assistant":
        problems.append(f"line {n}: bad roles {roles}")
        continue
    system_prompts[msgs[0]["content"] if roles[0] == "system" else ""] += 1
    key = hashlib.sha256(" ".join(m["content"] for m in msgs[:-1]).lower().encode()).hexdigest()
    if key in seen:
        problems.append(f"line {n}: duplicate input")
    seen.add(key)
    try:
        output = json.loads(msgs[-1]["content"])
        labels[output["category"]] += 1
    except (json.JSONDecodeError, KeyError) as exc:
        problems.append(f"line {n}: assistant output not valid JSON with category ({exc})")
 
print(f"{len(seen)} unique inputs, {len(system_prompts)} distinct system prompts")
print("label distribution:", labels.most_common())
print("\n".join(problems[:50]) or "no problems found")

More than one distinct system prompt is usually a mistake. A heavily skewed label distribution is a decision you should make on purpose, not discover after training.

Train with LoRA and QLoRA

Full fine-tuning updates every weight, which means holding weights, gradients and optimizer states in GPU memory and risks degrading general abilities. LoRA (low-rank adaptation) freezes the base model and learns a small pair of matrices for selected layers whose product is added to the frozen weights, scaled by alpha / r. Only those small matrices train, and the saved adapter is a small fraction of the base model's size.

QLoRA goes further: it stores the frozen base model in 4-bit NF4 precision and trains LoRA adapters on top. The QLoRA paper demonstrated fine-tuning a 65-billion-parameter model on a single 48 GB GPU. Models in the 7 to 8 billion parameter range, which handle many narrow enterprise tasks well once tuned, fit on a single 24 GB card with moderate sequence lengths.

Here is a QLoRA supervised fine-tuning run with Hugging Face TRL and PEFT. These libraries rename arguments between releases, so pin exact versions in your requirements file and check the docs for the version you install.

"""train_lora.py: QLoRA supervised fine-tuning with TRL and PEFT.
pip install transformers trl peft bitsandbytes datasets accelerate (pin versions)
"""
import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer
 
BASE_MODEL = "your-org/open-weight-instruct-model"  # check the license for commercial use
 
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=True,
        bnb_4bit_compute_dtype=torch.bfloat16,
    ),
    device_map="auto",
)
 
peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",
    task_type="CAUSAL_LM",
)
 
data = load_dataset("json", data_files={"train": "train.jsonl", "validation": "val.jsonl"})
 
args = SFTConfig(
    output_dir="adapters/ticket-triage-v1",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_steps=0.03,  # Transformers 5 reads a float below 1 as a fraction of total steps
    max_length=2048,
    bf16=True,
    gradient_checkpointing=True,
    eval_strategy="steps",
    eval_steps=50,
    save_steps=50,
    load_best_model_at_end=True,
    logging_steps=10,
    report_to="none",
)
 
trainer = SFTTrainer(
    model=model,
    args=args,
    train_dataset=data["train"],
    eval_dataset=data["validation"],
    processing_class=tokenizer,
    peft_config=peft_config,
)
trainer.train()
trainer.save_model(args.output_dir)  # writes the adapter, not a full model copy

TRL applies the tokenizer's chat template to messages data, which keeps the training format identical to inference as long as you serve with the same template. Also check that the loss is computed only on the assistant's tokens; training on the prompt tokens too wastes capacity and can teach the model to echo inputs. Recent TRL versions expose an option for this, and its exact name depends on your version.

Starting hyperparameters

Loss curves only tell you training worked mechanically. They say nothing about whether the model is good at your task. That is the next section.

Evaluate against a real baseline

The question is never "is the fine-tuned model good?" It is "is it better than the cheapest alternative?" Run the same frozen test set through three candidates:

1. The base model with your best prompt, including examples.
2. A larger model with the same prompt.
3. The fine-tuned model.

If the fine-tuned model does not clearly win on task metrics, or match them at a much lower cost or latency, do not ship it. You would be taking on training, serving and retraining work for nothing.

What to measure:

• Task metrics per class or field: precision, recall and F1 for classification, exact match per field for extraction, schema validity rate for structured output.
• Human-rated samples for generation quality, scored against a written rubric by people who know the domain.
• Regression checks on general abilities your application relies on, such as following instructions in a different format or handling another language.
• Safety and abuse cases: prompt injection attempts, requests the model should refuse, inputs that must escalate.

"""eval_triage.py: compare candidates on the same frozen test set."""
import json
from sklearn.metrics import classification_report
 
 
def evaluate(generate, test_path: str) -> dict:
    """generate(messages) returns the model's text for the conversation without its final answer."""
    y_true, y_pred, invalid = [], [], 0
    with open(test_path, encoding="utf-8") as f:
        for line in f:
            msgs = json.loads(line)["messages"]
            expected = json.loads(msgs[-1]["content"])["category"]
            try:
                predicted = json.loads(generate(msgs[:-1]))["category"]
            except (json.JSONDecodeError, KeyError, TypeError):
                predicted, invalid = "INVALID", invalid + 1
            y_true.append(expected)
            y_pred.append(predicted)
    print(classification_report(y_true, y_pred, zero_division=0))
    return {"invalid_rate": invalid / len(y_true), "n": len(y_true)}

Run it once per candidate and keep the reports with the adapter. The same harness becomes your regression test for every future retrain.

Serve and operate the model

Merge or serve adapters

For a single use case, merge the adapter into the base weights and serve the result like any model: no runtime overhead. For several tuned variants on the same base, serve the base once and load adapters per request. vLLM supports this directly:

vllm serve your-org/open-weight-instruct-model \
  --enable-lora \
  --lora-modules triage=./adapters/ticket-triage-v3 summarizer=./adapters/summaries-v1

Requests then name the adapter as the model. This is how one GPU pool serves many tasks, or many tenants, without one deployment per adapter. If you serve per-tenant adapters, treat the adapter name as an authorization decision, exactly like a multi-tenant data boundary.

Version everything together

An adapter is meaningless without the exact base model revision it was trained on. Treat these as one release unit, stored in a model registry:

• Base model ID and revision
• Adapter weights
• Hash of the training and validation data
• Training configuration and library versions
• Evaluation reports against the baseline

When production quality drops, this is what lets you answer "what changed?" in minutes. The general practice of versioning, registries and deployment pipelines for models is covered in our MLOps guide.

Keep it healthy

• Sample production traffic into a review queue every week and score it with the same harness.
• Watch for drift: new ticket categories, new products, new phrasing. Rising "needs human" rates or falling confidence are early signals.
• Define retrain triggers in advance: a metric threshold, a new category, a base model upgrade.
• Shadow before switching. Run a new adapter alongside the current one on live traffic and compare before routing users to it.
• Plan for deletion requests. If training data included personal data and someone asks for deletion, you may need to retrain without it. Keep lineage from every example to its source.

What goes wrong in production

• Chat template mismatch. The model was trained with one prompt format and is served with another, so outputs degrade in ways that look like a bad model. Serve with the tokenizer's own chat template and test one training example end to end through the production stack.
• Prompt tokens in the loss. The model learns to reproduce inputs and wastes capacity. Train on assistant tokens only.
• Leaky test sets. Random splits put near-duplicates in train and test, offline scores look excellent, and production disappoints. Split by time or source.
• Format overfitting. The model nails the training distribution and breaks on anything new. Include varied and malformed inputs, and keep a held-out set of genuinely new cases.
• Forgetting. Aggressive training erodes general abilities the application quietly depended on. Keep regression checks and prefer lower learning rates and fewer epochs.
• Wrong base revision. An adapter loaded onto a slightly different base model produces fluent nonsense. Pin and verify the base revision at load time.
• Generation that never stops. A missing or misconfigured end-of-sequence or padding token in training leads to rambling output. Check that examples end with the end-of-turn token your template expects.
• Orphaned models. The person who trained it leaves, the hosted base model is deprecated, and nobody can retrain. Script the whole pipeline from data to evaluation and run it in CI.

Fine-tuning is also not the only way to get small, fast models for language tasks. For many classification and extraction jobs, an embedding model plus a light classifier is cheaper still; see our guide to NLP with transformers in business.

Frequently Asked Questions

When should you fine-tune an LLM instead of using RAG or prompting?

Fine-tune when you need to change how the model behaves: a fixed output format, a house style, a narrow classification or extraction task, or the same quality from a smaller and cheaper model. Use RAG when the model needs knowledge it lacks, especially knowledge that changes or must be cited. Always try a well-written prompt with examples first, because it is the baseline the fine-tuned model has to beat.

How much data do you need to fine-tune an LLM?

For a narrow task such as classification, extraction or a fixed format, a few hundred clean, consistent examples is often enough to see whether fine-tuning helps. Broader generation tasks usually need low thousands. Coverage of real cases and consistency between examples matter far more than raw volume.

What is the difference between LoRA and QLoRA?

LoRA freezes the base model and trains small low-rank matrices added to selected layers, so only a tiny fraction of parameters is updated. QLoRA does the same but stores the frozen base model in 4-bit precision, which cuts GPU memory enough to tune much larger models on a single card. Quality is usually close, and QLoRA trades some training speed for the memory savings.

How much GPU memory do you need to fine-tune an LLM?

It depends on model size, sequence length, batch size and method. Full fine-tuning has to hold weights, gradients and optimizer states, which is several times the model size, while LoRA and QLoRA only train small adapters. The QLoRA paper fine-tuned a 65-billion-parameter model on a single 48 GB GPU, and models in the 7 to 8 billion parameter range fit on a single 24 GB card with QLoRA and moderate sequence lengths.

Does fine-tuning reduce hallucinations?

Not reliably for facts. Fine-tuning teaches behavior and format, and it can teach a model to decline or escalate on certain inputs, but it is a poor way to add knowledge, and research has found that tuning on facts the base model did not already know can make hallucination worse. Put facts in a retrieval layer and use fine-tuning for how the model responds.

Can you fine-tune a hosted model through an API?

Several model providers offer hosted fine-tuning for some of their models, and you get a private endpoint rather than the weights. It removes the training and serving work, but your tuned model is tied to that provider's base model lifecycle and data terms. Open-weight models give you the weights and full control in exchange for running the infrastructure yourself.

Need help deciding whether to fine-tune?

We run this decision with data, not opinions: a baseline, a frozen test set, and a clear bar the tuned model has to clear. Senior engineers build the dataset pipeline, the training and evaluation harness, and the serving setup, and stay on the pager after launch. If we built it, we're on the pager.

• AI and machine learning services
• Talk to an engineer