How to Fine-Tune an LLM on Your Custom Dataset

Fine-tuning adapts a pre-trained LLM to excel at your specific task. This guide walks through fine-tuning an open-source model using LoRA.

When to Fine-Tune

Fine-tuning is ideal when:

• You need consistent output formatting
• Your domain has specialized vocabulary
• You want to reduce prompt length and cost
• You're building a specialized product

Setup

pip install transformers datasets accelerate peft bitsandbytes

Data Preparation

Format your data as JSONL with "instruction" and "output" fields:

{"instruction": "What is prompt engineering?", "output": "Prompt engineering is the practice of designing inputs to guide AI model outputs."}
{"instruction": "Explain RAG", "output": "RAG (Retrieval-Augmented Generation) combines LLMs with external knowledge retrieval."}

Loading the Base Model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model

model_name = "microsoft/phi-2"  # or any open-source model

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

Configuring LoRA

lora_config = LoraConfig(
    r=8,  # rank
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)
print(f"Trainable parameters: {model.num_parameters(only_trainable=True):,}")

Training

from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(
    output_dir="./phi2-finetuned",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    num_train_epochs=3,
    logging_steps=10,
    save_strategy="epoch",
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer,
)

trainer.train()

Evaluation

Test your fine-tuned model against a held-out set:

• Compare outputs before and after fine-tuning
• Measure task-specific metrics
• Do human evaluation for quality

Inference

def generate_response(instruction):
    inputs = tokenizer(instruction, return_tensors="pt").to("cuda")
    outputs = model.generate(
        **inputs,
        max_new_tokens=200,
        temperature=0.7
    )
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

Production Deployment

• Export to GGUF format for llama.cpp
• Deploy using vLLM for production
• Use Ollama for local deployment
• Monitor for drift and quality degradation

When NOT to Fine-Tune

• If prompt engineering solves your problem
• If you need to change behaviors frequently
• If you don't have high-quality training data
• Start with RAG before fine-tuning

Our "Build with LLMs" programme covers fine-tuning with hands-on projects.