Fine-Tuning a Small Model on a Laptop with LoRA: Start to Finish

First, the honest framing: fine-tuning teaches style, format, and vocabulary — not facts. If you want the model to know your product documentation, use retrieval. If you want it to reply the way your team replies, fine-tune. I wasted a week learning this the wrong way round.

The dataset is 90% of the work

Related: Midjourney vs DALL-E vs Flux: A Practical Image-Tool Comparison Framework →

Related: Which GPU Should You Buy for Local AI in 2026? (I Tested Five Price Brackets) →

Format is boring JSONL:

{"messages":[{"role":"system","content":"You are a support agent for Acme."},{"role":"user","content":"My invoice shows the wrong VAT."},{"role":"assistant","content":"Thanks for flagging that..."}]}

Rules I now follow after getting it wrong:

• Consistency beats volume. 500 examples in one consistent voice beat 5,000 mixed ones. My first dataset mixed formal and casual replies and the model learned to be randomly inconsistent.
• Include the hard cases. If 5% of tickets are refund disputes, make sure the dataset contains them, or the model will confidently improvise on exactly the cases that matter.
• Strip real names, emails and order numbers. Models memorise. This is not optional.
• Hold back 10% for evaluation before you start, not after.

The training script

Related: 9 Free AI Coding Tools Every Developer Should Try in 2026 →

Related: MCP Servers, Explained by Someone Who Wired Five of Them Up →

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer, SFTConfig

base = "Qwen/Qwen3-3B-Instruct" tok = AutoTokenizer.from_pretrained(base)

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype="bfloat16", bnb_4bit_use_double_quant=True) model = AutoModelForCausalLM.from_pretrained(base, quantization_config=bnb, device_map="auto") model = prepare_model_for_kbit_training(model)

peft = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM", target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"])

ds = load_dataset("json", data_files={"train":"train.jsonl","test":"eval.jsonl"})

trainer = SFTTrainer( model=get_peft_model(model, peft), train_dataset=ds["train"], eval_dataset=ds["test"], args=SFTConfig(output_dir="out", num_train_epochs=3, per_device_train_batch_size=1, gradient_accumulation_steps=8, learning_rate=2e-4, lr_scheduler_type="cosine", warmup_ratio=0.03, logging_steps=10, eval_strategy="steps", eval_steps=50, save_steps=50, bf16=True, gradient_checkpointing=True, max_seq_length=1024), ) trainer.train() trainer.save_model("out/final")

The hyperparameters that mattered

Related: Sora 2 Review: OpenAI's Video Model Is Finally Useful for Real Work →

Related: Transcribing Audio Locally with Whisper: My Actual Workflow (No Uploads) →

Peak VRAM with these settings on a 3B model: about 9.5GB. A 7B model with the same config lands near 14GB — reduce max_seq_length to 512 if you are on 12GB.

Experience log: how I knew it went wrong

Related: How to Clone Your Voice with AI in 2026 (Free and Paid Options) →

Related: Self-Hosting n8n on a $6 VPS: Full Setup, Backups Included →

Run 1: loss dropped beautifully, output was garbage. Cause: my JSONL had the assistant turn included in the prompt formatting twice. The model learned to repeat the question. Always print three fully-formatted training samples before starting a run — print(tok.apply_chat_template(ds["train"][0]["messages"], tokenize=False)). Thirty seconds of checking, hours saved.

Run 2: training loss 0.11, eval loss climbing after step 120. Textbook overfitting. Fix: dropped from 6 epochs to 3 and added lora_dropout=0.05. If eval loss turns upward, stop; nothing later in the run will save it.

Run 3: the model answered every question in the support format, including "what is 2+2". Over-specialisation. This is expected and is fine if the model has one job. If you need general ability too, mix in 10–15% general instruction examples.

Evaluating without kidding yourself

Related: CUDA Out of Memory: 9 Fixes That Actually Worked on My GPU →

Loss numbers are not quality. I built a 40-question held-out set and scored answers on three things: correct format, correct tone, no invented policy. Base model scored 12/40 on format. Fine-tuned scored 37/40. That number is the one I reported, not the loss curve.

Using the result

Related: How I Cut My OpenAI API Bill by 71% Without Changing the Product →

# merge adapter into the base for deployment
python -c "
from peft import AutoPeftModelForCausalLM
m = AutoPeftModelForCausalLM.from_pretrained('out/final', device_map='cpu')
m.merge_and_unload().save_pretrained('merged')"

Then convert to GGUF if you want it in Ollama. The adapter file itself is about 40MB — small enough to version-control alongside the dataset that produced it, which I now do.

FAQ

Do I need thousands of examples? No. 300–1,000 high-quality, consistent examples covers most style-transfer tasks.

Can I fine-tune on CPU? Technically. Practically, no — expect days instead of minutes.

LoRA or full fine-tuning? LoRA, unless you have multiple high-memory GPUs and a specific reason. The quality gap for style tasks is small and the cost gap is enormous.

How do I add new facts? You do not, reliably. Use retrieval. This is the single most common misunderstanding about fine-tuning.