How to Fine-Tune an LLM with LoRA (Beginner Guide)

Prompting tells a model what to do. Fine-tuning changes what the model is. When a general-purpose LLM does not reliably write in your company's style, answer in a specific domain, or follow a strict output format, fine-tuning is how you bake those behaviors in permanently rather than fighting them with longer and longer system prompts.

• A GPU with at least 16 GB VRAM for a serious fine-tune (RTX 3090, 4090, or an A10G cloud instance)
• Python 3.10+ with pip and a working CUDA environment
• A dataset of at least 100 examples in JSONL format (more on this below)
• Unsloth installed — it cuts VRAM usage by roughly half and speeds training 2x

Fine-tuning vs prompting: when to use which

Prompting is fast to iterate but pays a cost every inference: the instruction repeats in every request, burning tokens and sometimes getting ignored. Fine-tuning compiles behavior into the weights so the model already knows what you want. Use prompting when you need flexibility. Use fine-tuning when you need consistency at scale, a specific domain voice, or a strict structured output format the base model keeps breaking.

LoRA vs QLoRA: the practical difference

Full fine-tuning updates every weight in the model. For a 7B-parameter model that is gigabytes of gradient data per step. LoRA (Low-Rank Adaptation) adds small trainable adapter matrices on top of frozen base weights. You train only the adapters, which are a fraction of the size. QLoRA goes further by also quantizing the base model to 4-bit first, cutting VRAM usage dramatically. The trade-off is a slight quality loss from quantization, which is usually acceptable. On a single RTX 3090, QLoRA lets you fine-tune a Llama 3 8B model in a few hours. Full fine-tuning of the same model would need a multi-GPU server.

Why Unsloth

Unsloth rewrites the attention kernels used during training to use significantly less memory and run faster than the standard Hugging Face + PEFT stack. On identical hardware, Unsloth typically gives 2x the throughput and fits a larger batch size, which means fewer training steps to reach the same result. It supports Llama 3, Mistral, Phi-3, Gemma, and most popular open-weight architectures. You use the same familiar Trainer API, so switching from PEFT to Unsloth is a small change.

Hardware requirements

The minimum for a real fine-tune is 16 GB VRAM with QLoRA enabled. Below that you hit out-of-memory errors or have to reduce batch size to 1, which makes training unstable. Common setups that work well:

Dataset format: JSONL

Most fine-tuning frameworks expect one JSON object per line (JSONL). Each object holds a conversation with roles. The most common format used by Llama 3 and Mistral models is the ShareGPT / conversations format:

{"conversations": [{"role": "system", "content": "You are a support agent for Acme Corp. Reply concisely and always offer a next step."}, {"role": "user", "content": "My order hasn't arrived."}, {"role": "assistant", "content": "I'm sorry to hear that. Can you share your order number? I'll look it up and give you a status update right away."}]}
{"conversations": [{"role": "system", "content": "You are a support agent for Acme Corp. Reply concisely and always offer a next step."}, {"role": "user", "content": "I was charged twice."}, {"role": "assistant", "content": "That should not have happened. Please share the email on your account and I'll flag it for our billing team to reverse the duplicate charge today."}]}

Each line is a complete, self-contained example. The system prompt sets the persona; the user turn is the input; the assistant turn is what you want the model to learn to produce. You can have multi-turn conversations in a single example by adding more role/content pairs, but single-turn examples are easier to collect and work well for most tasks.

What comes next

Now that you understand the mechanics, you are ready to run an actual training job. The next guide in this series walks through fine-tuning Llama 3 on your own JSONL dataset using Unsloth, from pip install to a GGUF file ready to load into Ollama.