In this categoryLocal AI · 24

Run Models

Local Apps & UIs

Hardware & Builds

Model Spotlights

Fine-Tuning

Local AIAdvanced

How to Fine-Tune Llama 3 on Your Own Data with Unsloth

Install Unsloth, load Llama 3, format your JSONL dataset, run a LoRA training job, and export the result as a GGUF file ready for Ollama.

30 minAdvanced

Visit Llama

This guide is the hands-on companion to the LoRA basics overview. You will go from a clean Python environment to a trained adapter merged into a GGUF model file. The whole pipeline runs on a single 24 GB GPU using Unsloth's 4-bit QLoRA path. Expect the training itself to take 20 to 90 minutes depending on dataset size and your hardware.

A GPU with 16+ GB VRAM and CUDA 11.8 or 12.1 installed
Python 3.10+ in a virtual environment (conda or venv)
A JSONL dataset ready to go (see the dataset preparation guide for format)
A Hugging Face account with access to meta-llama/Meta-Llama-3-8B-Instruct (free to request)

Step 1: Install Unsloth and dependencies

Unsloth has different wheels for different CUDA versions. The command below covers the most common CUDA 12.1 setup. Check the Unsloth README if you are on 11.8.

bash - training env

$conda create -n finetune python=3.11 -y && conda activate finetune

$pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

$pip install unsloth

$pip install datasets trl transformers accelerate bitsandbytes

verify GPU is visible

$python -c "import torch; print(torch.cuda.get_device_name(0))"

NVIDIA GeForce RTX 4090

Step 2: Load the base model with Unsloth

Create a training script. The FastLanguageModel loader handles 4-bit quantization, flash attention, and Unsloth's kernel optimizations in one call. The max_seq_length controls how long each training example can be — 2048 is a good default.

train.py

from unsloth import FastLanguageModel
import torch

MAX_SEQ_LENGTH = 2048
DTYPE = None          # auto-detect: float16 on older GPUs, bfloat16 on Ampere+
LOAD_IN_4BIT = True   # QLoRA — set False for full LoRA if VRAM allows

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3-8B-Instruct",
    max_seq_length=MAX_SEQ_LENGTH,
    dtype=DTYPE,
    load_in_4bit=LOAD_IN_4BIT,
)

# Wrap with LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,                        # rank — higher = more capacity, more VRAM
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,              # Unsloth recommends 0
    bias="none",
    use_gradient_checkpointing="unsloth",  # cuts VRAM further
    random_state=42,
)

r=16 is a safe starting rank

LoRA rank controls how many new parameters the adapters add. r=16 gives a good balance for most tasks. Increase to r=32 or r=64 if you have headroom and the task is complex; decrease to r=8 if you are running out of VRAM.

Step 3: Load and format your dataset

Unsloth uses the Hugging Face datasets library. Load your JSONL file and apply the Llama 3 chat template so the tokens are formatted exactly as the model was pre-trained to expect.

train.py

from datasets import load_dataset

dataset = load_dataset("json", data_files="dataset.jsonl", split="train")

def format_conversations(example):
    # tokenizer.apply_chat_template handles <|begin_of_text|>, roles, and <|eot_id|>
    text = tokenizer.apply_chat_template(
        example["conversations"],
        tokenize=False,
        add_generation_prompt=False,
    )
    return {"text": text}

dataset = dataset.map(format_conversations, remove_columns=dataset.column_names)
print(f"Loaded {len(dataset)} examples")
print(dataset[0]["text"][:300])  # sanity check the first example

Step 4: Configure and run training

Plug the model and dataset into Hugging Face's SFTTrainer. The key hyperparameters are learning rate, batch size, and number of epochs. For most fine-tunes, 3 epochs with a learning rate of 2e-4 is a reliable starting point.

train.py

from trl import SFTTrainer
from transformers import TrainingArguments

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=MAX_SEQ_LENGTH,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,   # effective batch = 2 * 4 = 8
        warmup_steps=10,
        num_train_epochs=3,
        learning_rate=2e-4,
        fp16=not torch.cuda.is_bf16_supported(),
        bf16=torch.cuda.is_bf16_supported(),
        logging_steps=10,
        output_dir="./checkpoints",
        save_steps=100,
        optim="adamw_8bit",              # Unsloth's memory-efficient optimizer
        weight_decay=0.01,
        lr_scheduler_type="cosine",
        seed=42,
    ),
)

trainer.train()

bash - training

$python train.py

Unsloth: Will use gradient checkpointing.

{'loss': 1.842, 'learning_rate': 0.0002, 'epoch': 0.10}

{'loss': 1.421, 'learning_rate': 0.00018, 'epoch': 0.50}

{'loss': 0.983, 'learning_rate': 0.00010, 'epoch': 1.50}

{'loss': 0.712, 'learning_rate': 0.00002, 'epoch': 2.90}

Training complete.

loss dropping steadily is a good sign; plateau early = too few examples

Step 5: Save to GGUF for Ollama

Unsloth can merge the LoRA adapters into the base model and quantize directly to GGUF in one step. The q4_k_m quantization gives the best balance between file size and quality for most use cases.

train.py

# Merge adapters into base weights and export
model.save_pretrained_gguf(
    "my-llama3-finetuned",
    tokenizer,
    quantization_method="q4_k_m",   # ~4.7 GB for an 8B model
)
print("Saved: my-llama3-finetuned/model-q4_k_m.gguf")

bash - export

Unsloth: Merging LoRA adapters into base model...

llama_model_quantize: model size = 14899.16 MB

llama_model_quantize: quant size = 4685.31 MB

Saved: my-llama3-finetuned/model-q4_k_m.gguf

Other quantization options

q8_0 keeps more quality but doubles the file size. q2_k is tiny but noticeably worse. For most fine-tuning purposes where the base model is already strong, q4_k_m is the right default. Run your eval on q4_k_m first before experimenting with other levels.

Result

You have a .gguf file that contains Llama 3 with your fine-tuned behavior baked in. Copy it to any machine running Ollama and you can run it locally with zero cloud dependency. The next guide in this series covers creating a Modelfile and loading the GGUF into Ollama.

Local models run better with more VRAM. CompareRTX GPUs on Amazonbefore you upgrade.(affiliate link. We may earn a commission at no extra cost. Disclosure)