How to Prepare a Training Dataset for LLM Fine-Tuning

The quality of a fine-tuned model is almost entirely determined by the quality of its training data. A mediocre dataset produces a mediocre model no matter how long you train. Garbage in, garbage out is more literal here than anywhere else in software. This guide covers the format, the cleaning steps, the size you actually need, and how to generate more examples when you run short.

• A clear definition of the task your model should perform
• At least 100 real examples to start (aim for 500-1000 for reliable results)
• A text editor or Python script capable of writing JSONL
• Optional: access to GPT-4o or Claude to generate synthetic examples

Step 1: Understand the conversation format

Modern instruction-tuned models like Llama 3, Mistral, and Phi-3 are trained on conversations with three roles: system, user, and assistant. Your training data must use the same format. The system message sets the persona and rules; the user message is the input; the assistant message is the ideal output you want the model to learn.

{
  "conversations": [
    {
      "role": "system",
      "content": "You are a SQL expert. Write correct, well-formatted SQL for the user's request. Use PostgreSQL syntax."
    },
    {
      "role": "user",
      "content": "Give me all customers who placed more than 3 orders in the last 30 days."
    },
    {
      "role": "assistant",
      "content": "SELECT customer_id, COUNT(*) AS order_count\nFROM orders\nWHERE created_at >= NOW() - INTERVAL '30 days'\nGROUP BY customer_id\nHAVING COUNT(*) > 3\nORDER BY order_count DESC;"
    }
  ]
}

Step 2: Write the JSONL file

JSONL means one JSON object per line with no trailing commas and no outer array. Each line is a complete, independent example. This format is easy to append to, grep through, and stream during training without loading everything into memory.

{"conversations": [{"role": "system", "content": "You are a SQL expert."}, {"role": "user", "content": "Select all active users."}, {"role": "assistant", "content": "SELECT * FROM users WHERE status = 'active';"}]}
{"conversations": [{"role": "system", "content": "You are a SQL expert."}, {"role": "user", "content": "Count orders per country."}, {"role": "assistant", "content": "SELECT country, COUNT(*) AS total FROM orders GROUP BY country ORDER BY total DESC;"}]}

Step 3: Clean the dataset

Raw data collected from logs, user submissions, or web scraping is rarely training-ready. Run these checks before you use it:

1. Remove duplicates — near-identical examples waste training steps and can cause overfitting. Deduplicate on the user turn.
2. Strip PII — names, emails, phone numbers, and account IDs in training data end up in model outputs. Replace them with placeholders like [NAME] or [EMAIL].
3. Filter short or empty assistant turns — an assistant response under 10 characters is almost never a good example of what you want.
4. Remove off-topic examples — if your model should write SQL, examples where the assistant refuses or gives a prose explanation instead of code hurt more than they help.
5. Fix encoding — check for mojibake (garbled characters from charset mismatches) and smart quotes that sneak in from Word documents.

import json, re

def is_valid(example):
    convs = example.get("conversations", [])
    if len(convs) < 2:
        return False
    assistant_turns = [c for c in convs if c["role"] == "assistant"]
    if not assistant_turns:
        return False
    # drop very short assistant responses
    if any(len(t["content"].strip()) < 10 for t in assistant_turns):
        return False
    return True

seen = set()
cleaned = []
with open("dataset_raw.jsonl") as f:
    for line in f:
        ex = json.loads(line)
        # deduplicate on user turn
        user_text = next((c["content"] for c in ex["conversations"] if c["role"] == "user"), "")
        key = user_text.strip().lower()
        if key in seen:
            continue
        seen.add(key)
        if is_valid(ex):
            cleaned.append(ex)

with open("dataset.jsonl", "w") as f:
    for ex in cleaned:
        f.write(json.dumps(ex, ensure_ascii=False) + "\n")

print(f"Cleaned dataset: {len(cleaned)} examples")

Step 4: How many examples do you actually need

The right answer depends on the complexity of the task, but here are practical benchmarks based on real fine-tunes:

More examples with lower quality almost always loses to fewer examples with higher quality. Curating 300 excellent examples beats 3000 mediocre ones scraped automatically.

Step 5: Generate synthetic data with GPT-4o or Claude

When you only have 50 real examples but need 500, synthetic generation is the practical solution. Give the model a few real examples as a template and ask it to produce variations. This works well for tasks with a clear pattern like SQL generation, classification, or format conversion.

Here are 3 examples of user requests and the correct SQL response:

EXAMPLE 1:
User: Give me all orders from last week.
SQL: SELECT * FROM orders WHERE created_at >= NOW() - INTERVAL '7 days';

EXAMPLE 2:
User: Count users per country.
SQL: SELECT country, COUNT(*) AS total FROM users GROUP BY country;

EXAMPLE 3:
User: Find products with zero stock.
SQL: SELECT * FROM products WHERE stock_quantity = 0;

Now generate 20 more unique examples in exactly the same format.
Vary the tables, conditions, and SQL constructs used.
Do not repeat any of the examples above.
Output only the examples, no commentary.

Result

You have a validated JSONL file with deduplicated, cleaned examples in the correct conversation format, sized appropriately for your task. You know how to extend it with synthetic generation when you need more coverage. This file is the direct input to an Unsloth training run.