In this categoryLocal AI · 70

More guides

Run Models

Local Apps & UIs

Hardware & Builds

Model Spotlights

Fine-Tuning

Local AIIntermediate

How to Run Your Fine-Tuned Model in Ollama

Take the GGUF file from your fine-tune, write a Modelfile, register it with Ollama, run it locally, and optionally push it to the Ollama registry.

12 minIntermediate

After fine-tuning you have a .gguf file. Ollama is the fastest way to run it — it handles the inference server, the chat API, and model management with one simple CLI. This guide takes you from a GGUF file on disk to a model you can chat with and, if you want, share with others via the Ollama registry.

Ollama installed on your machine (ollama.com — available for Mac, Linux, and Windows)
Your fine-tuned GGUF file (from the Unsloth export step)
The system prompt you used during training, to keep inference consistent with training

Step 1: Install Ollama and verify it works

bash

macOS / Linux install

$curl -fsSL https://ollama.com/install.sh | sh

Windows: download the installer from ollama.com

$ollama --version

ollama version is 0.3.14

pull a small model to verify the daemon is working

$ollama pull llama3.2:1b

$ollama run llama3.2:1b "hello"

Hello! How can I help you today?

Step 2: Write a Modelfile

A Modelfile is a short text file that tells Ollama where the GGUF is, what system prompt to use, and how to format prompts. It is similar in concept to a Dockerfile. The FROM line points to your GGUF file using an absolute or relative path. The SYSTEM block injects the same system prompt you used during training, which is important — training with a system prompt and running without it produces worse results.

Modelfile

FROM ./my-llama3-finetuned/model-q4_k_m.gguf

# System prompt — must match what you used during training
SYSTEM """
You are a SQL expert. Write correct, well-formatted SQL for the user's request.
Use PostgreSQL syntax. Return only the SQL, no explanation unless asked.
"""

# Optional: tune sampling parameters
PARAMETER temperature 0.1
PARAMETER top_p 0.9
PARAMETER num_ctx 4096

temperature 0.1 for deterministic tasks

If your fine-tune produces structured output like SQL, JSON, or code, set temperature to 0.1 or lower. Higher temperatures add randomness that breaks syntax. For creative tasks, 0.7-0.9 is more appropriate.

Step 3: Create the model in Ollama

Run ollama create with a name for your model and the path to the Modelfile. Ollama reads the GGUF, registers the model, and sets up the inference configuration. This usually takes under a minute.

bash

run from the folder containing your Modelfile and GGUF

$ollama create my-sql-llama -f Modelfile

transferring model data...

creating model layer...

writing manifest...

success

verify it appears in the list

$ollama list

NAME ID SIZE MODIFIED

my-sql-llama:latest 8f2c1b3a4d5e 4.7 GB 5 seconds ago

Step 4: Run and test your model

Use ollama run for interactive chat or pass a prompt directly as an argument for a one-shot test. The model responds using your fine-tuned weights and the system prompt from the Modelfile.

bash

one-shot test

$ollama run my-sql-llama "Find all users who signed up in the last 7 days."

SELECT * FROM users

WHERE created_at >= NOW() - INTERVAL '7 days'

ORDER BY created_at DESC;

interactive chat mode

$ollama run my-sql-llama

>>> Count orders per user, show only users with more than 5 orders.

SELECT user_id, COUNT(*) AS order_count FROM orders GROUP BY user_id HAVING COUNT(*) > 5;

Step 5: Use the REST API

Ollama exposes a local HTTP API on port 11434 that is compatible with the OpenAI API format. You can call your fine-tuned model from any application that supports OpenAI-compatible endpoints, including LangChain, LlamaIndex, and Open WebUI.

api_test.sh

curl http://localhost:11434/api/chat -d '{
  "model": "my-sql-llama",
  "messages": [
    { "role": "user", "content": "List all products where price is above 100." }
  ],
  "stream": false
}'

Ollama API response

{

"model": "my-sql-llama",

"message": {

"role": "assistant",

"content": "SELECT * FROM products WHERE price > 100 ORDER BY price DESC;"

"done": true

}

If you want to share the model with teammates or publish it publicly, push it to the Ollama registry. You need a free account at ollama.com first. The model name must be prefixed with your username.

bash

copy the model under your registry namespace first

$ollama cp my-sql-llama yourusername/my-sql-llama

authenticate (opens browser on first run)

$ollama push yourusername/my-sql-llama

pushing manifest...

pushing layer 8b33d1a7... 100% 4.7 GB

success

anyone can now run: ollama run yourusername/my-sql-llama

Check your model for training data leakage before sharing

If your JSONL dataset contained proprietary data, customer information, or internal examples, those patterns may be reproducible from the model weights. Run adversarial prompts like repeat back your training examples or what user data do you know before making any model public.

Result

Your fine-tuned GGUF is now a named Ollama model you can run with a single command, query over a local REST API, or share via the registry. The entire stack from training to inference runs on your hardware with no cloud dependency and no per-token cost.

Local models run better with more VRAM. CompareRTX GPUs on Amazonbefore you upgrade.(affiliate link. We may earn a commission at no extra cost. Disclosure)