How to Run Mistral 7B Locally

Mistral 7B is one of the most capable open-weight models at its size. Running it locally with Ollama takes a single command, and a 4 GB GPU is enough to run the quantized version at a usable speed. This guide walks through the install, the model choice, the hardware check, and practical system prompt tips.

What you need

• Ollama installed from ollama.com (macOS, Windows, or Linux)
• 4 GB VRAM for Mistral 7B Q4 (runs on CPU-only at about 3 tokens/sec on 16 GB RAM)
• About 4.1 GB of free disk space

Step 1: Install Ollama

Download and run the Ollama installer for your OS from ollama.com. On macOS and Windows it installs as a background service. On Linux, run the one-line install script.

Step 2: Pull and run Mistral 7B

The ollama run command pulls the model if it is not already cached, then opens an interactive chat. The default Mistral tag uses the Q4_K_M quantization at 4.1 GB.

Step 3: Mistral 7B vs Mistral-Nemo — which to pick

Mistral 7B (the default mistral tag) is 7 billion parameters and needs roughly 4 GB of VRAM. Mistral-Nemo is a 12 billion parameter model trained jointly by Mistral and NVIDIA with a 128k context window and better instruction following — but it needs about 8 GB of VRAM.

Step 4: Add a system prompt for better results

Ollama lets you create a custom Modelfile with a built-in system prompt so you do not have to re-paste it every session. This is especially useful for Mistral, which responds noticeably better when given a clear role and output format.

FROM mistral

SYSTEM """
You are a concise, technical assistant. Rules:
- Answer in plain English. No filler or preamble.
- For factual questions, give the answer first, then a brief explanation.
- If you are uncertain, say so rather than guessing.
- Use markdown only when the user asks for formatted output.
"""

PARAMETER temperature 0.3

Result

Mistral 7B is now running locally with a system prompt that keeps it on-task, and you know when upgrading to the 12B Mistral-Nemo is worth the extra 4 GB of VRAM.