How to Pick the Right Local AI Model for Your Hardware

Match your GPU VRAM tier to concrete model recommendations, understand Q4/Q5/Q8 quantization, and use a simple rule of thumb to avoid out-of-memory crashes before you download.

7 minBeginner

The single biggest mistake when running local AI is downloading a model that does not fit in your GPU memory. The model loads partly or falls back to slow CPU inference without warning, and you spend 20 minutes downloading 15 GB to discover it is unusable. This guide gives you a table to check before you pull anything.

The rule of thumb

Multiply the model's parameter count by the bytes per weight for your chosen quantization, then add about 20% for the key-value cache and overhead. That gives you the minimum VRAM you need.

VRAM estimate formula

VRAM (GB) ≈ (parameters in billions × bytes per weight × 1.2)

Q4: 0.5 bytes per weight  →  7B × 0.5 × 1.2 = 4.2 GB
Q5: 0.625 bytes per weight →  7B × 0.625 × 1.2 = 5.3 GB
Q8: 1.0 byte per weight   →  7B × 1.0 × 1.2 = 8.4 GB

Quantization explained

Quantization reduces the number of bits used to store each model weight. The full-precision model (FP16) uses 16 bits per weight. Quantized versions trade a small amount of accuracy for a large reduction in size and VRAM usage.

Format	Bits per weight	Size vs FP16	Quality
Q3	3	~19%	Noticeable quality drop; only use if very constrained
Q4_K_M	4	~25%	Good; recommended minimum for most use cases
Q5_K_M	5	~31%	Very good; sweet spot between size and accuracy
Q8_0	8	~50%	Near-lossless; use when VRAM allows
FP16	16	100%	Full precision; needs the most VRAM

K_M variants are better than plain Q4

Ollama and LM Studio default to Q4_K_M, not plain Q4. The K_M suffix means the quantization is non-uniform — it allocates more bits to the weights that matter most. The quality difference versus plain Q4 is meaningful, and the file size is almost identical.

VRAM tier recommendations

The table below covers the five most common consumer VRAM tiers. All models listed run at a useful speed (at least 10 tokens per second on a mid-range GPU). Models with a star are the recommended first pick for each tier.

VRAM	Best model choices	Notes
4 GB	Phi-3.5-mini Q4 (2.2 GB) ★, Llama 3.2 3B Q4 (2.0 GB), Gemma 2 2B Q8 (2.7 GB)	Close background apps before loading. Mistral 7B Q4 is too large.
6 GB	Mistral 7B Q4 (4.1 GB) ★, Llama 3.1 8B Q4 (4.7 GB), Phi-3.5-mini Q8 (3.9 GB)	Good everyday tier. Handles most coding and writing tasks well.
8 GB	Mistral 7B Q5 (5.1 GB) ★, Llama 3.1 8B Q5 (5.7 GB), Gemma 2 9B Q4 (5.5 GB)	Sweet spot. Can also run Mistral-Nemo 12B at Q3 in a pinch.
12 GB	Mistral-Nemo 12B Q4 (7.1 GB) ★, Llama 3.1 8B Q8 (8.5 GB), Gemma 2 9B Q5 (6.7 GB)	High quality output. 128k context with Mistral-Nemo.
16 GB	Qwen 2.5 14B Q5 (9.8 GB) ★, Gemma 2 27B Q3 (11 GB), Mistral-Nemo Q8 (13 GB)	Strong reasoning at this tier. 27B models at Q3 are viable.
24 GB	Llama 3.3 70B Q3 (28 GB — needs 2× GPU) ★, Qwen 2.5 32B Q5 (22 GB), Qwen 2.5 14B Q8 (14 GB)	Near-frontier quality. 70B at Q3 needs two 16 GB cards or one 24 GB card with shared RAM.

How to check your VRAM

Check GPU VRAM

Windows / Linux (NVIDIA)

$nvidia-smi --query-gpu=name,memory.total --format=csv,noheader

NVIDIA GeForce RTX 3070, 8192 MiB

macOS (Apple Silicon — unified memory)

$system_profiler SPDisplaysDataType | grep 'Total Number of Cores'

For Apple Silicon: RAM is shared with GPU. Use the same VRAM table against your total RAM.

Apple Silicon unified memory

Apple Silicon Macs share RAM between the CPU and GPU. An M2 with 16 GB of RAM can use most of that 16 GB for a model — treat it the same as a 16 GB VRAM tier. The main constraint is leaving 4 to 6 GB free for macOS and other apps.

What if the model does not fit?

Ollama and LM Studio will both fall back to CPU inference automatically when a model exceeds your VRAM. The model still runs, but expect 1 to 5 tokens per second instead of 30 to 80. For occasional use that may be acceptable; for interactive chat it becomes unusable quickly. Choose a smaller model or a lower quantization instead.

Result

Use the formula to estimate VRAM before you download, match your GPU tier to the recommendation table, and default to Q4_K_M when in doubt. You will land on a model that loads cleanly and responds fast the first time.

Local models run better with more VRAM. CompareRTX GPUs on Amazonbefore you upgrade.(affiliate link. We may earn a commission at no extra cost. Disclosure)

Related guides

Local AI

Beginner10 min

Best GPU for Running AI Locally in 2026

VRAM is the single spec that determines which models you can run. Here is how every major GPU stacks up for local LLMs and image generation in 2026.

Local AI

Beginner6 min

How Much RAM Do You Need for Local AI?

RAM requirements for local AI depend on whether you are talking about GPU VRAM or system RAM, and which models you want to run. Here are the real numbers.