How to Pick the Right Local AI Model for Your Hardware

The single biggest mistake when running local AI is downloading a model that does not fit in your GPU memory. The model loads partly or falls back to slow CPU inference without warning, and you spend 20 minutes downloading 15 GB to discover it is unusable. This guide gives you a table to check before you pull anything.

The rule of thumb

Multiply the model's parameter count by the bytes per weight for your chosen quantization, then add about 20% for the key-value cache and overhead. That gives you the minimum VRAM you need.

VRAM (GB) ≈ (parameters in billions × bytes per weight × 1.2)

Q4: 0.5 bytes per weight  →  7B × 0.5 × 1.2 = 4.2 GB
Q5: 0.625 bytes per weight →  7B × 0.625 × 1.2 = 5.3 GB
Q8: 1.0 byte per weight   →  7B × 1.0 × 1.2 = 8.4 GB

Quantization explained

Quantization reduces the number of bits used to store each model weight. The full-precision model (FP16) uses 16 bits per weight. Quantized versions trade a small amount of accuracy for a large reduction in size and VRAM usage.

VRAM tier recommendations

The table below covers the five most common consumer VRAM tiers. All models listed run at a useful speed (at least 10 tokens per second on a mid-range GPU). Models with a star are the recommended first pick for each tier.

How to check your VRAM

What if the model does not fit?

Ollama and LM Studio will both fall back to CPU inference automatically when a model exceeds your VRAM. The model still runs, but expect 1 to 5 tokens per second instead of 30 to 80. For occasional use that may be acceptable; for interactive chat it becomes unusable quickly. Choose a smaller model or a lower quantization instead.

Result

Use the formula to estimate VRAM before you download, match your GPU tier to the recommendation table, and default to Q4_K_M when in doubt. You will land on a model that loads cleanly and responds fast the first time.