How to Set Up Local AI on 8GB of VRAM

An 8GB GPU runs the 7B to 8B model class comfortably at Q4 quantization, and Phi-4 14B just barely fits. Here is exactly what to install, which quantization format to pick, and real measured tokens-per-second so you know what to expect before you download anything.

10 minBeginner

An 8GB GPU is enough for real local AI work: Qwen3 8B and Llama 3.1 8B both need about 5GB of VRAM at Q4 quantization, leaving headroom for context. Install Ollama, pull a model, and you are generating text in under five minutes. On an RTX 4060 8GB, Llama 3.1 8B generates around 42 tokens per second, fast enough for interactive chat and coding help.

Written by Priya Raghunathan, local-AI hardware reviewer. I test consumer GPUs and Apple Silicon against real llama.cpp and MLX benchmarks before recommending them, because vendor VRAM numbers rarely match what actually loads.

What actually fits in 8GB

At Q4 quantization, the default Ollama serves for every model, the 7B to 8B class is the comfortable zone for an 8GB card. Phi-4 14B is the outlier: its 14 billion parameters compress down to about 8GB at Q4, so it fits, but with almost nothing left over for a long context window.

Model	Params	VRAM at Q4	License	Best for
Qwen3 8B	8B	~5 GB	Apache 2.0	All-round chat, tool calling
Llama 3.1 8B	8B	~5 GB	Llama 3.1 Community	128K context, general use
Qwen 2.5 Coder 7B	7B	~4 GB	Apache 2.0	Coding
Mistral 7B v0.3	7B	~4 GB	Apache 2.0	Chat, summarization
Phi-4 14B	14B	~8 GB (tight)	MIT	Coding, math reasoning

Full model specs and VRAM verification

This table is the 8GB slice of setuproll's verified local-AI model catalog. See the full breakdown on Best Local LLM for 8GB VRAM for FAQs, headroom figures and every model that fits.

Which quantization to pick

On an 8GB card the answer is simple: GGUF at Q4_K_M, which is what Ollama pulls by default. MLX only matters on Apple Silicon and NVFP4 only matters on NVIDIA's Blackwell cards (RTX 5090/5080/5070 Ti), so neither applies here unless your 8GB card happens to be a 5070-class Blackwell part, in which case NVFP4 checkpoints will run a touch more accurately at the same 4-bit size. For the full comparison, see the GGUF vs MLX vs NVFP4 explainer.

Install and run

zsh - Ollama on an 8GB GPU

Installs Ollama and pulls Qwen3 8B, which needs about 5GB of VRAM at Q4

$curl -fsSL https://ollama.com/install.sh | sh

$ollama pull qwen3:8b

pulling manifest

pulling model weights... ████████████ 100% 5.2 GB

success

$ollama run qwen3:8b

>>> Send a message

Real-world speed: what to expect

GPU rental provider DatabaseMart's RTX 4060 Ollama benchmark (Ollama 0.5.11, updated August 2026) measured Llama 3.1 8B at 41.7 tokens per second and Mistral 7B at 50.9 tokens per second on an 8GB RTX 4060 at Q4 quantization. Models under 5GB stayed above 40 tokens per second consistently; the moment you cross into the 13B to 14B range, that same card dropped to roughly 8 tokens per second, low GPU utilization the benchmark attributed to the model no longer fitting cleanly in 8GB. That is the practical ceiling of this tier: the 7B to 8B class is fast, Phi-4 14B works but generates noticeably slower.

13B+ models slow down hard on 8GB

Once a model's weights plus KV cache exceed your 8GB budget, Ollama offloads layers to system RAM and speed drops sharply, in the DatabaseMart benchmark a 13B model fell to about 8 tokens per second with GPU utilization down to 25-42%. Stay in the 7B to 8B class, or the tight-fit Phi-4 14B, for consistently fast generation.

Where to go from here

Need more headroom for bigger models? See the 12GB and 16GB VRAM setup guides.
Want to understand quantization formats before you pick a model? Read the GGUF vs MLX vs NVFP4 explainer.
Comparing GPUs before you buy? Best Local LLM for 8GB VRAM has verified hardware picks and FAQs.

Local models run better with more VRAM. CompareRTX GPUs on Amazonbefore you upgrade.(affiliate link. We may earn a commission at no extra cost. Disclosure)

Related guides

Local AI

Beginner10 min

How to Set Up Local AI on 12GB of VRAM

12GB of VRAM is the mixture-of-experts sweet spot: OpenAI's gpt-oss-20b needs about 12GB at Q4 and, because only 3.6B of its 20B parameters activate per token, generates at nearly 7B speed. Here is how to install it, which quantization it actually ships in, and cited real-world tokens-per-second.

Local AI

Intermediate14 min

GGUF vs MLX vs NVFP4: Local AI Quantization Formats Explained

Three names keep showing up when you go to download a local model: GGUF, MLX, and the newer NVFP4. They are not interchangeable, and picking the wrong one wastes memory or leaves your hardware idle. Here is what each format is, which one your machine actually wants, and how to choose in ten seconds.