How to Set Up Local AI on 12GB of VRAM

12GB of VRAM is the mixture-of-experts sweet spot: OpenAI's gpt-oss-20b needs about 12GB at Q4 and, because only 3.6B of its 20B parameters activate per token, generates at nearly 7B speed. Here is how to install it, which quantization it actually ships in, and cited real-world tokens-per-second.

10 minBeginner

Visit Mistral

A 12GB GPU's headline model is OpenAI's gpt-oss-20b: 20B total parameters, but only 3.6B active per token thanks to its mixture-of-experts design, so it needs about 12GB at Q4 and runs close to 7B-model speed. Pull it with ollama run gpt-oss:20b. Mistral Nemo 12B (about 7GB) and everything from the 8GB tier fit too, with room left over.

Written by Priya Raghunathan, local-AI hardware reviewer. I test consumer GPUs and Apple Silicon against real llama.cpp and MLX benchmarks before recommending them, because vendor VRAM numbers rarely match what actually loads.

What actually fits in 12GB

Model	Params	VRAM at Q4	License	Best for
OpenAI gpt-oss-20b	20B total / 3.6B active (MoE)	~12 GB	Apache 2.0	Chat, coding, reasoning, tool use
Mistral Nemo 12B	12B	~7 GB	Apache 2.0	Chat, analysis
Qwen3 8B	8B	~5 GB	Apache 2.0	All-round, tool calling
Llama 3.1 8B	8B	~5 GB	Llama 3.1 Community	128K context, general use

Full model specs and VRAM verification

This is the 12GB slice of setuproll's verified local-AI catalog. See Best Local LLM for 12GB VRAM for the full model list, headroom figures and FAQs.

Which quantization to pick

gpt-oss-20b is a special case: OpenAI shipped it natively in MXFP4, a 4-bit floating-point format baked into the model release rather than a community quantization pass, and Ollama serves that MXFP4 build by default when you pull gpt-oss:20b. Everything else on this list follows the usual rule: GGUF at Q4_K_M through Ollama, which is what fits a 12GB card without any extra steps. See the GGUF vs MLX vs NVFP4 explainer for how MXFP4 and NVFP4 relate to the more common GGUF formats.

Install and run

zsh - gpt-oss-20b on a 12GB GPU

Pulls the native MXFP4 build; needs about 12GB VRAM and 16GB system RAM

$ollama pull gpt-oss:20b

pulling manifest

pulling model weights... ████████████ 100% 13 GB

success

$ollama run gpt-oss:20b

>>> Send a message

Real-world speed: what to expect

Hardware Corner's real-hardware GPU LLM benchmarks (tested on Ubuntu 24.04 with llama.cpp's llama-bench, updated December 2025) measured an RTX 4070 at 52 tokens per second on an 8B model at 16K context, and an RTX 4070 Ti Super at 113.6 tokens per second on gpt-oss's 20B mixture-of-experts model at the same context length, well ahead of dense models its size because of the low active-parameter count. On the official Ollama repository, a separate user-reported thread on gpt-oss-20b performance put a 12GB RTX 3060 at roughly 27 tokens per second on the same model via LM Studio, a useful same-VRAM-class reference point since 12GB cards vary in bandwidth.

Watch your context length

gpt-oss-20b supports up to 128K context, but pushing context that high on a 12GB card leaves almost nothing for the KV cache and can collapse generation speed to single digits regardless of GPU. Keep working context under 8K to 16K tokens for the speeds above.

Where to go from here

Want more headroom for Gemma 3 27B? See the 16GB VRAM setup guide.
Coming from an 8GB card? The 8GB VRAM setup guide covers the models below this tier.
Comparing GPUs before you buy? Best Local LLM for 12GB VRAM has verified hardware picks and FAQs.

Local models run better with more VRAM. CompareRTX GPUs on Amazonbefore you upgrade.(affiliate link. We may earn a commission at no extra cost. Disclosure)

Related guides

Local AI

Beginner10 min

How to Set Up Local AI on 8GB of VRAM

An 8GB GPU runs the 7B to 8B model class comfortably at Q4 quantization, and Phi-4 14B just barely fits. Here is exactly what to install, which quantization format to pick, and real measured tokens-per-second so you know what to expect before you download anything.

Local AI

Beginner11 min

Best Local LLM for 16 GB VRAM: Setup, Quantization and Real Speed

16GB of VRAM unlocks Gemma 3 27B, which needs about 15GB at Q4 and just fits, plus OpenAI's gpt-oss-20b with real headroom to spare. Here is how to install both, which quantization format to use, and measured tokens-per-second on the RTX 4060 Ti 16GB and RTX 5070 Ti.