How to Set Up Local AI on 24GB of VRAM

A 24GB card runs Ornith 1.0 35B, the top open-weight coding model on SWE-bench Verified at 75.6%, which needs about 20GB at Q4. Gemma 3 27B and everything smaller fit with headroom left for long context. Here is the install path, quantization choice, and cited RTX 3090 vs RTX 4090 speed.

11 minIntermediate

A 24GB GPU's headline model is Ornith 1.0 35B: it needs about 20GB at Q4 quantization and scores 75.6% on SWE-bench Verified, one of the highest results for any open-weight coding model. Gemma 3 27B (about 15GB) and every lighter model fit too, leaving room for longer context windows.

Written by Priya Raghunathan, local-AI hardware reviewer. I test consumer GPUs and Apple Silicon against real llama.cpp and MLX benchmarks before recommending them, because vendor VRAM numbers rarely match what actually loads.

What actually fits in 24GB

Model	Params	VRAM at Q4	License	Best for
Ornith 1.0 35B	35B	~20 GB	MIT	Coding (75.6% SWE-bench Verified)
Gemma 3 27B	27B	~15 GB	Gemma Terms of Use	Instruction following, general use
OpenAI gpt-oss-20b	20B total / 3.6B active (MoE)	~12 GB	Apache 2.0	Chat, coding, tool use
Mistral Nemo 12B	12B	~7 GB	Apache 2.0	Chat, analysis

Full model specs and VRAM verification

This is the 24GB slice of setuproll's verified local-AI catalog. See Best Local LLM for 24GB VRAM for the full list and FAQs, or Best Local LLM for the RTX 3090 for a single-card deep dive.

Which quantization to pick

GGUF at Q4_K_M through Ollama is the easiest path and works on any 24GB card, NVIDIA or AMD. If your whole model fits in VRAM with no CPU offload, which it does at this tier, the older int4 formats (GPTQ, AWQ, EXL2) served through vLLM or text-generation-webui can run faster than GGUF on NVIDIA cards, since they are built for all-on-GPU inference rather than the run-anywhere flexibility GGUF trades for. See the GGUF vs MLX vs NVFP4 explainer for when each format is worth the extra setup.

Install and run

zsh - Ornith 1.0 35B on a 24GB GPU

Pulls the GGUF build from Hugging Face; needs about 20GB VRAM at Q4

$ollama pull hf.co/deepreinforce-ai/Ornith-1.0-35B-GGUF

pulling manifest

pulling model weights... ████████████ 100% 21 GB

success

$ollama run hf.co/deepreinforce-ai/Ornith-1.0-35B-GGUF

>>> Send a message

Real-world speed: what to expect

Hardware Corner's real-hardware GPU LLM benchmarks (llama.cpp's llama-bench on Ubuntu 24.04, updated December 2025, Q4_K_XL quantization, 16K context) measured both 24GB cards on a 30B-class dense model, the closest tested size to Ornith 1.0 35B and Gemma 3 27B:

GPU	8B model	~20B MoE model	~30B dense model
RTX 3090 24GB	87.5 tok/s	128.5 tok/s	113.8 tok/s
RTX 4090 24GB	104.3 tok/s	163.9 tok/s	139.7 tok/s

The RTX 3090 lands within roughly 15-20% of the RTX 4090 across every size tested, which matches the general finding that VRAM capacity, not raw compute, is what decides which models run on a given card. A used 3090 covers the identical model list as a 4090 at a fraction of the price.

Which 24GB card should I buy?

RTX 3090 (used, ~24GB): two generations old and out of production, but VRAM matters more than raw speed for local inference, so it fits the same model list as newer cards.
RTX 4090 24GB (~$2,755, discontinued): the fastest single-card option tested above, now scarce and priced accordingly.
AMD RX 7900 XTX 24GB (~$1,339): same 24GB ceiling, 960 GB/s bandwidth, runs well through llama.cpp's Vulkan/HIP backend without a full ROCm install.

Where to go from here

Need a 70B-class model? That needs about 48GB, see the Mac M4 Pro 48GB setup guide.
Coming from 16GB? The 16GB VRAM setup guide covers Gemma 3 27B and gpt-oss-20b.
Comparing cards? Best Local LLM for the RTX 3090 has a single-card deep dive.

Local models run better with more VRAM. CompareRTX GPUs on Amazonbefore you upgrade.(affiliate link. We may earn a commission at no extra cost. Disclosure)

Related guides

Local AI

Beginner11 min

Best Local LLM for 16 GB VRAM: Setup, Quantization and Real Speed

16GB of VRAM unlocks Gemma 3 27B, which needs about 15GB at Q4 and just fits, plus OpenAI's gpt-oss-20b with real headroom to spare. Here is how to install both, which quantization format to use, and measured tokens-per-second on the RTX 4060 Ti 16GB and RTX 5070 Ti.

Local AI

Intermediate12 min

Best Local LLM on Mac M4 Pro 48GB: Setup, Quantization and Real Speed

A 48GB M4 Pro Mac mini fits Llama 3.3 70B at Q4 quantization, about 48GB, right at the ceiling, and runs Ornith 1.0 35B or Gemma 3 27B with real headroom. Here is how to install both paths, which format to pick, and what's actually measured versus what isn't for this exact chip.