Local AI

Run AI on Your Own Hardware

No cloud. No subscription. No data leaving your machine. This section covers everything you need to run, configure, and fine-tune AI models locally โ€” from a $400 mini PC to a $2000 GPU workstation.

100% private

Your prompts and data never leave your machine. No API logs, no training on your conversations.

No subscription

Pay once for hardware and run unlimited queries for free. One GPU pays for itself in weeks.

Works offline

Run a full LLM on a plane, in a bunker, or on an air-gapped server. No internet required.

Where do you want to start?

Step-by-step guides

All local AI guides

Speridlabs Idol: Turn One Photo into a 3D Gaussian Splat, and How to Run It

Idol is an open image-to-3D model from Sperid Labs that turns a single photo into a 3D Gaussian splat, generating up to 1,048,576 Gaussians in under 2 seconds on a consumer GPU with 7.2 GB of VRAM. It runs locally through the idol-sdk Python package under Apache 2.0.

intermediate7 min

LFM3-Omni-2B: Liquid AI's 2.2B Any-to-Any Model, and How to Run It Locally

LFM3-Omni-2B is Liquid AI's 2.17 billion parameter any-to-any model: one checkpoint that reads text, images, video and audio, and answers in text or 24 kHz speech. Built for phones and laptops, it ships as a BF16 checkpoint and a GGUF build as small as 1.05 GB, and runs through llama.cpp, Ollama or a browser WebGPU demo.

intermediate7 min

SenseNova-Loop-9B-Preview: A Looped Multimodal Model for Long-Context Work, and How to Run It

SenseNova-Loop-9B-Preview is a 9.66 billion parameter multimodal model that accepts text and images and generates text. Its unusual design runs 32 distinct decoder layers through 56 layer-executions per forward pass by looping a shared block of weights twice, giving it deeper computation without a proportional increase in stored parameters.

intermediate8 min

How to Run WeVisOne-VL-4B Locally: Tencent's All-in-One Vision Model

WeVisOne-VL-4B is a 4.83 billion parameter vision-language model from Tencent's WeChat Vision team, built on Qwen3-VL, released under Apache 2.0. One model handles object detection, referring localization, counting, pointing, OCR, segmentation and promptable alpha matting, and it runs locally through a pip-installable package with either a transformers or vLLM backend.

advanced7 min

How to Run Mistral Large 4 (1T-A52B) Locally: The Hardware It Actually Needs

Mistral Large 4 is Mistral AI's largest release yet: a 1 trillion parameter multimodal mixture-of-experts model with 49 billion active parameters per token, a 1 million token context window, and FP8 weights split across 54 safetensors shards. Mistral's own vLLM example asks for 8 GPUs, so this is a datacenter model, not something a single workstation can load.

advanced8 min

How to Run Hy-SimulMT-30B-A3B-FP8 Locally: Tencent's Real-Time Speech Translator

Hy-SimulMT-30B-A3B-FP8 is Tencent Hunyuan's 30.39 billion parameter model for real-time Chinese-English speech translation, with an FP8-quantized mixture-of-experts core that cuts the weight footprint roughly in half next to the BF16 release. It runs through Tencent's own SimulInfer server on an NVIDIA GPU with vLLM 0.31 or newer, and streams translated text and cloned speech while the speaker is still talking.

advanced8 min

How to Run Chroma1-HD Locally: An Open Base Model for Text-to-Image

Chroma1-HD is an 8.9 billion parameter text-to-image model built on FLUX.1-schnell and released under a fully permissive Apache 2.0 license. It runs through Hugging Face Diffusers on an NVIDIA GPU, and its real selling point is that it's meant as a clean, unrestricted base for finetuning rather than a finished chatbot-style product.

intermediate7 min

How to Run Nemotron 3.5 Parse Locally: NVIDIA's 630M Document Parsing Model

Nemotron 3.5 Parse is NVIDIA's 630.8 million parameter vision-language model built to turn document images into structured text with layout classes, bounding boxes, and reading order. It runs through Hugging Face Transformers or vLLM, needs roughly 1.3 GB of bfloat16 weights, and hits 15.42 pages per second in NVIDIA's single-pass benchmark on an H100.

intermediate7 min

Models to run locally

22 models

Every model is free to download and run. Requirements listed are for the Q4 quantized version โ€” the best balance of quality and size.

Light

Llama 3.1 8B

Meta ยท Llama 3.1 Community

Meta's 8B workhorse with a massive 128K context window. Punches well above its weight class for everyday coding and chat tasks, runs comfortably on any consumer GPU with 6 GB VRAM, and delivers near-70B quality on many benchmarks.

Min VRAM

5 GB

Context

128K

codingchatanalysis

Run with Ollama

ollama run llama3.1:8b
GiantBest all-around

Llama 3.3 70B

Meta ยท Llama 3.3 Community

Meta's flagship open-weight model and the go-to choice for users who want GPT-4-class performance without sending data to the cloud. Consistently tops open-source leaderboards. Needs ~48 GB VRAM at Q4_K_M โ€” a dual-GPU rig or Apple Silicon with 48 GB unified memory.

Min VRAM

48 GB

Context

128K

codingchatanalysiscreative writing

Run with Ollama

ollama run llama3.3:70b
LightFastest

Llama 3.2 3B

Meta ยท Llama 3.2 Community

The smallest production-quality Llama model, designed to run on laptops, Raspberry Pis, and other edge devices with minimal VRAM. Response latency is sub-second on modern CPUs, making it ideal for real-time assistants and local automation scripts.

Min VRAM

2 GB

Context

128K

chatedge deploymentsummarization

Run with Ollama

ollama run llama3.2:3b
LightMost popular

Qwen3 8B

Alibaba ยท Apache 2.0

One of the most-downloaded mid-size AI models on HuggingFace in 2026, with tens of millions of downloads. Qwen3 8B outperforms the previous Qwen2.5 14B on most benchmarks, supports hybrid thinking mode and tool calling, and runs on any GPU with 6 GB VRAM. Apache 2.0 license โ€” fully commercial.

Min VRAM

5 GB

Context

128K

codingchatreasoningtool use

Run with Ollama

ollama run qwen3:8b
HeavyVision capable

Qwen3.8-27B

Alibaba ยท Apache 2.0

Alibaba's newest Qwen release, published August 14, 2026 under Apache 2.0. A 27B dense model that adds native image and video understanding on top of text, with a 256K context window long enough for hour-scale video or entire codebases. At Q4 it needs about 18 GB of VRAM, so it fits a single 24 GB card with room to spare. Pull it with ollama run qwen3.8:27b.

Min VRAM

18 GB

Context

256K

codingchatanalysislong documents

Run with Ollama

ollama run qwen3.8:27b
Light

Mistral 7B v0.3

Mistral AI ยท Apache 2.0

The model that proved 7B parameters can match much larger models on reasoning tasks when trained carefully. Apache 2.0 licensed, so it is fully free to use commercially with no restrictions.

Min VRAM

4 GB

Context

32K

chatanalysissummarization

Run with Ollama

ollama run mistral:7b

Best local LLM for your hardware

Already know your VRAM budget, GPU or Mac? These pages list every model from the verified catalog that fits, with headroom, run commands and spec-sheet links.

Hardware picks

Full buyer's guide

These are affiliate links. We earn a small commission if you buy through them โ€” at no extra cost to you. Disclosure

ProPro pick

NVIDIA RTX 5090 32GB

32 GB GDDR7, 512-bit bus, 575W TDP, 21,760 CUDA cores

32 GB VRAM

Most VRAM on any consumer card โ€” the current flagship for 34Bโ€“70B local inference

Runs these models

7B13B34B70B Q4
32 GB VRAM is the largest on any consumer GPU โ€” fits 70B at Q4 on a single card
GDDR7 on a 512-bit bus delivers ~1.8 TB/s bandwidth, far ahead of the 4090
Current-generation Blackwell โ€” full CUDA and FP4/FP8 support for the newest quant formats
GPU shortage keeps street prices well above the $1,999 MSRP
575W TDP demands a 1000W+ PSU and serious case airflow
High-end

NVIDIA RTX 5080 16GB

16 GB GDDR7, 256-bit bus, 360W TDP, 10,752 CUDA cores

16 GB VRAM

Current-gen 16 GB card with GDDR7 speed for fast 7Bโ€“34B inference

Runs these models

7B13B34B Q470B Q2
GDDR7 bandwidth (~960 GB/s) beats the previous 4080 Super on token generation
16 GB handles 34B at Q4 without CPU offload
Blackwell architecture supports the newest FP4 quantization formats
Still only 16 GB โ€” 70B needs heavy Q2 quantization
Shortage pricing pushes it above the $999 MSRP
Mid-rangeBest value

NVIDIA RTX 5070 Ti 16GB

16 GB GDDR7, 256-bit bus, 300W TDP, 8,960 CUDA cores

16 GB VRAM

Best current-gen value โ€” 16 GB GDDR7 for 7Bโ€“34B at a mid-tier price

Runs these models

7B13B34B Q4
16 GB of GDDR7 at the lowest price in NVIDIA's current stack
Comfortably runs 13B at full precision and 34B at Q4
Lower 300W draw than the 5080 with most of the real-world inference speed
Shortage pricing sits above the $749 MSRP
16 GB cap means 70B is out of reach without heavy offload
BudgetBudget pick

NVIDIA RTX 4060 Ti 16GB

16 GB GDDR6, 128-bit bus, 165W TDP, 4,352 CUDA cores

16 GB VRAM

Best VRAM-per-dollar entry card for running 13B models locally

Runs these models

7B13B34B Q4
Most VRAM available at the $400โ€“450 price point
Runs 13B models at full FP16 precision without quantization
Low power draw keeps electricity costs minimal
Narrow 128-bit bus creates a memory bandwidth ceiling for large batches
Inference speed noticeably slower than mid-tier cards at 34B
Previous-gen (Ada) โ€” the current-gen RTX 5070 Ti offers more bandwidth for not much more money
Mid-rangeEditor's pick

NVIDIA RTX 4070 Super 12GB

12 GB GDDR6X, 192-bit bus, 220W TDP, 7,168 CUDA cores

12 GB VRAM

Sweet-spot performance for fast 7B and 13B token generation

Runs these models

7B13B
Fastest token-per-second in its price range for 7Bโ€“13B models
504 GB/s memory bandwidth keeps the GPU fed without stalls
Broad llama.cpp and Ollama support out of the box
12 GB VRAM cannot load 34B models without heavy quantization offload
Less VRAM than the cheaper RTX 4060 Ti 16GB
High-endLegacy

NVIDIA RTX 4080 Super 16GB

16 GB GDDR6X, 256-bit bus, 320W TDP, 10,240 CUDA cores

16 GB VRAM

High-throughput inference for 7Bโ€“34B models at serious speed

Runs these models

7B13B34B Q470B Q2
672 GB/s memory bandwidth delivers fast sustained token generation
16 GB VRAM comfortably handles 34B at Q4 without CPU offload
Excellent multi-user inference server card for home labs
Discontinued โ€” out of production, so new stock is scarce and overpriced
Superseded by the current-gen RTX 5080, which adds GDDR7 bandwidth at a similar price
70B models require heavy Q2 quantization and still push VRAM limits
Free weekly email

Weekly local AI drops

New models, what runs on your hardware, and the guides to set them up. One email a week, unsubscribe any time.

Affiliate disclosure

Hardware links on this page go to Amazon via our affiliate program. We earn a small commission when you buy through them. This does not change the price you pay or influence our recommendations โ€” every pick is chosen based on VRAM, value, and real performance for local AI workloads.