Local AI How-To Guides
Run AI models on your own hardware. These guides cover Ollama and LM Studio setup, picking a model that fits your VRAM, quantization, and getting usable speed.
More guides
Speridlabs Idol: Turn One Photo into a 3D Gaussian Splat, and How to Run It
Idol is an open image-to-3D model from Sperid Labs that turns a single photo into a 3D Gaussian splat, generating up to 1,048,576 Gaussians in under 2 seconds on a consumer GPU with 7.2 GB of VRAM. It runs locally through the idol-sdk Python package under Apache 2.0.
LFM3-Omni-2B: Liquid AI's 2.2B Any-to-Any Model, and How to Run It Locally
LFM3-Omni-2B is Liquid AI's 2.17 billion parameter any-to-any model: one checkpoint that reads text, images, video and audio, and answers in text or 24 kHz speech. Built for phones and laptops, it ships as a BF16 checkpoint and a GGUF build as small as 1.05 GB, and runs through llama.cpp, Ollama or a browser WebGPU demo.
SenseNova-Loop-9B-Preview: A Looped Multimodal Model for Long-Context Work, and How to Run It
SenseNova-Loop-9B-Preview is a 9.66 billion parameter multimodal model that accepts text and images and generates text. Its unusual design runs 32 distinct decoder layers through 56 layer-executions per forward pass by looping a shared block of weights twice, giving it deeper computation without a proportional increase in stored parameters.
How to Run WeVisOne-VL-4B Locally: Tencent's All-in-One Vision Model
WeVisOne-VL-4B is a 4.83 billion parameter vision-language model from Tencent's WeChat Vision team, built on Qwen3-VL, released under Apache 2.0. One model handles object detection, referring localization, counting, pointing, OCR, segmentation and promptable alpha matting, and it runs locally through a pip-installable package with either a transformers or vLLM backend.
How to Run Mistral Large 4 (1T-A52B) Locally: The Hardware It Actually Needs
Mistral Large 4 is Mistral AI's largest release yet: a 1 trillion parameter multimodal mixture-of-experts model with 49 billion active parameters per token, a 1 million token context window, and FP8 weights split across 54 safetensors shards. Mistral's own vLLM example asks for 8 GPUs, so this is a datacenter model, not something a single workstation can load.
How to Run Hy-SimulMT-30B-A3B-FP8 Locally: Tencent's Real-Time Speech Translator
Hy-SimulMT-30B-A3B-FP8 is Tencent Hunyuan's 30.39 billion parameter model for real-time Chinese-English speech translation, with an FP8-quantized mixture-of-experts core that cuts the weight footprint roughly in half next to the BF16 release. It runs through Tencent's own SimulInfer server on an NVIDIA GPU with vLLM 0.31 or newer, and streams translated text and cloned speech while the speaker is still talking.
How to Run Chroma1-HD Locally: An Open Base Model for Text-to-Image
Chroma1-HD is an 8.9 billion parameter text-to-image model built on FLUX.1-schnell and released under a fully permissive Apache 2.0 license. It runs through Hugging Face Diffusers on an NVIDIA GPU, and its real selling point is that it's meant as a clean, unrestricted base for finetuning rather than a finished chatbot-style product.
How to Run Nemotron 3.5 Parse Locally: NVIDIA's 630M Document Parsing Model
Nemotron 3.5 Parse is NVIDIA's 630.8 million parameter vision-language model built to turn document images into structured text with layout classes, bounding boxes, and reading order. It runs through Hugging Face Transformers or vLLM, needs roughly 1.3 GB of bfloat16 weights, and hits 15.42 pages per second in NVIDIA's single-pass benchmark on an H100.
How to Run Recursive Flint Locally: a 39.8B Coding Model You Can Actually Fit on a Mac
Flint is Recursive's 39.8 billion parameter mixture-of-experts model for coding agents, with only about 2.5 billion parameters active per token. The full BF16 checkpoint wants a datacenter GPU, but Recursive's own rec engine runs it on an Apple Silicon Mac with 48 GB of unified memory, and community GGUF quantizations bring it to llama.cpp as well.
How to Run Phonon-40M-en Locally: Gradium's 40M-Parameter Text-to-Speech Model
Phonon-40M-en is a 40 million parameter text-to-speech model from Gradium. It runs on CPU with a 74.6 MB quantized weight file, outputs 24 kHz mono audio, supports around 100 voices, and streams audio as it generates. License is CC BY-NC 4.0.
How to Run Timestral-209M Locally: Mistral's 209M-Parameter Time Series Forecasting Model
Timestral-209M is a 209 million parameter time series foundation model from Mistral AI. It runs zero-shot forecasting on CPU or GPU, accepts up to 32,768 historical time steps, and outputs 21 quantiles per prediction. It tops all three major zero-shot forecasting benchmarks as of October 2026.
How to Run Indra OCR Locally: a 1M-Parameter CPU Model That Hits 49 Pages a Second
Indra OCR is an open-weight document parsing model from Datalab that runs on CPU through ONNX Runtime, no PyTorch and no GPU required. It turns PDFs and images into markdown, plain text or JSON with layout boxes, and on Datalab's own benchmark it processes about 49 pages a second at 32 cores.
How to Run AuroraIMG-6M Locally: AuroraAI-Research's 6.82M-Parameter Text-to-Image Model
AuroraIMG-6M is a text-to-image model from AuroraAI-Research with just 6.82 million parameters, including its text encoder, denoiser and VAE. Released October 11, 2026 under Apache 2.0, it generates 64 by 64 pixel images from short English captions using latent flow matching, and runs comfortably on CPU with the diffusers library.
How to Run Maya-1 Locally: a 3.3B-Parameter Text-to-Speech Model with Voice Design
Maya-1 is an open-weight, 3.3 billion parameter text-to-speech model from Maya Research. Instead of picking from a fixed list of speakers, you describe the voice you want in plain English, add inline tags like <laugh> or <whisper> for delivery, and the model generates SNAC audio tokens that decode to 24 kHz speech. It is licensed under Apache 2.0 and needs 16 GB or more of VRAM to run at BF16.
How to Run Sopro V2 Turbo Locally: a 120M-Parameter Voice Cloning TTS Model
Sopro V2 Turbo is an open-weight text-to-speech model from Samuel Vitorino with 120 million parameters. It clones a voice from 5 to 20 seconds of reference audio, streams audio back in about 300 milliseconds on a laptop CPU, and speaks English, European Portuguese, French and German. Code and weights are Apache 2.0.
How to Run Kova TTS 1 Locally: a 1.25B-Parameter Voice Cloning Model for Narration
Kova TTS 1 is an open-weight, 1.25 billion parameter English text-to-speech model built on Llama 3.2 1B. It clones a voice from reference audio and a matching transcript, streams audio in under 200 milliseconds, and outputs 48 kHz mono audio. The license covers research and non-commercial use only.
How to Run Whistle Locally: Cactus's 16.9MB On-Device Speech Recognition Model
Whistle is a speech-to-text model from Cactus Compute that ships as a single 16.9 megabyte file and runs on CPU with no GPU and no internet connection. It transcribes seven languages, times every word, and shares its runtime with Cactus's Needle text model. Here is what it does and how to install it.
How to Run Antalia-2 Mini Locally: a 7.6M-Parameter Turkish TTS Model
Antalia-2 Mini is a small Turkish text-to-speech model from cloud0day3 built for one voice and one job: fast, accurate Turkish speech on ordinary hardware. It has 7.62 million parameters, outputs 48 kHz audio, and runs well faster than real time on CPU. Here is what it does and how to install it.
How to Run Supra2-IMG Locally: SupraLabs' 104M-Parameter Text-to-Image Model
Supra2-IMG is a 104.1 million parameter text-to-image model from SupraLabs, trained from scratch on 5.6 million synthetic images and released under Apache 2.0. Here is what it is, how it was trained, and how to run it from the single inference script on its Hugging Face page.
How to Run VisionHOPE Locally: PSRben's Self-Modifying Vision Backbone
VisionHOPE is a PyTorch vision backbone from PSRben that ships in Tiny, Small and Base sizes for image classification, object detection and semantic segmentation, released under the MIT license. Here is what its self-modifying architecture actually does, how the three sizes compare on ImageNet, COCO and ADE20K, and how to install and run it yourself.
Fish Audio S1-Mini: What It Is and How to Actually Try It
S1-mini is Fish Audio's 0.5 billion parameter text-to-speech model, a distilled sibling of the 4B S1 flagship that topped the HuggingFace TTS-Arena-V2 leaderboard. There is no documented local install path right now, so here is what the model actually is and where you can hear it.
How to Run SupraTTS-0.1-Beta Locally: A Tiny Glow-TTS Voice Model
SupraTTS-0.1-Beta is a 29.6 million parameter Glow-TTS model from SupraLabs, paired with a 14 million parameter HiFi-GAN vocoder, released under the MIT license. Here is what changed from its predecessor and how to run it from the files on its Hugging Face page.
How to Run Nemotron 3 Diarization Locally: Streaming Speaker Tagging in 99M Parameters
Nemotron 3 Diarization is a 99 million parameter speaker diarization model from NVIDIA that figures out who spoke when, in real time or offline, for up to 8 speakers. It runs on CPU, needs no GPU, and ships a C++ runtime for native deployment. Here is how to install it and what the streaming latency tradeoffs look like.
How to Run Freya-TTS Locally: A 183M-Parameter Turkish Text-to-Speech Model
Freya-TTS (FreyaTTS-small) is a 183 million parameter open-weight Turkish text-to-speech model built on a tokenizer-free flow-matching transformer. It needs under 2GB of VRAM and runs with a real-time factor around 0.10 on a consumer GPU. Here is what it costs to run and how to install it.
How to Run Audio8 ASR Infinite Locally: Streaming Speech Recognition That Never Stops
Audio8 ASR Infinite is a 4.09 billion parameter streaming speech recognition model from Edge0 that transcribes Chinese and English audio 24/7 with a rolling KV cache, so memory and latency stay flat no matter how long the recording runs. Here is what it costs in VRAM and how to run it.
How to Run TeleOCR Locally: One Model for Scanned and Photographed Documents
TeleOCR is a 1.42 billion parameter open-weight vision-language model from StarDoc-AI that parses both digital PDFs and camera-photographed documents with the same weights. Here is what it scores on, what it costs in VRAM, and how to run it.
How to Run Xing4.0-29B-A4B Locally: A 256K-Context MoE Model for Agent Tasks
Xing4.0-29B-A4B is a 31.22 billion parameter mixture-of-experts model from China Telecom's Xing series that only activates 4 billion parameters per token and natively handles a 256K token context. Here is what it costs in VRAM and how to actually serve it.
How to Run IndicF5 Locally: Voice Cloning for 11 Indian Languages
IndicF5 is a 0.35 billion parameter open weight text-to-speech model from AI4Bharat that clones a voice from a short reference clip and speaks the result in any of 11 Indian languages. Here is what it takes to install it and generate your first clip.
How to Run Fish Audio S2 Pro Locally: Multilingual TTS with Inline Emotion Control
Fish Audio S2 Pro is a 4.56 billion parameter text-to-speech model that covers 80+ languages and lets you control prosody and emotion with plain text tags embedded in the input. Here is what the model costs in VRAM and how to install it.
How to Run ZDTaichu5.0-9B Locally: A 9.79B Vision-Language Model Built for Spatial Reasoning
ZDTaichu5.0-9B is a 9.79 billion parameter open weight vision-language model from TaichuAI that pairs a Qwen3.5-9B backbone with an NVIDIA C-RADIOv4-H vision encoder. It reads images, multiple images, and video, and leads its size class on spatial reasoning and agent benchmarks. Here is what it takes to run it and how to send it your first image.
How to Run Qwen3-TTS-12Hz-1.7B-VoiceDesign Locally: Voice Design From a Text Description
Qwen3-TTS-12Hz-1.7B-VoiceDesign is a 1.92 billion parameter open weight text-to-speech model from Qwen that builds a voice from a written description instead of a reference clip, streams audio in as little as 97ms, and covers 10 languages. Here is what it takes to run it and how to design your first voice.
How to Run YuE2-3B Locally: Open Source Music Generation That Beats Suno v5 on Benchmarks
YuE2-3B is a 3.63 billion parameter open source music model from m-a-p that writes an editable musical score alongside the audio, so you can hand it lyrics and a style prompt, get a full song back, then tweak the melody or harmony and regenerate. On the WildSongBench benchmark its best-of-8 score edges out Suno v5. Here is what it takes to run, what the benchmarks actually show, and how to generate your first song.
MiniCPM5-2B: Install and Run OpenBMB's 2B Model That Outscores 4B Rivals
MiniCPM5-2B is OpenBMB's 2.52 billion parameter model, released under Apache 2.0, and it averages 53.9 on OpenBMB's own benchmark set, ahead of Qwen3.5-4B's 51.1 despite having roughly half the parameters. Here is how to install it through vLLM, SGLang or Transformers, what the GGUF builds actually weigh, and what the benchmark table does and doesn't prove.
BERT Base Uncased: Specs, How to Run It, and Why It's Still Trending in 2026
google-bert/bert-base-uncased is a 110 million parameter masked-language model from 2018 that just re-entered Hugging Face's overall trending top 20. It has 48.8 million downloads, apache-2.0 licensing, and runs comfortably on a CPU. Here is how to run it, what it's actually still useful for, and why an eight-year-old model outranks this week's releases.
How to Run tencent/AuK Locally: Zero-Shot TTS, Voice Cloning and Speech Editing
tencent/AuK is a 1.5 billion parameter open source speech model that handles zero-shot voice cloning, instruction-guided TTS, pitch and speed editing, and audio source separation through one natural-language interface. It also depends on a separate 3 billion parameter Qwen2.5-Omni encoder at runtime, which changes the real hardware math. Here is what the specs actually are, how to install it, and how to run your first zero-shot clone.
Stable Diffusion 3.5 Medium: What's Different From SDXL and Flux
Stable Diffusion 3.5 Medium is Stability AI's 2B-parameter MMDiT-X text-to-image model. Here is how it differs from the SDXL and Flux coverage already on this site, how to install it with Diffusers, and the licensing catch that kicks in above $1M in revenue.
Qwen-Image-2512: The Text-to-Image Model That Renders Real Text
Qwen-Image-2512 is Alibaba's 20B-parameter text-to-image model, and its standout feature is generating readable English and Chinese text inside images, something SDXL and Flux still struggle with. Here is how to install it with Diffusers, what its Apache 2.0 license actually allows, and why the card does not publish a VRAM number.
How to Run Irodori-TTS Anime Locally: Japanese Voice Cloning and Voice Design
Irodori-TTS-v4.1-Anime is a free, MIT-licensed Japanese text-to-speech model fine-tuned on anime-style speech. Clone a voice from a short reference clip, or skip the clip entirely and describe the voice you want in words.
How to Run Breeze TTS 2 Locally: Voice Clone, Voice Design and Voice Direction
Breeze TTS 2 is a 3 billion parameter open-weight speech model that clones a voice from one reference clip and speaks English or Chinese in under 40 ms to first audio.
How to Run Kokoro-82M Locally: the Fastest Open-Weight TTS Model
Kokoro-82M is an 82 million parameter, fully Apache 2.0 text-to-speech model with 54 voices across 8 languages. At that size the weights are small enough to run on a CPU or almost any GPU, and installing it is one pip command.
Run Models
How to Install Ollama on macOS
Get Ollama running on a Mac in under 10 minutes: install via Homebrew or the native app, pull your first model, and confirm the local server is responding.
How to Install Ollama on Windows
Download the Windows installer, confirm PATH is set, verify the local API server is up, and understand when WSL2 gives you better GPU performance.
How to Run Llama 3 Locally with Ollama
Pull the Llama 3.3 model, run an interactive chat session, switch between the 8B and 70B variants, and understand the VRAM requirements for each size.
How to Pick the Right Local AI Model for Your Hardware
Match your GPU VRAM tier to concrete model recommendations, understand Q4/Q5/Q8 quantization, and use a simple rule of thumb to avoid out-of-memory crashes before you download.
Best GGUF Models to Run by VRAM Tier (8GB, 12GB, 16GB, 24GB, 48GB)
A current, no-nonsense map of which local models actually fit your GPU. Pick a VRAM tier, get the right GGUF model and the exact ollama pull command, plus the quant and context math that decides whether it loads or crashes.
Run LLMs in Your Browser With WebGPU: No Install, No Server (WebLLM)
Try local AI before you commit to Ollama or buy a GPU. WebLLM runs a real quantized model entirely inside a browser tab using WebGPU, with nothing sent to a server. This is the zero-install path, the model size limits, and the minimal code to embed it in your own page.
How to Use Ollama as a Drop-In OpenAI API
Point any OpenAI-compatible SDK or tool at your local Ollama server by changing one URL, and use OLLAMA_HOST to expose it on your network.
GGUF vs MLX vs NVFP4: Local AI Quantization Formats Explained
Three names keep showing up when you go to download a local model: GGUF, MLX, and the newer NVFP4. They are not interchangeable, and picking the wrong one wastes memory or leaves your hardware idle. Here is what each format is, which one your machine actually wants, and how to choose in ten seconds.
Local Apps & UIs
How to Set Up LM Studio for Local AI
Download LM Studio, grab a model from the built-in hub, start the local server, and have a working private AI chat running on your own hardware in about ten minutes.
Replace GitHub Copilot With a Local Coding Model (Qwen2.5-Coder + Continue.dev)
Drop the $10–19/month subscription and run your own coding assistant inside VS Code. This is the exact setup: Ollama plus Qwen2.5-Coder for chat and tab-autocomplete through the free Continue.dev extension, with honest notes on what you give up.
How to Set Up Open WebUI for Local AI
Run Open WebUI in Docker, connect it to Ollama, switch between models in one click, and create custom personas that save your best system prompts.
How to Run Mistral 7B Locally
Pull Mistral 7B with Ollama, understand when to use mistral-nemo instead, check your VRAM requirement, and tune a system prompt to get sharper answers.
Hardware & Builds
Best GPU for Running AI Locally in 2026
VRAM is the single spec that determines which models you can run. Here is how every major GPU stacks up for local LLMs and image generation in 2026.
How Much RAM Do You Need for Local AI?
RAM requirements for local AI depend on whether you are talking about GPU VRAM or system RAM, and which models you want to run. Here are the real numbers.
Mac vs PC for Local AI: Which Should You Choose?
Apple Silicon unified memory and PC discrete GPUs take completely different approaches to local AI. Here is what each does better and how to pick based on your workload.
How to Build a Local AI Workstation on Any Budget
Four budget tiers from $500 to $5,000+ with specific part recommendations and the real trade-offs at each level.
How to Set Up Local AI on 8GB of VRAM
An 8GB GPU runs the 7B to 8B model class comfortably at Q4 quantization, and Phi-4 14B just barely fits. Here is exactly what to install, which quantization format to pick, and real measured tokens-per-second so you know what to expect before you download anything.
How to Set Up Local AI on 12GB of VRAM
12GB of VRAM is the mixture-of-experts sweet spot: OpenAI's gpt-oss-20b needs about 12GB at Q4 and, because only 3.6B of its 20B parameters activate per token, generates at nearly 7B speed. Here is how to install it, which quantization it actually ships in, and cited real-world tokens-per-second.
Best Local LLM for 16 GB VRAM: Setup, Quantization and Real Speed
16GB of VRAM unlocks Gemma 3 27B, which needs about 15GB at Q4 and just fits, plus OpenAI's gpt-oss-20b with real headroom to spare. Here is how to install both, which quantization format to use, and measured tokens-per-second on the RTX 4060 Ti 16GB and RTX 5070 Ti.
How to Set Up Local AI on 24GB of VRAM
A 24GB card runs Ornith 1.0 35B, the top open-weight coding model on SWE-bench Verified at 75.6%, which needs about 20GB at Q4. Gemma 3 27B and everything smaller fit with headroom left for long context. Here is the install path, quantization choice, and cited RTX 3090 vs RTX 4090 speed.
Best Local LLM on Mac M4 16 GB: Setup, MLX vs GGUF and Real Speed
On a 16GB M4 Mac, the sweet spot is the 7B to 8B class: Qwen3 8B and Llama 3.1 8B both need about 5GB at Q4, leaving plenty of unified memory for macOS. Here is how to install Ollama or mlx-lm, which format to pick, and llama.cpp's own measured M4 speed.
Best Local LLM on Mac M4 Pro 48GB: Setup, Quantization and Real Speed
A 48GB M4 Pro Mac mini fits Llama 3.3 70B at Q4 quantization, about 48GB, right at the ceiling, and runs Ornith 1.0 35B or Gemma 3 27B with real headroom. Here is how to install both paths, which format to pick, and what's actually measured versus what isn't for this exact chip.
Model Spotlights
How to Run OpenAI's gpt-oss-20b Locally with Ollama
gpt-oss-20b is OpenAI's first open-weight model under Apache 2.0, and it runs on a 16 GB machine. Here is the exact, step-by-step setup with Ollama, from install to your first chat and API call.
Run DeepSeek V4 Locally: Which Size Actually Fits Your Hardware (Flash vs Pro)
DeepSeek V4 is a frontier open-weight release, but the headline size and the memory you actually need are not the same number. Here is what fits a single consumer GPU, what needs a workstation, and the realistic fallback for normal hardware.
Local OCR and Document AI: Self-Host baidu Unlimited-OCR for Private Document Parsing
Invoices, receipts, contracts and scanned PDFs are exactly the documents you do not want to ship to a per-page cloud API. baidu Unlimited-OCR is a brand-new open-weight model that parses whole PDFs in one pass, on your own hardware. Here is what it is, what it runs on, and how to go from a scanned page to structured JSON.
gpt-oss-20b vs gpt-oss-120b: Which Open Model Should You Run?
OpenAI shipped two open-weight models. One runs on a laptop, the other needs a serious GPU. Here is exactly how they differ and which one fits your hardware and use case.
Fine-Tuning
How to Fine-Tune an LLM with LoRA (Beginner Guide)
Understand what fine-tuning actually does, why LoRA makes it possible on consumer hardware, and what you need before you write a single line of training code.
How to Fine-Tune Llama 3 on Your Own Data with Unsloth
Install Unsloth, load Llama 3, format your JSONL dataset, run a LoRA training job, and export the result as a GGUF file ready for Ollama.
How to Prepare a Training Dataset for LLM Fine-Tuning
Build a clean JSONL dataset using the right conversation format, apply quality filters, hit the minimum example count, and use synthetic generation to fill gaps.
How to Run Your Fine-Tuned Model in Ollama
Take the GGUF file from your fine-tune, write a Modelfile, register it with Ollama, run it locally, and optionally push it to the Ollama registry.