Run LLMs in Your Browser With WebGPU: No Install, No Server (WebLLM)

Everyone tells you to install Ollama or buy a GPU to run AI locally. There is a step before that with zero install and zero cost: run the model inside a browser tab. WebLLM, the in-browser inference engine from the MLC project, downloads a quantized model once, runs it on your GPU through WebGPU, and answers entirely on your machine. No Python, no server, no API key, and nothing ever leaves the tab. This guide shows the fastest no-code path, which models realistically fit, and the minimal code to drop a private chatbot into your own page.

What WebGPU is and why this works now

WebGPU is the browser API that gives JavaScript direct, low-level access to the GPU for general compute, not just graphics. That is the missing piece that makes real LLM inference in a tab practical: instead of crawling along on the CPU through WebAssembly alone, WebLLM compiles the model's math to GPU shaders and runs the heavy matrix work on your graphics hardware. The result is roughly native-class throughput for small models. WebGPU is no longer experimental in 2026: it ships enabled by default in Chrome and Edge (113+), Firefox, and Safari on recent macOS and iOS, so the audience that can run this is most desktop users and a growing share of mobile.

When the browser is the right call (and when it is not)

Reach for WebLLM when you want a try-it-now demo, a private chat widget on a page you control, or a way to test local AI before installing anything. Reach for Ollama when you need bigger models, long context windows, or an always-on local server that many apps share. Transformers.js sits beside WebLLM for smaller task models and embeddings inside a JavaScript app.

Requirements: browser, GPU, and memory

• A WebGPU-capable browser: Chrome or Edge 113+ is the safest bet on desktop; recent Firefox and Safari also work.
• A GPU with enough free memory to hold the model. A quantized 3B model needs roughly 2–3 GB; an 8B model wants closer to 6 GB of available GPU/unified memory.
• Patience for the first load only: the model weights download once (hundreds of MB to a few GB), then get cached so reloads are instant.
• To check support, open the browser console and confirm navigator.gpu is defined. If it is undefined, WebGPU is off or unsupported and WebLLM cannot run.

Fastest path: run a model with zero code

You do not need to write anything to feel this work. The MLC team hosts a full WebLLM Chat demo. Open it, pick a model from the dropdown, and the browser downloads the weights and starts a chat session that runs entirely on your hardware. The first model load shows a progress bar while it pulls and compiles the weights; after that it answers locally. Watch your GPU usage spike during generation to confirm it is real on-device inference, not a hidden API call.

1. Open chat.webllm.ai (the hosted WebLLM Chat) in Chrome or Edge.
2. Pick a small model first, such as a 1.5B–3B instruct model, so the download is quick.
3. Wait for the one-time download and compile to finish (progress is shown).
4. Type a prompt. Responses stream token by token, generated on your GPU.
5. Reload the page: the model is now cached, so it loads in seconds with no re-download.

Which models realistically run in-browser

WebLLM ships a registry of pre-compiled, quantized models you select by id. The practical ceiling in a tab is around 8B parameters quantized, because everything has to fit in available GPU memory alongside the browser itself. Smaller is smoother. Good starting points come from the Llama, Qwen, Phi, Gemma, and Mistral families that WebLLM pre-builds:

• A 1B–3B instruct model (for example a Llama 3.2 3B or a small Qwen instruct build) — fastest, lowest memory, ideal for a first demo or a low-end laptop.
• Phi-3.5-mini class models — strong reasoning for their size, comfortable on a mid-range GPU.
• An 8B model such as a Llama-3.1-8B build — the realistic top end; great quality but needs ~6 GB of free GPU memory and a slower first load.
• You never type the exact id from memory: read the live list from `webllm.prebuiltAppConfig.model_list` and copy the precise model_id (it includes a quantization suffix like q4f16_1 or q4f32_1).

Embed WebLLM in your own page

To put a private chatbot on a page you control, install the package and create an engine. WebLLM is OpenAI-compatible, so if you have ever called the OpenAI SDK the chat code will look familiar. Install it first:

Create the engine with a model id and an initProgressCallback so you can show a loading bar during the one-time download. The factory returns an engine that exposes an OpenAI-style chat.completions API:

import { CreateMLCEngine } from "@mlc-ai/web-llm";

// Copy the exact id from webllm.prebuiltAppConfig.model_list
const selectedModel = "Llama-3.1-8B-Instruct-q4f32_1-MLC";

const engine = await CreateMLCEngine(selectedModel, {
  initProgressCallback: (p) => {
    // p.progress is 0..1 during the first download + compile
    console.log(p.text, Math.round(p.progress * 100) + "%");
  },
});

Stream chat completions in JS

Because the engine mirrors the OpenAI chat completions shape, you send a messages array and iterate the streamed deltas. This is the same pattern you would use against api.openai.com or a local Ollama /v1 endpoint — only here the tokens are generated inside the tab:

const messages = [
  { role: "system", content: "You are a helpful assistant." },
  { role: "user", content: "Explain WebGPU in one sentence." },
];

const chunks = await engine.chat.completions.create({
  messages,
  temperature: 0.7,
  stream: true,
});

let reply = "";
for await (const chunk of chunks) {
  reply += chunk.choices[0]?.delta.content || "";
  // render `reply` into your UI as it streams
}

First-load size and caching

The one cost users feel is the very first download. A small model is a few hundred MB; an 8B quantized model can be a couple of gigabytes. After that, WebLLM caches the weights in the browser using the Cache API by default, so reloads and repeat visits load from disk in seconds with no network. You can switch the cache to IndexedDB if you prefer, by setting the cache backend in the engine's app config. Either way, nothing about that cache leaves the device.

Performance and what makes it crash

• Speed scales with the GPU. A modern discrete card or Apple Silicon runs small models at a comfortable interactive pace; an old integrated GPU will be slow but usable on a 1B–3B model.
• Memory is the hard wall. If the model plus the browser exceed available GPU memory, the tab will fail to load the model or crash. Drop to a smaller model rather than fighting it.
• Bigger model, slower first token. Larger weights mean a longer download and a longer compile before the first response.
• Keep one model loaded at a time. Switching models reloads weights into GPU memory; do not try to hold two large models at once in a single tab.

The privacy pitch

The strongest reason to use this is also the simplest: nothing leaves the tab. The model weights download to the browser cache, and every prompt and response is computed locally on your GPU. There is no API endpoint, no logging server, and no account. For sensitive notes, draft text, or anything you would not paste into a cloud chatbot, in-browser inference is genuinely private by construction, and you can verify it by running the tab with your network disconnected after the model is cached.

Limits and when to graduate to Ollama

Be honest about the ceiling. In-browser models top out around 8B parameters quantized, context windows are smaller than what a native runner gives you, mobile and Safari support trail desktop Chrome, and the first cold-start download is unavoidable. WebLLM is the perfect on-ramp: zero install, real local AI, total privacy. The moment you want a 14B–70B model, a long context window, or an always-on local server that several apps share, that is the signal to install Ollama and move up. Many people start in the browser to see that local AI is real, then graduate to Ollama once they are sold.