GGUF vs MLX vs NVFP4: Local AI Quantization Formats Explained

When you go to download a local model, you do not pick a model so much as you pick a file format. The same Qwen3 or Gemma weights show up as GGUF on one page, MLX on another, and lately as NVFP4 on a third. The names look like trivia, but they decide whether the model fits your memory, how fast it runs, and whether your tooling can even load it. This guide explains the three that matter in 2026, plus the older GPU formats you will still bump into, and ends with a flowchart that picks one for you.

Why quantization exists at all

A model's weights are just numbers. Train it and each weight is typically a 16-bit float (fp16 or bf16), so an 8-billion-parameter model is roughly 16 GB before you load a single token of context. Quantization shrinks those numbers to fewer bits, 8, 6, 5, or 4 each, which cuts the file size and the memory footprint by half or more. You trade a small amount of accuracy for a large win in memory and speed. That trade is what lets a model that needs 16 GB at full precision fit on an 8 GB GPU at 4-bit. Every format below is a different way of doing that same shrink.

The bits primer: what Q4, Q5, Q6, Q8 actually mean

The number after the Q is the average bits stored per weight. Fewer bits means a smaller file and more quality loss. The rough rule: Q8 is near-lossless and roughly half the size of fp16, Q6 is very close to the original, Q5 is a solid middle, and Q4 is the popular default where the file is small enough to fit consumer hardware while the quality loss is still hard to notice on most tasks. Below Q4 (Q3, Q2) the loss starts to bite and the model gets noticeably worse.

GGUF: the run-anywhere default

GGUF is the format used by llama.cpp, and therefore by Ollama and LM Studio, which are built on it. It is the cross-platform default: one GGUF file runs on a CPU, an NVIDIA GPU, an AMD GPU, or Apple Silicon, and on Linux, Windows, or macOS. It supports the full K-quant range (Q2 through Q8 plus fp16) in a single, self-contained file that bundles the weights and the metadata. If you have ever run `ollama pull`, you have used GGUF without thinking about it. The trade-off is that GGUF is a generalist: it runs everywhere but is not the fastest possible option on any one platform.

MLX: Apple Silicon native

MLX is Apple's own array framework for machine learning on Apple Silicon. Its quantized format is built to exploit the unified memory architecture of M-series chips, where the CPU and GPU share one memory pool, so there is no copying weights back and forth across a bus. On a Mac, an MLX build of a model usually loads faster and runs at higher tokens per second than the same model as GGUF, because it is using the hardware the way Apple intended rather than going through a cross-platform layer. The catch is right there in the name: MLX runs only on Apple Silicon. There is no MLX path on an NVIDIA box or a Windows PC.

NVFP4: NVIDIA Blackwell's hardware-native 4-bit float

NVFP4 is a 4-bit floating-point format introduced by NVIDIA for its Blackwell architecture, which is the RTX 50-series generation (5090, 5080, 5070 Ti). The key word is floating-point. Most 4-bit quantization, including the int4 schemes below, stores weights as 4-bit integers. NVFP4 stores them as a tiny 4-bit float with a scaling factor, and Blackwell GPUs have hardware that natively understands that format. The payoff is that a 4-bit float keeps more of the original number's dynamic range than a 4-bit integer, so accuracy holds up better at the same bit count, while the dedicated hardware runs it fast. NVFP4 is only meaningful if you own a Blackwell card; on older NVIDIA generations there is no hardware support for it.

GPTQ, AWQ, EXL2: the older GPU formats

Before GGUF became the default and before NVFP4 existed, GPU users ran int4 formats like GPTQ, AWQ, and EXL2. They are still around and still good on NVIDIA cards, especially when the whole model fits in VRAM with no CPU offload. GPTQ and AWQ are int4 quantization schemes that squeeze a model onto a single GPU; EXL2 (from the ExLlama project) offers flexible mixed bit-rates for fast all-on-GPU inference. You will mostly meet these in vLLM and text-generation-webui setups. For a newcomer in 2026 they are not where you start, but if you find a model that ships only as GPTQ or AWQ, it will run fine on an NVIDIA GPU.

Side by side

Quality loss in practice: when would you actually notice

The honest answer for most people: at Q4 and above on a decent-size model, you usually will not notice in everyday chat, summarizing, or coding help. The gap between Q4 and Q8 is real but small, and it shows up most on the hardest tasks: long multi-step reasoning, precise math, or code where one wrong token breaks the program. The general guidance is to default to Q4 for the best size-to-quality balance, step up to Q5 or Q6 if you have the memory and your task is reasoning-heavy, and only reach for Q8 when you want the model as close to full precision as a quant gets. Going below Q4 is where degradation becomes obvious, so avoid Q3 and Q2 unless you are desperate to fit a model that is simply too big for your hardware.

How to find or convert a model into each format

You rarely need to quantize anything yourself; the community publishes pre-made files for the popular models. The fastest path is to search Hugging Face for the format you want, or just let Ollama handle GGUF for you.

• GGUF: easiest is ollama pull <model>, or download a .gguf from Hugging Face (look for repos tagged GGUF) and point Ollama or LM Studio at it.
• MLX: search the mlx-community organization on Hugging Face for a ready 4-bit build, or convert one yourself with mlx_lm.convert.
• NVFP4: look for NVFP4 checkpoints published for Blackwell, then serve them through vLLM or NVIDIA's TensorRT stack on an RTX 50-series card.
• GPTQ / AWQ / EXL2: search Hugging Face for the format tag; these are typically loaded in vLLM or text-generation-webui.

Where this is heading

The clear direction is hardware-native low-bit formats. For years quantization was a software trick layered on top of GPUs that only really understood fp16 and int8. NVFP4 flips that: the 4-bit float is something the Blackwell silicon was designed to compute on directly, which is why it can be both small and accurate. Expect future hardware generations to keep pushing native support for ever-lower-bit formats, narrowing the quality gap that used to make 4-bit a compromise. The practical upshot for you is that the format you pick will increasingly be dictated by the chip you bought, which is exactly what the flowchart below assumes.

TL;DR: pick a format in ten seconds

1. On a Mac (Apple Silicon)? Use MLX for speed, or GGUF via Ollama if you want the easiest start. Both work; MLX is faster.
2. On an NVIDIA Blackwell card (RTX 5090 / 5080 / 5070 Ti)? NVFP4 gives you the best 4-bit accuracy your hardware supports. GGUF still works and is simpler.
3. On an older NVIDIA GPU, AMD GPU, or running on CPU? Use GGUF. It is the run-anywhere default and the path of least resistance.
4. Only found the model as GPTQ, AWQ, or EXL2? Run it in vLLM or text-generation-webui on an NVIDIA GPU; no need to convert.
5. Not sure and just want it to work? Use GGUF through Ollama. It runs on everything and you can always switch to a faster format later.