- Key idea
- Match the format to your hardware and use: GGUF for laptops and CPUs, INT4 or FP8 for GPU serving, FP4 on Blackwell. Then measure against the original, on your own task.
- Real world
- Choosing a photo format: RAW for editing, high-quality JPEG for a print, a small JPEG for a message. The “best” format depends on the screen it’s going to and what you’ll do with it.
- In an LLM
- The same Qwen or Llama model is published as a dozen files: Q8_0, Q6_K, Q5_K_M, Q4_K_M, IQ3_XXS GGUFs for llama.cpp and Ollama; MLX 4-bit for Macs; AWQ, GPTQ and FP8 checkpoints for vLLM; NVFP4 for Blackwell. They hold the same model at different sizes and speeds.
Rules of thumb
- 8 bits is free. Q8_0, FP8 or INT8 weights are practically indistinguishable from 16-bit. If it fits, use it.
- 4–5 bits is the sweet spot. Q4_K_M, Q5_K_M, AWQ or GPTQ INT4 (group size 128), NVFP4: a third of the size of 16-bit with small, usually acceptable, loss.
- Below about 3.5 bits, think twice. Quality falls off quickly (see the GGUF scoreboard). A bigger model at 4 bits usually beats a smaller model at 8 bits of the same size, but a smaller model at 4 bits often beats a bigger model at 2.
- Weights first, then the KV cache. For long contexts or many users, the KV cache can outgrow the weights; FP8 KV is close to free.
- Hardware decides the format. A format only speeds things up if your hardware and software have kernels for it.
Try it
weights in each format, against your memory budget
→ 8-bit (MLX) / Q8_0
8.5 GB
Q6_K
6.6 GB
Q5_K_M
5.7 GB
4-bit (MLX) / Q4_K_M
4.8 GB
IQ4_XS
4.4 GB
Q3_K_M
3.9 GB
IQ3_XXS
3.3 GB
IQ2_M
2.8 GB
budget for weights 9.7 GBtotal memory 16 GB
Use 8-bit (MLX) / Q8_0 (8.5 bits per weight on average) with MLX or llama.cpp. It is the most precise option that leaves room for context. At 8 bits or more the loss is usually unmeasurable. In the same budget you could also run a 14B model at about 4.8 bits, or only a 8B model at 8.5 bits; the bigger model at 4 bits usually wins.
example · photo formats
A photographer keeps RAW files for editing, exports a high-quality JPEG for print and a small one for messaging. Nobody argues about which format is “best”; each suits a destination. The same goes for a model: a GGUF Q4_K_M for a laptop, FP8 for an H100 serving thousands of requests, NVFP4 on a Blackwell card.
example · one model, three machines
An 8B instruct model:
- MacBook with 16 GB: 8-bit (about 8.5 GB) fits, but the 4-bit MLX or Q4_K_M GGUF (about 4.6–4.9 GB) generates nearly twice as fast, since decoding speed is set by bytes read per token, and leaves room for a long context.
- Gaming PC with a 12 GB GPU: Q5_K_M or Q6_K GGUF in llama.cpp (5.3–6.1 GiB) for one user, or an AWQ INT4 checkpoint in vLLM to serve several at once.
- H100 serving a product: FP8 W8A8 with an FP8 KV cache in vLLM or SGLang, near-lossless, with half the weight memory and faster matrix multiplies than BF16.
By hardware
| Hardware | Typical choice | Tools |
|---|---|---|
| Mac (Apple silicon) | MLX 4-bit or 8-bit; GGUF Q4_K_M–Q8_0 | MLX, llama.cpp, Ollama, LM Studio |
| CPU, or GPU too small for the model | GGUF, partly offloaded to the GPU | llama.cpp, Ollama |
| NVIDIA RTX, one user | GGUF, or EXL2/EXL3 | llama.cpp, ExLlama |
| NVIDIA RTX/A-series, serving | AWQ or GPTQ INT4 (W4A16); FP8 on RTX 40 and later | vLLM, SGLang, TGI |
| H100 / H200 | FP8 W8A8 + FP8 KV; INT4 W4A16 when memory-bound | vLLM, SGLang, TensorRT-LLM |
| B200 / RTX 50 (Blackwell) | NVFP4 or MXFP4; FP8 where quality matters most | TensorRT-LLM, vLLM, SGLang |
Making a quantized model
Most popular models already have quantized versions on Hugging Face, so you rarely need to make your own. When you do:
# GGUF with llama.cpp: convert, collect an importance matrix, quantize
python convert_hf_to_gguf.py ./my-model --outtype f16 --outfile my-model-f16.gguf
llama-imatrix -m my-model-f16.gguf -f calibration.txt -o imatrix.dat
llama-quantize --imatrix imatrix.dat my-model-f16.gguf my-model-Q4_K_M.gguf Q4_K_M
# MLX on a Mac: 4 bits, one scale per 64 weights
mlx_lm.convert --hf-path ./my-model -q --q-bits 4 --q-group-size 64# FP8 or INT4 checkpoints for vLLM with llm-compressor
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier, QuantizationModifier
# FP8 weights, dynamic per-token FP8 activations: no calibration data needed
oneshot(model="my-model", output_dir="my-model-FP8",
recipe=QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"]))
# 4-bit weights with GPTQ, using a calibration set
oneshot(model="my-model", output_dir="my-model-W4A16", dataset="open_platypus", num_calibration_samples=512,
recipe=GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"]))# serve it; FP8 KV cache halves context memory
vllm serve ./my-model-FP8 --kv-cache-dtype fp8Library APIs change often; check each project’s current docs for exact arguments.
Checking you haven’t broken it
A quantized model can look fine in a quick chat and still be worse in ways that matter. In increasing order of effort:
- KL divergence against the original on some representative text: how far the next-token distributions moved. Cheap and very sensitive, as on the measuring the damage page.
llama-perplexity --kl-divergencedoes this for GGUF. - Perplexity on your own domain, not just Wikipedia text: code, another language, your documents.
- Task evals you care about, with an evaluation harness. Run the original and the quantized model on the same questions and compare them as a paired comparison, with error bars: a one-point drop on a 500-question benchmark is usually noise.
- Long generations. Reasoning models and long outputs can be more sensitive: small per-token errors have thousands of tokens to compound. A study of quantized reasoning models found 8-bit and 4-bit weight-only quantization close to lossless, but larger drops at lower bit widths, and larger on harder problems.
Where this shows up
- The formats in detail: GGUF, FP8, MXFP4 and NVFP4, AWQ, GPTQ.
- Memory arithmetic: how much memory a model needs, the KV cache.
- Comparing two models fairly: comparing models.
- Everything on one page: cheat sheet.
Why bigger-and-coarser usually wins
Dettmers & Zettlemoyer studied thousands of quantized models and found that, for a fixed number of total model bits, 4-bit precision gave the best zero-shot accuracy almost universally: halving the bits and doubling the parameters is a good trade down to about 4 bits, and a bad one below. With modern methods (GPTQ, AWQ, imatrix k-quants), the break-even has moved somewhat lower, but the shape holds.
Heavily trained models quantize worse
Models trained on many more tokens per parameter pack more information into each weight, and lose more when it’s rounded. Ouyang et al. fit scaling laws showing that low-bit quantization damage grows with the number of training tokens, so small models trained on tens of trillions of tokens can be more fragile at 4 bits than older models of the same size.
In papers
- The case for 4-bit precision: k-bit Inference Scaling Laws (Dettmers & Zettlemoyer, 2022).
- Low-Bit Quantization Favors Undertrained LLMs (Ouyang et al., 2024).
- Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models (Liu et al., 2025).