Quantization for LLMs
How a model’s billions of numbers are stored in fewer bits, from how a computer stores 0.1 up to GPTQ, AWQ, GGUF and FP4. Every idea has a widget you can click. Logarithms help for the parts about error.
Numbers in a computer
Quantization basics
quantization- Scale, round, clamp.zero-point- Symmetric vs asymmetric grids.granularity- Per-tensor, per-channel and per-group scales.outliers- Handling outliers by clipping.measuring-error- MSE, SQNR, perplexity and KL.what-to-quantize- Weights, activations and the KV cache.ptq-and-qat- PTQ vs QAT, and the straight-through estimator.