Formats and methods at a glance

One page to come back to. Every number format, every quantization method and every GGUF type in this course, with its bits, its idea and where to read more.

Key idea: Reference tables: number formats, quantization methods, GGUF types, formulas and vocabulary
Real world: The conversion table on the inside cover of a cookbook: cups to grams, Fahrenheit to Celsius. You don’t read it; you look things up.
In an LLM: A model card says “W4A16, AWQ, group size 128” or a file is called IQ3_XXS. Find the row, and you know how many bits each weight takes and what was done to it.

Number formats

Format	Bits (sign · exponent · mantissa)	Largest value	Notes	Page
FP32	1 · 8 · 23	$3.4 \times 10^{38}$	master weights, optimizer states	floating point
BF16	1 · 8 · 7	$3.4 \times 10^{38}$	FP32’s range, less precision; default for training	float formats
FP16	1 · 5 · 10	65,504	more precision, small range; needs loss scaling	low-precision training
FP8 E4M3	1 · 4 · 3	448	weights and activations	FP8 and FP4
FP8 E5M2	1 · 5 · 2	57,344	gradients	FP8 and FP4
FP4 E2M1	1 · 2 · 1	6	only 0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6	FP8 and FP4
INT8	8, two’s complement	127	−128…127	bits and bytes
INT4	4, two’s complement	7	−8…7	bits and bytes
NF4	4, lookup table	1 (× scale)	16 levels at normal quantiles	NF4

Block formats

Format	Elements	Block	Scale	Bits per value
MXFP4	E2M1	32	E8M0 (power of two)	4.25
NVFP4	E2M1	16	E4M3, plus FP32 per tensor	4.5
MXFP8	E4M3 or E5M2	32	E8M0	8.25
INT4, group 128	INT4	128	FP16	4.125 (4.25 with zero-points)
NF4 + double quant	NF4	64	FP8, with FP32 per 256 blocks	≈ 4.127

Methods

Method	What it does	Weights / activations	Needs	Page
Round-to-nearest (RTN)	scale, round, clamp	any	nothing	what quantization is
LLM.int8()	outlier channels in 16-bit, rest INT8	W8A8	nothing	LLM.int8()
SmoothQuant	move activation range into weights	W8A8	calibration	SmoothQuant
GPTQ	round column by column, fix later columns	W4A16, W3A16	calibration	GPTQ
AWQ	scale up weights that meet large activations	W4A16	calibration	AWQ
NF4 / QLoRA	normal-quantile levels; train adapters on top	W4, compute in BF16	nothing	NF4
QuaRot / SpinQuant	rotate to remove outliers	W4A4KV4	calibration (SpinQuant trains rotations)	rotations
QuIP# / AQLM	codebooks for groups of 8 weights	≈ 2 bits	calibration + fine-tuning	2 bits and below
BitNet b1.58	ternary weights from the start	1.58-bit weights, INT8 activations	training from scratch	2 bits and below
KIVI	per-channel keys, per-token values	2-bit KV cache	nothing	KV cache
QAT	train with fake quantization	any	training	PTQ vs QAT

GGUF types

Type	Kind	Bits per weight (per block)	Llama 3 8B file	KLD vs FP16
Q8_0	legacy	8.5	7.96 GiB	0.0014
Q6_K	k-quant	6.5625	6.14 GiB	0.0055
Q5_K_M	k-quant mix	5.5 (+ Q6_K tensors)	5.33 GiB	0.011
Q4_K_M	k-quant mix	4.5 (+ Q6_K tensors)	4.58 GiB	0.031
IQ4_XS	i-quant	4.25	4.14 GiB (imatrix)	0.036
Q4_0	legacy	4.5	4.34 GiB	0.072
Q3_K_M	k-quant mix	3.4375 (+ Q4_K/Q5_K tensors)	3.74 GiB	0.10
IQ3_XXS	i-quant	3.0625	3.05 GiB (imatrix)	0.18
Q2_K	k-quant	2.625	2.96 GiB	0.45
IQ2_XXS	i-quant	2.0625	2.24 GiB (imatrix)	0.81
IQ1_S	i-quant	1.5625	1.88 GiB (imatrix)	2.2

File sizes and KLD from the llama.cpp perplexity README.

Names you’ll see

W4A16, W8A8, W4A4KV4: bits for weights (W), activations (A) and KV cache (KV).
g128, group size 128: one scale per 128 weights.
bpw: bits per weight, averaged over the whole model, scales included.
PTQ / QAT: quantize after training / train while simulating quantization.
Calibration data: a few hundred samples run through the model to measure activations.
imatrix: llama.cpp’s per-column activation statistics, used to weight errors.
_S, _M, _L: small, medium, large mixes of GGUF types within one file.
Dynamic vs static activation quantization: scales computed per token at run time, or fixed in advance from calibration.

going deeper

Formulas

Symmetric (absmax) quantization with $b$ bits, and asymmetric with a zero-point:

s = \frac{\max |x|}{2^{b-1} - 1}, \quad \hat x = s \cdot \operatorname{clamp}\Big(\Big\lfloor \frac{x}{s} \Big\rceil\Big) \qquad\qquad s = \frac{\max x - \min x}{2^b - 1}, \quad z = \Big\lfloor -\frac{\min x}{s} \Big\rceil, \quad \hat x = s\,(q - z)

Model and KV-cache memory:

\text{weights (bytes)} = \frac{\text{params} \times \text{bpw}}{8} \qquad \text{KV (bytes)} = 2 \cdot L \cdot h_{kv} \cdot d_{\text{head}} \cdot T \cdot \frac{b_{kv}}{8}

Per-group storage cost and signal-to-noise of uniform quantization:

\text{bpw} = b + \frac{b_{\text{scale}}}{g} \qquad \text{SQNR} \approx 6.02\,b + \text{const (dB)}