Measuring the damage: MSE, SQNR, perplexity and KL

Error in one tensor is easy to measure, but what matters is how the whole model behaves. KL divergence from the original model is the most sensitive single number; real evaluations are the final word.

Key idea: Tensor-level error (MSE, SQNR, about 6 dB per bit) vs model-level damage (perplexity, KL divergence from the original, top-token agreement, task evals)
Real world: Compressing a song to MP3: you can measure the difference in the waveform, but what matters is whether listeners can tell. The same file can be measurably different and audibly identical.
In an LLM: Llama 3 8B at Q4_K_M has perplexity 6.38 instead of 6.23, about 2.4% worse. Its KL divergence from the original is 0.028 nats per token. Below about 3.5 bits per weight both numbers rise steeply.

Level 1: one tensor

The simplest measure compares each weight with its quantized version: the mean squared error (MSE). Since MSE depends on how big the weights are, it is often reported as a ratio, the signal-to-quantization-noise ratio (SQNR), in decibels:

\text{SQNR} = 10 \log_{10} \frac{\text{average of } x^2}{\text{average of } (\hat x - x)^2}

Every extra bit halves the step size, which divides the noise power by 4: about 6 dB per bit. An 8-bit tensor has roughly 24 dB more SQNR than a 4-bit one.

Tensor error is useful for comparing methods on one layer, but a model is not just its tensors. Some layers are far more sensitive than others, errors in one layer feed into the next, and a small error on an important weight can matter more than a large error on an unimportant one.

Level 2: the model’s predictions

Perplexity measures how well the model predicts real text: lower is better (likelihood). Comparing the quantized model’s perplexity with the original’s is the traditional quality check.

KL divergence compares the two models directly. For each token of a test text, compute the original model’s next-token distribution $P$ and the quantized model’s $Q$ , and average $\mathrm{KL}(P \,\Vert\, Q)$ . It is zero if the quantized model is identical and grows as its predictions drift. It is more sensitive than perplexity, because errors in opposite directions can cancel out in perplexity but not in KL. llama.cpp reports it with llama-perplexity --kl-divergence, along with how often the two models agree on the top token.

Try it

SQNR of a tensor: signal power ÷ noise power, in decibels

absmax quantizationslope of 6 dB per bit, anchored at 8 bits

6.23 = original

IQ1_SIQ2_XXSIQ2_XSQ2_KIQ3_XXSQ3_K_MIQ4_XSQ4_K_MQ5_K_MQ6_KQ8_0F16

2.02.42.63.23.34.04.44.95.76.68.516.0

bits per weight →

typeQ4_K_M

bits per weight4.90

file4.58 GiB

perplexity6.383

KL from fp160.028

Q4_K_M is the popular sweet spot: about a third of the size of fp16, perplexity only 2.4% worse. Each tensor loses about 6 dB of SQNR per bit removed, but the model as a whole holds up until a cliff.

Top: one tensor of Gaussian weights. Bottom: Llama 3 8B quantized with llama.cpp and measured against the 16-bit original (numbers from the llama.cpp perplexity README; the IQ types, Q2_K and Q4_K_M used an importance matrix). Click or hover a bar.

Level 3: what people actually use it for

Perplexity and KL use ordinary text. The real question is whether the model still does its job: following instructions, reasoning through maths, writing code, handling long contexts. Those can degrade in ways average-token metrics miss:

Long generations compound small errors. A reasoning model writing 10,000 tokens has many more chances to go off track.
Rare skills (a less common language, a niche API) rely on a small set of weights that can be damaged without moving average perplexity much.
Small models suffer more at the same bit width, because each weight carries more of the model’s knowledge.

So the standard practice is: KL or perplexity to compare quantization settings quickly, then real benchmarks for the ones you might deploy.

example · an MP3

A 128 kbit/s MP3 discards most of the information in a CD-quality track, and its waveform is measurably different everywhere. Most listeners can’t tell. Push it to 32 kbit/s and everyone hears the artefacts. Waveform error grows steadily as the bit rate drops; what people perceive stays fine and then falls off a cliff.

example · choosing a GGUF

Using the numbers in the widget: Q8_0 (8.5 bits per weight) has KL 0.0014, practically identical to the original. Q4_K_M (4.9) has 0.028, a slight change most users won’t notice. IQ2_XXS (2.4) has 0.81 and perplexity 14 against 6.2: the model now often predicts something quite different. Each step down the list saves less memory and costs more quality. Choosing a quantization turns this into a rule of thumb.

Where this shows up

KL divergence from scratch: KL divergence.
The GGUF types in the chart: GGUF and llama.cpp.
Error that matters vs error that doesn’t, used by GPTQ and AWQ: GPTQ, AWQ.

going deeper

Definitions

\mathrm{MSE} = \frac1n \sum_i (\hat x_i - x_i)^2 \qquad \mathrm{SQNR} = 10 \log_{10} \frac{\sum_i x_i^2}{\sum_i (\hat x_i - x_i)^2}\ \text{dB}

For a uniform quantizer with step $s$ on a signal that spans the full range, the noise variance is $s^2/12$ and $s \propto 2^{-b}$ , so

\mathrm{SQNR} \approx 6.02\, b + C \text{ dB}

where $C$ depends on the signal’s distribution (it is negative for heavy-tailed data quantized with absmax, because most values use only part of the range).

Model-level metrics

For a test text $x_{1:T}$ , original model $P$ and quantized model $Q$ :

\mathrm{PPL}_Q = \exp\Big(-\frac1T \sum_t \log Q(x_t \mid x_{<t})\Big) \qquad \overline{\mathrm{KL}} = \frac1T \sum_t \mathrm{KL}\big(P(\cdot \mid x_{<t}) \,\Vert\, Q(\cdot \mid x_{<t})\big)

Perplexity can stay flat while KL rises, because the quantized model might gain probability on some actual tokens and lose it on others. KL counts every change in the distribution.

Layer-wise objective

Most post-training methods minimise not the weight error but the output error of each layer on calibration inputs $X$ :

\min_{\hat W} \Vert W X - \hat W X \Vert_2^2

This automatically cares more about weights that multiply large activations. GPTQ solves it with second-order information; AWQ approximates it by protecting channels with large activations.

Source

The Llama 3 8B numbers in the widget come from the KL-divergence table in the llama.cpp perplexity README (wikitext, context 512). The i-quants, Q2_K and Q4_K_M rows use an importance matrix.