Per-tensor, per-channel and per-group scales

One scale for a whole matrix lets a single outlier ruin everything. Smaller blocks each get their own scale and their own grid, at the cost of storing more scales. Groups of 32 to 128 weights are the sweet spot for 4-bit LLMs.

Key idea: Granularity: per-tensor, per-channel (per-row), per-group/block scales, and the bits each scale costs
Real world: Setting the exposure for a photo of a room with a bright window. One exposure for the whole picture leaves the room black or the window white; exposing each region separately (HDR) keeps both.
In an LLM: A 4096 × 4096 layer with one scale is ruined by its single largest weight. With a scale per 128 weights, each of the 131,072 groups gets its own grid, and an outlier only coarsens the 127 weights next to it.

One scale, one grid

The absmax scale is set by the largest value in the set it covers. If that set is a whole weight matrix with 16 million weights, the single largest one sets the grid for all of them. LLM weight matrices typically have a few weights many times larger than the rest, so a per-tensor grid ends up far too coarse for the typical weight.

The fix is to quantize in smaller pieces, each with its own scale:

Granularity	One scale per	Scales in a 4096 × 4096 matrix
Per tensor	whole matrix	1
Per channel	row (output channel)	4,096
Per group (block)	$g$ consecutive weights in a row	$16{,}777{,}216 / g$

Try it

First, the map. Each scheme decides which numbers share a scale. Click through them:

weights W: rows = output channels, columns = input channels

★

one scale covers: 128 consecutive weights in a row
used by: GPTQ and AWQ 4-bit checkpoints.

scales in a 4096 × 4096 layer131,072

extra bits per weight0.125 bits (fp16 scale), 0.25 with a zero-point

numbers sharing the lit scale128

The ★ outlier sets the scale for its whole group: here 128 real numbers get a coarser grid because of it. Small blocks keep the damage local, at the cost of storing more scales.

A toy 16 × 64 matrix, coloured by which numbers share a scale. The ★ is one large outlier weight; its group is lit up by default. Hover or tap any square to light up its group. Row-wise blocks are drawn at a quarter of their real length (one square ≈ 4 weights); the numbers below use real sizes for a 4096 × 4096 layer.

Now see what the choice does to the error, on a matrix with three outliers:

group size32

bits4

error 0≥ 0.5outlier

weights per scale32

RMS error0.216

bits per weight, with scales4.500

8B model at this setting4.50 GB

Groups of 32 isolate each outlier to its own little block, and the scales cost only 0.50 extra bits per weight. Group sizes of 32 to 128 are the standard choice for 4-bit LLM weights. The ringed outliers themselves barely move (they set their group’s scale); it’s their neighbours that pay.

A 12 × 64 weight matrix with three outliers (ringed). Each block between the lines shares one scale. Colour shows how far each weight moved when quantized.

The price of a scale

Every scale is a number that has to be stored, usually in fp16. With a group of $g$ weights at $b$ bits each, the cost per weight is

\text{bits per weight} = b + \frac{16}{g}

Group size	4-bit weights become
32	4.5 bits
64	4.25 bits
128	4.125 bits

If each group also stores a minimum or zero-point, add another $16/g$ . This is why a “4-bit” GGUF Q4_0 file is really 4.5 bits per weight: blocks of 32 weights with one fp16 scale each. Formats like the GGUF k-quants and NVFP4 go further and quantize the scales themselves (to 6 or 8 bits) with a second, coarser scale on top: see GGUF and FP8, MXFP4 and NVFP4.

example · exposing a photo

A room with a sunlit window: expose for the room and the window is pure white; expose for the window and the room is black. Phones solve this with HDR: expose different regions separately and combine. Per-group scales are HDR for weights. Each region of the matrix gets the exposure (scale) that suits it.

example · Llama 3 8B at group size 128

Llama 3 8B has about 7 billion weights in its linear layers. At group size 128 that is about 55 million scales. Stored in fp16, they take 110 MB, about 3% of a 4-bit model. In exchange, 4-bit quantization error falls dramatically compared with per-channel scales. This is the default setting for GPTQ and AWQ checkpoints.

Which direction to group

Groups run along the input dimension of a weight matrix (along a row, for a matrix that multiplies a column vector). That is the direction a dot product sums over, so each output is the sum of a few groups, each multiplied by its own scale. The scales can be applied cheaply inside the matrix-multiply kernel.

For activations the natural granularities are different:

Per token: one scale per row of the activation matrix, computed on the fly as each token is processed.
Per channel: one scale per feature. Ideal for activations with outlier channels, but awkward for integer matrix multiplies, which is the problem SmoothQuant solves.

Where this shows up

What makes some weights so large: outliers and clipping.
Block formats with two levels of scales: GGUF and llama.cpp, FP8, MXFP4 and NVFP4.
Quantizing the scales themselves: NF4 and double quantization.

going deeper

Group-wise quantization

For weight matrix $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ and group size $g$ , each row $i$ is split into $d_{\text{in}} / g$ groups $G$ , each with its own scale:

s_{i,G} = \frac{\max_{j \in G} |W_{ij}|}{q_{\max}} \qquad \hat W_{ij} = s_{i,G(j)} \cdot \Big\lfloor \frac{W_{ij}}{s_{i,G(j)}} \Big\rceil

The matrix-vector product becomes a sum over groups:

y_i = \sum_{G} s_{i,G} \sum_{j \in G} q_{ij}\, x_j

Two-level scales

A block format with super-blocks stores a high-precision scale $d$ per super-block and low-precision sub-scales $\sigma_k$ per sub-block, so each weight’s effective scale is $d \cdot \sigma_k$ . GGUF’s Q4_K, for instance, uses super-blocks of 256 weights with 8 sub-blocks of 32, 6-bit sub-scales and minimums, and fp16 super-block values, for 4.5 bits per weight in total. NVFP4 uses an FP8 (E4M3) scale per block of 16 and an fp32 scale per tensor.

In papers

Dettmers & Zettlemoyer, 2022 (“The case for 4-bit precision”) found that a small block size and the choice of data type (float rather than integer) are the two things that most improve the quality of 4-bit models. GPTQ and AWQ papers report results with group size 128 as their main 4-bit setting.