LLM.int8() and SmoothQuant

Large models have a few activation channels that are huge in every token. LLM.int8() pulls them out and keeps them in 16-bit; SmoothQuant moves the problem from activations into weights, where it is easy. Both make 8-bit matrix multiplies work.

Key idea: Outlier features, mixed-precision decomposition (LLM.int8()), and difficulty migration from activations to weights (SmoothQuant, W8A8)
Real world: A choir where two singers are far louder than everyone else. Turn the microphone gain down to suit the two, and the rest of the choir disappears into the noise. Either give the loud singers their own microphones, or ask them to step back and everyone else to step forward.
In an LLM: In OPT-66B, a handful of the 9,216 hidden channels carry values far larger than the rest, in almost every token. With one INT8 scale per token, those channels set the step size and everything else rounds to a few levels. LLM.int8() gives them their own 16-bit path; SmoothQuant scales them down and folds the scale into the weights.

The problem: activations, not weights

Weights are fixed and can be quantized carefully once, offline. Activations, the numbers flowing between layers, change with every token and must be quantized on the fly. For fast INT8 matrix multiplies (W8A8: weights and activations in 8 bits), both sides have to be integers.

Small models quantize to W8A8 easily. Around 6.7 billion parameters, something changes: a few outlier features appear. These are specific hidden dimensions whose values are many times larger than the rest, and they show up in nearly every layer and nearly every token. They are not noise; removing them wrecks the model.

Recall from granularity that a scale is set by the largest value it covers. Activations are usually quantized per token (one scale per row), because per-channel activation scales don’t fit an integer matrix multiply. But every token contains the outlier channels, so every token’s scale is set by them.

Fix 1: give the outliers their own lane (LLM.int8())

LLM.int8() splits the multiply in two:

Find the input channels where any value exceeds a threshold (6 in the paper). That’s a tiny fraction of channels.
Multiply those channels in 16-bit.
Multiply everything else in INT8, with one scale per token row of $X$ and one per output column of $W$ .
Add the two results.

About 99.9% of the values go through INT8, and the accuracy matches 16-bit up to 175B parameters. The cost is speed: the split and the extra 16-bit multiply make it slower than plain 16-bit for small models. LLM.int8() is mainly a memory saver. It made 8-bit loading in bitsandbytes (load_in_8bit=True) possible.

example · the choir, with two microphones

The sound engineer gives the two loud singers their own microphones with the gain turned down, and sets the main microphone for everyone else. Nothing is lost, but there are two mixing desks to run.

example · one token through a layer

A token’s hidden vector has 4,096 values between −2 and 2, except channels 1,512 and 2,817, which are 45 and −60. With one INT8 scale for the whole row, the step is $60/127 \approx 0.47$ : every ordinary value is rounded to one of about 9 levels. LLM.int8() sends channels 1,512 and 2,817 (and the matching two rows of $W$ ) through 16-bit, and the remaining 4,094 channels get a step of $2/127 \approx 0.016$ , about 30 times finer.

Fix 2: move the difficulty into the weights (SmoothQuant)

SmoothQuant notices an asymmetry: activations are hard to quantize, weights are easy. And a linear layer has a free knob. For any positive numbers $s_j$ , one per input channel:

Y = XW = \big(X \operatorname{diag}(s)^{-1}\big)\big(\operatorname{diag}(s) W\big)

Divide activation channel $j$ by $s_j$ and multiply row $j$ of the weights by the same $s_j$ . In exact arithmetic nothing changes. But choose $s_j$ large for outlier channels and the activations become flat, while the weights, which had room to spare, absorb the range. The division of $X$ is folded into the previous layer (the LayerNorm’s weights), so at run time it costs nothing.

How much to move is set by the migration strength $\alpha$ :

s_j = \frac{\max |X_j|^{\alpha}}{\max |W_j|^{1-\alpha}}

$\alpha = 0$ leaves the activations alone and $\alpha = 1$ moves all the difficulty into the weights. The paper uses 0.5 for most models.

Try it

The same layer three ways: plain INT8, LLM.int8()’s split with the outlier channels in 16-bit, and SmoothQuant’s rebalancing.

activations X

X: INT8 step = max ÷ 127 = 0.664 · a typical channel uses about ±4 of the 127 levels

weights W

W: INT8 step = max ÷ 127 = 0.010 · a typical channel uses about ±67 of the 127 levels

plain INT8

output error3.63%

largest |x| in INT884.3

largest |w| in INT81.28

Two red channels are 20–30 times larger than the other 62, so they decide the INT8 step for the whole activation tensor, and every ordinary activation is squeezed into a few levels. The weights, meanwhile, are tiny and easy. Output error: 3.63%. Try the two fixes.

One linear layer, Y = XW, with X and W each quantized to INT8 with a single scale. Each column is the largest value in one of 64 input channels, drawn on the same scale for X and W; the dashed amber line is one INT8 step.

example · the choir, rearranged

Instead of a second microphone, the conductor asks the two loud singers to step back from the main microphone and everyone else to step a little closer. One microphone now picks up a balanced choir. That’s SmoothQuant: same output, one INT8 path, and the balance is set once, ahead of time.

example · what you get

With SmoothQuant, OPT-175B and other large models run in W8A8 with essentially no accuracy loss. Because both operands are INT8, the multiply runs on fast integer tensor cores: the paper reports up to 1.56× speed-up and half the memory of 16-bit. On GPUs with FP8 support (H100 and later), FP8 W8A8 has largely taken over this role, but the same smoothing trick is still used to tame activation outliers before quantizing them.

Where the outliers come from

Later work traced many outliers to attention mechanics: some heads learn to “park” attention on a fixed token (often the first one) when they have nothing useful to do, and the hidden dimensions that make this possible become huge (Bondarenko et al., 2023; Sun et al., 2024 call them “massive activations”). Methods that avoid outliers rather than work around them include rotations, which spread every channel’s energy across all channels, and training changes that stop the outliers forming in the first place.

Where this shows up

Why one large value ruins a scale: outliers and clipping.
Weight-only vs weight-and-activation quantization: weights, activations and the KV cache.
Protecting weights that meet large activations: AWQ.
Spreading outliers out with a rotation: rotations.

going deeper

LLM.int8() decomposition

With $X \in \mathbb{R}^{T \times d}$ (tokens × channels) and $W \in \mathbb{R}^{d \times o}$ , let $O = \{ j : \max_t |X_{tj}| \ge \tau \}$ be the outlier channels ( $\tau = 6$ ). Then

Y = \underbrace{X_{:,O}\, W_{O,:}}_{\text{fp16}} + \underbrace{\operatorname{diag}(c_X)^{-1} \cdot \big(Q(X_{:,\bar O})\, Q(W_{\bar O,:})\big) \cdot \operatorname{diag}(c_W)^{-1}}_{\text{int8, vector-wise}}

where $c_X$ holds one absmax scale per row of $X$ and $c_W$ one per column of $W$ , so each output element is rescaled by the product of its row’s and column’s scales.

SmoothQuant

For per-channel smoothing factors $s \in \mathbb{R}^d$ :

\hat X = X \operatorname{diag}(s)^{-1}, \quad \hat W = \operatorname{diag}(s)\, W, \qquad s_j = \frac{\max_t |X_{tj}|^{\alpha}}{\max_k |W_{jk}|^{1-\alpha}}

$\max_t |X_{tj}|$ is measured on calibration data. With $\alpha = 0.5$ , both $\hat X$ and $\hat W$ have channel maxima $\sqrt{\max|X_j| \cdot \max|W_j|}$ : the geometric mean, the same for both sides. $\hat X$ is then quantized per tensor or per token, $\hat W$ per channel.

In papers

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (Dettmers et al., 2022): outlier features emerging around 6.7B parameters, and vector-wise quantization with mixed-precision decomposition.
SmoothQuant (Xiao et al., 2022): W8A8 for models up to 530B via difficulty migration.
The same “scale a channel, fold the inverse elsewhere” identity reappears in AWQ and, generalised from diagonal scales to full rotations, in QuaRot and SpinQuant.