AWQ: protect the weights that matter

A weight that meets a large activation does more damage when it’s rounded. AWQ finds those weights by looking at activations, and scales them up before quantizing so they get relatively finer levels, without any mixed precision.

Key idea: Salient weight channels chosen by activation size, per-channel scaling before quantization, a search over one exponent α
Real world: An invoice where every unit price is rounded to the nearest £10. For an item bought once, that’s fine. For screws bought 1,000 at a time at £0.40 each, rounding to £0 loses £400. Price them per box of 100 instead (£40 a box, 10 boxes) and the same rounding costs at most £50.
In an LLM: In a 4-bit Llama layer, about 1% of input channels see activations far larger than the rest. Keeping just those weights in 16-bit recovers most of the lost quality. AWQ gets the same effect while keeping every weight 4-bit: it multiplies those channels’ weights by a factor s and divides their inputs by s.

Not all weights are equal

The error a rounded weight causes in the output is its rounding error times the input it multiplies:

\text{output error from weight } j = (w_j - \hat w_j) \times x_j

So a weight that always meets a large input $x_j$ matters far more than one that meets a small input. AWQ (activation-aware weight quantization) measures this directly: run calibration text through the model, and call an input channel salient if its average activation magnitude is large. In LLMs a small fraction of channels, around 1%, stand out.

The paper’s first experiment is striking: keep just those 1% of weight channels in FP16 and quantize the rest to 4 bits, and most of the quality lost by round-to-nearest comes back. Choosing the 1% by weight size instead barely helps. It’s the activations that tell you what matters.

Scaling instead of mixed precision

Keeping some weights in 16-bit and the rest in 4-bit is awkward for hardware: kernels want every weight in a group stored the same way. AWQ uses the same identity as SmoothQuant instead. For a salient input channel $j$ :

w_j x_j = (s \cdot w_j) \cdot \frac{x_j}{s}

Multiply the weight by $s > 1$ before quantizing and divide its input by $s$ . The rounding error of $s \cdot w_j$ is about the same absolute size as before (the grid step barely changes as long as $s \cdot w_j$ isn’t the largest weight in its group), but after dividing by $s$ , its effect on the output shrinks by a factor of $s$ . The division of the input is folded into the previous operation, so it’s free at run time.

Push $s$ too far and $s \cdot w_j$ becomes the largest weight in its group. Then it sets the group’s scale, the grid gets coarser for every other weight, and the error goes back up. There is a sweet spot.

Try it

scale s for channel c3×1.00

how much each input channel adds to the output error

row 6 on its 4-bit grid: ticks are the 15 levels, red lines the rounding error

4-bit with scalingkeep c3 in 16-bit (mixed precision)

output error vs no scaling100%

keeping c3 in 16-bit10%

best s here×2.13

Without scaling, channel c3 (amber) causes most of the error even though its weights are ordinary: every tiny rounding mistake on them is multiplied by an activation 8× larger than usual. Drag s up.

A 4-bit layer: 256 output rows, 8 input channels, one scale per row. Channel c3 carries activations 8× larger than the others. Error is the expected squared error of the layer’s output, as a percentage of the error with no scaling.

example · pricing per box

The invoice rounds unit prices to the nearest £10. Screws cost £0.40 and you buy 1,000: rounded to £0, the invoice is £400 short. Re-list them as boxes of 100 at £40 a box, 10 boxes. Rounding £40 to £40 costs nothing here, and even a bad case (£45 → £50) is £5 a box, £50 in total. Same purchase, same rounding rule; you just moved a factor of 100 from the quantity to the price. That’s AWQ’s $s$ : move a factor from the activation into the weight, where the rounding happens.

example · one salient channel

In the widget, channel c3 sees activations 8× larger than the others. Without scaling it causes most of the layer’s output error, although its weights are ordinary. Scaling it by about 2.5 cuts the total error to under 40% of where it started, and every weight is still a 4-bit integer with one scale per group. Push past about ×5 and the layer becomes worse than with no scaling at all. Keeping c3 entirely in 16-bit (the dashed line) does better still, but that needs a second storage format inside the same layer.

Searching for s

AWQ doesn’t tune each channel’s $s$ separately. It uses one formula with one knob:

s_j = \bar{|x_j|}^{\,\alpha}

where $\bar{|x_j|}$ is channel $j$ ’s average activation magnitude on the calibration data. $\alpha = 0$ means no scaling; larger $\alpha$ scales salient channels more. AWQ tries a grid of values of $\alpha$ between 0 and 1 for each layer, quantizes, measures the layer’s output error on calibration data, and keeps the best. No gradients, no weight updates beyond the scaling.

That simplicity is a feature. Because AWQ never fits the weights to the calibration set (unlike GPTQ, which adjusts weights to cancel errors on it), it is less sensitive to which calibration data you use, and generalises well to other domains and to instruction-tuned and multimodal models.

In practice

AWQ is weight-only: weights in 4-bit (usually group size 128), activations stay in 16-bit (W4A16). It speeds up generation because decoding is limited by reading weights from memory: see what to quantize.
AWQ checkpoints are served by vLLM, SGLang, TGI and TensorRT-LLM. The original AutoAWQ library is no longer maintained; its functionality moved into llm-compressor.
AWQ and GPTQ can be combined: apply AWQ’s scaling first, then GPTQ’s error feedback. Many toolkits offer both as steps in one recipe.

Where this shows up

The same scaling identity used for activations: LLM.int8() and SmoothQuant.
Error feedback instead of scaling: GPTQ.
llama.cpp’s importance matrix also weights errors by activation size: GGUF.

going deeper

Why scaling helps

For a group of weights quantized with step $\Delta = \max_k |w_k| / q_{\max}$ , the error on weight $j$ is $\Delta \cdot \epsilon_j$ , where $\epsilon_j$ is the rounding error in units of the step (roughly uniform on $[-\tfrac12, \tfrac12]$ ). Its contribution to the output is $\Delta \cdot \epsilon_j \cdot x_j$ . After scaling,

Q(s \cdot w_j) \cdot \frac{x_j}{s} - w_j x_j = \Delta' \cdot \epsilon'_j \cdot \frac{x_j}{s}

If scaling one weight doesn’t change the group maximum, $\Delta' = \Delta$ and $\epsilon'$ has the same distribution as $\epsilon$ , so the expected error on that channel falls by a factor of $s$ . Once $s \cdot w_j$ exceeds the group maximum, $\Delta' = s|w_j| / q_{\max}$ grows with $s$ and every other weight in the group gets worse.

The search

AWQ minimises the output error of each linear layer on calibration inputs $X$ :

s^* = \arg\min_{s}\; \big\Vert Q\big(W \operatorname{diag}(s)\big)\big(\operatorname{diag}(s)^{-1} X\big) - W X \big\Vert, \qquad s = s_X^{\alpha},\; \alpha \in [0, 1]

with $s_X$ the per-channel average activation magnitude. The paper searches 20 values of $\alpha$ . Weight clipping (shrinking each group’s maximum slightly, as in outliers and clipping) is searched the same way.

In papers

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (Lin et al., 2023). Also introduces TinyChat, a 4-bit inference engine for edge devices.
SqueezeLLM (Kim et al., 2023) and SpQR (Dettmers et al., 2023) take the other route: keep the few sensitive weights in high precision as a sparse matrix next to a low-bit dense one.