Post-training quantization vs quantization-aware training

Quantize a finished model in minutes with a little calibration data, or keep training it with the rounding in the loop so it learns to live with the grid. The straight-through estimator is the trick that makes the second possible.

Key idea: Post-training quantization (PTQ), calibration data, quantization-aware training (QAT), fake quantization and the straight-through estimator
Real world: Translating a speech after it’s written vs writing it knowing it will be translated. The second speaker avoids idioms that don’t survive translation, and the result is better.
In an LLM: Almost every GGUF, GPTQ and AWQ file is PTQ: done in minutes to hours on a finished model. Some labs ship QAT models, trained briefly with the 4-bit rounding in the loop, which hold up noticeably better at 4 bits or below.

Two ways to get a quantized model

Post-training quantization (PTQ) takes a trained model and quantizes it, without changing the training. The simplest version, round-to-nearest (RTN), just applies scales and rounding. Better PTQ methods use a small calibration set (a few hundred example texts) to see what the activations look like and choose roundings that hurt the outputs least: GPTQ and AWQ are PTQ methods.

Quantization-aware training (QAT) keeps training the model with quantization simulated in the forward pass, so the weights adapt to the grid. It costs a training run, but recovers much of what PTQ loses, especially at 4 bits and below.

The problem: rounding has no gradient

Training needs gradients. Rounding is a staircase: flat almost everywhere, with jumps at the edges. Its gradient is zero almost everywhere, so if you differentiate straight through $\operatorname{round}(w / s)$ , no learning signal reaches the weights.

The straight-through estimator (STE) is a deliberate cheat: in the forward pass, use the rounded weights; in the backward pass, pretend rounding was the identity and pass the gradient straight through to the full-precision weights. The model keeps a full-precision copy of every weight, nudges it with these gradients, and re-rounds it each step. When a weight drifts far enough, it crosses into a neighbouring grid value.

float weight w0.33

forward: q(w) = round(w ÷ 0.25) × 0.25

backward: the slope dq/dw

rounding (forward)true slope: 0 everywhere, undefined at the jumpsstraight-through estimate: 1 inside the range

Forward, w = 0.33 is used as 0.25. The real slope of the staircase here is 0, so an honest gradient would tell training “nothing you do to w matters”. The straight-through estimator pretends the slope is 1, so w keeps moving until it crosses a step and its rounded value changes.

Left: the forward pass rounds the weight to the grid (staircase) instead of using it directly (dashed). Right: the slope of that staircase, which is what training needs, and the slope the straight-through estimator uses instead.

Try it

The next widget shows two weights. Press Train with the grid to watch the straight-through estimator at work. The loss valley is tilted because the two inputs are correlated, so the best grid point is not always the one nearest the best full-precision weights.

how correlated the inputs are0.85grid step0.30

step 300 / 300

full-precision bestround each weight (PTQ)QAT: best rounded point foundQAT’s float weight (solid trail)best grid point

loss, rounding0.0578

loss, QAT0.0086

best possible on grid0.0086

Rounding each weight on its own lands on the nearest dot, but the tilted valley means another dot has lower loss. Training through the grid finds a better one because it follows the loss, not the distance. GPTQ gets a similar effect without training, by correcting each weight’s rounding error using the others.

Each axis is one weight. The ellipses are contours of the loss around the best full-precision weights (blue). Dots are the combinations a low-bit format allows.

example · writing for translation

A speech written in English and translated afterwards loses its puns, idioms and rhythm: that’s PTQ. A speaker who knows it will be translated chooses phrases that survive the translation, so the translated version is better, even if the English is a bit plainer: that’s QAT. The model learns weights that still work after rounding, not just weights that are best before it.

example · choosing a grid point

In the widget, with correlated inputs and a coarse grid, rounding each weight independently lands on the nearest dot, with loss 0.058. Training through the grid with the straight-through estimator wanders between dots and finds one with loss 0.009, the best on the grid. Rounding each weight on its own ignores that the two weights work together; training doesn’t. In a real layer with thousands of correlated inputs, this effect is what separates naive rounding from good methods.

In between: smarter PTQ

Full QAT means training a large model again, which is expensive. Several PTQ methods capture part of the benefit cheaply, by solving the same “which grid point” problem one layer at a time:

GPTQ rounds weights one column at a time and adjusts the remaining weights to cancel each rounding error, using second-order (Hessian) information from calibration data: GPTQ.
AWQ rescales the weights that multiply large activations so they are rounded more finely: AWQ.
AdaRound and similar methods learn whether to round each weight up or down, layer by layer.

And there is a popular way to fine-tune on top of a quantized model without QAT at all: QLoRA freezes the 4-bit base model and trains small 16-bit adapters next to it: NF4 and QLoRA.

Where this shows up

Layer-wise error correction: GPTQ.
Fine-tuning a quantized model: NF4 and QLoRA.
Models trained from the start with ternary weights: 2 bits and below.
Low-precision training more generally: training in low precision.

going deeper

Fake quantization and the STE

QAT inserts a fake-quantization operator in the forward pass:

\hat w = s \cdot \operatorname{clamp}\Big(\Big\lfloor \frac{w}{s} \Big\rceil, q_{\min}, q_{\max}\Big)

The straight-through estimator replaces its derivative with the identity inside the clipping range and zero outside:

\frac{\partial \hat w}{\partial w} \approx \mathbb{1}\big[q_{\min} \le w / s \le q_{\max}\big]

The update $w \leftarrow w - \eta\, \nabla_{\hat w} \mathcal{L}$ is applied to the latent full-precision weights. LSQ (learned step size quantization) also learns $s$ by gradient descent, with a similar straight-through derivative for $s$ .

With a coarse grid and a fixed learning rate, the STE update can bounce between two neighbouring grid points forever, because each one’s gradient points towards the other. Real QAT uses a decaying learning rate and many steps, and selects the final checkpoint by validation loss; the widget does the same at toy scale.

In papers

The PTQ/QAT framework, AdaRound and the STE in practice: Nagel et al., 2021.
Integer-only QAT for inference: Jacob et al., 2017.
QLoRA: Dettmers et al., 2023.
Ternary weights trained with an STE from scratch: Ma et al., 2024 (BitNet b1.58).