Rotations: QuaRot, SpinQuant and incoherence

Rotate the vector before quantizing it, rotate back afterwards. A good rotation smears one huge value across all the coordinates, so nothing sticks out and one scale fits everything. The rotation can be folded into the weights for free.

Key idea: Orthogonal rotations (Hadamard matrices) remove outliers; computational invariance lets them be merged into weights; learned rotations (SpinQuant); incoherence processing (QuIP, QuIP#)
Real world: JPEG doesn’t compress pixels directly. It first transforms each 8 × 8 block into frequencies, a reversible change of coordinates, and quantizes those instead, because in the new coordinates the information sits in a form that survives coarse rounding.
In an LLM: QuaRot multiplies a Llama model’s hidden states by a Hadamard matrix. The activation outliers that made 4-bit activations impossible disappear, and the whole model, weights, activations and KV cache, runs in 4 bits. The extra rotations are mostly merged into existing weight matrices.

The idea in one picture

An outlier is a problem because one coordinate is huge and the others are small, and a shared scale has to cover the huge one. But “which coordinate is huge” depends on the coordinate system. Rotate the vector, and its length stays the same while its energy can be spread across every coordinate.

A Hadamard matrix is a particularly good rotation for this. It’s a square matrix of $+1$ s and $-1$ s (scaled so lengths are preserved), where every output coordinate is a ± mix of all input coordinates. A vector with one huge entry comes out with every entry about the same size.

Try it

outlier size24

largest |value|24.0

step between levels3.43

error, quantize directly0.882

error, rotate → quantize → rotate back0.304

The outlier sets the scale, so the step between levels is 3.4 while ordinary values are around ±1: 12 of the 15 ordinary values round to 0 and their information is lost (error on them: 0.91). Switch to the rotated view.

Sixteen activations entering a layer; one of them is an outlier (red). Each version is quantized to 4 bits with a single scale. Dashed lines are the 15 available levels; purple marks show where each value lands. “Rotated back” shows the rotated version’s result in the original coordinates.

In the widget, quantizing directly wipes out the ordinary values: the outlier forces a step so large that they round to zero. Rotate first and the 16 values come out with similar magnitudes, the 4-bit levels are much finer relative to them, and after rotating back the error is several times smaller.

example · JPEG

A photo’s 8 × 8 pixel block is turned into 64 frequency coefficients by the discrete cosine transform, which is a rotation. Rounding coefficients is far less visible than rounding pixels, because the rotation concentrates the image’s structure into a few coefficients and turns the rest into small numbers that can be rounded hard. Undo the rotation and you get the picture back, slightly blurred. Quantizing in a well-chosen rotated basis is a decades-old trick; QuaRot brings it inside a transformer.

example · a hidden state with an outlier channel

A token’s 4,096-dimensional hidden state has one channel at 60 and the rest around ±1. Per-token 4-bit quantization would have a step of $60/7 \approx 8.6$ : everything except the outlier becomes 0. After a $4096 \times 4096$ Hadamard rotation, the outlier’s 60 is spread as about $60/\sqrt{4096} \approx 0.94$ into every coordinate, on top of the mixed ordinary values. The largest entry is now a few units, not 60, and the step shrinks by an order of magnitude.

Why it’s free: computational invariance

Inserting a rotation $Q$ and its inverse $Q^\top$ ( $Q^\top Q = I$ ) changes nothing mathematically:

y = W x = (W Q^\top)(Q x)

The trick is to choose where to put them so they cost nothing at run time. In a transformer, the residual stream passes through RMSNorm and into linear layers. RMSNorm only divides by the vector’s length, which a rotation doesn’t change, so a rotation of the residual stream commutes with it (after folding the norm’s per-channel weights into the next linear layer). QuaRot rotates the entire residual stream once, and merges $Q$ and $Q^\top$ into the weight matrices on either side: embeddings, attention and MLP projections, and the output head. Those weights are simply stored rotated.

A few rotations can’t be merged (for example inside the attention block, before the KV cache, and before the MLP’s down-projection). Those are applied online, but a Hadamard transform of size $d$ can be computed in $O(d \log d)$ steps without a matrix multiply, so the overhead is small.

Random, Hadamard, or learned?

QuIP introduced incoherence processing: multiply weights by random orthogonal matrices so that no single weight or direction is special, then quantize. It came with theoretical guarantees at 2 bits.
QuIP# replaced the random matrices with randomized Hadamard transforms (fast, and with better guarantees), and paired them with lattice codebooks: see 2 bits and below.
QuaRot applied Hadamard rotations to activations and the KV cache as well as weights, enabling 4-bit everything (W4A4KV4).
SpinQuant noticed that different random rotations give noticeably different quality, and learned the rotation instead, optimising it on a small calibration set while keeping it exactly orthogonal. It narrows the gap to 16-bit further.

The same idea now shows up in low-precision training: the NVFP4 pretraining recipe uses random Hadamard transforms on some gradients to tame outliers before quantizing them to 4 bits.

Where this shows up

The outliers rotations remove: outliers and clipping, LLM.int8() and SmoothQuant.
Rotations plus codebooks for 2-bit weights: 2 bits and below.
4-bit activations and KV cache: quantizing the KV cache.

going deeper

Hadamard matrices

The Sylvester construction builds Hadamard matrices of size $2^k$ recursively:

H_1 = [1], \qquad H_{2n} = \begin{bmatrix} H_n & H_n \\ H_n & -H_n \end{bmatrix}, \qquad \tfrac{1}{\sqrt n} H_n \text{ is orthogonal}

For a vector $x$ with a single non-zero entry $x_j = a$ , every entry of $\tfrac{1}{\sqrt n} H_n x$ has magnitude $|a| / \sqrt n$ . More generally, a randomized Hadamard transform $\tfrac{1}{\sqrt n} H_n \operatorname{diag}(\pm 1)$ makes the largest entry close to the vector’s RMS with high probability: the vector becomes incoherent. Sizes that aren’t powers of two use Kronecker products of a power-of-two Hadamard with a small known Hadamard matrix.

Computational invariance

For a linear layer $y = Wx$ fed by RMSNorm (with its scale folded into $W$ ) and an orthogonal $Q$ :

\operatorname{RMSNorm}(Qx) = Q \operatorname{RMSNorm}(x), \qquad W x = (W Q^\top)(Q x)

so storing $W' = W Q^\top$ and rotating the residual stream by $Q$ at the embedding gives an identical model. This observation is from SliceGPT (Ashkboos et al., 2024) and underpins QuaRot and SpinQuant.

SpinQuant’s optimisation

SpinQuant parameterises rotations $R$ on the Stiefel manifold (orthonormal matrices) and minimises the quantized network’s loss on calibration data with Cayley SGD, which keeps $R^\top R = I$ exactly at every step. Only the rotations are learned; the weights stay fixed.

In papers

QuIP: 2-Bit Quantization of Large Language Models With Guarantees (Chee et al., 2023).
QuIP# (Tseng et al., 2024): randomized Hadamard incoherence and E8 lattice codebooks.
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (Ashkboos et al., 2024).
SpinQuant: LLM quantization with learned rotations (Liu et al., 2024).