Rotations and orthogonal matrices

Key idea

Orthogonal matrices (QᵀQ = I) keep lengths, angles and dot products; 2D rotation [[cos θ, −sin θ], [sin θ, cos θ]]; rotate pairs of dimensions; RoPE; Hadamard rotations spread outliers

Real world

Turning a paper map on the table to face north. Every distance and angle on the map is the same afterwards; only the direction it faces has changed, and turning it back undoes it exactly.

In an LLM

Llama, Mistral and Qwen rotate each query and key by an angle proportional to the token’s position (RoPE). Because rotations keep dot products, the attention score ends up depending only on how far apart two tokens are.

Turning without stretching

Most matrices stretch, squash or skew space, as in the transformations lesson. An Orthogonal matrix does none of that. Its columns all have length 1 and are at right angles to each other, and as a result it keeps:

• every length: $\lVert Q\mathbf{x}\rVert = \lVert\mathbf{x}\rVert$,
• every angle between two vectors,
• and therefore every dot product: $(Q\mathbf{a})\cdot(Q\mathbf{b}) = \mathbf{a}\cdot\mathbf{b}$.

It is also trivial to undo: the inverse is the transpose, $Q^\top Q = I$. In 2D the orthogonal matrices are exactly the rotations (determinant 1) and the reflections (determinant −1). The rotation by an angle $\theta$ is

Its first column is where $(1, 0)$ lands, a point at angle $\theta$ on the unit circle; the second column is the same turned another 90°.

Rotating many dimensions in pairs

In higher dimensions a simple way to build a rotation is to split the vector into pairs, $(x_1, x_2), (x_3, x_4), \dots$, and rotate each pair by its own angle with a 2 × 2 rotation. The full matrix is block-diagonal: 2 × 2 rotation blocks along the diagonal and zeros elsewhere. Because each block keeps lengths, the whole thing does.

RoPE: position as an angle

Attention scores are dot products of queries and keys, and on their own they know nothing about where the tokens are. Rotary position embedding (RoPE) fixes that with rotations:

1. Split each query and key into pairs of numbers.
2. Rotate pair $k$ of the token at position $m$ by the angle $m\,\theta_k$, where $\theta_k = 10000^{-2k/d}$: fast for the first pairs, very slow for the last.
3. Compute attention scores as usual.

Because a rotation keeps dot products, rotating the query by $m\theta$ and the key by $n\theta$ gives the same score as rotating only the key by $(n - m)\theta$:

So the score depends on the distance between the tokens, not on where they are in the text. Slide the positions, then press “move both by 5”:

The widget uses 4 pairs and a base of 100 so that every dial moves visibly. A real Llama head has 64 pairs (head size 128), with base 10,000 in Llama 2 and 500,000 in Llama 3, so its slowest pairs turn a full circle only over tens of thousands of positions or more. See RoPE for how it fits into the model and how context is extended by changing the base.

Rotations against outliers

Hidden states in large models have a few huge numbers (outliers) in fixed dimensions. That is bad for quantization: a shared scale must cover the outlier, so all the small numbers round to zero. Rotating the vector first keeps its length but spreads the outlier over every coordinate. A Hadamard matrix, all $\pm 1$ divided by $\sqrt{n}$, does this cheaply:

Since $Q^\top Q = I$, the model can be rewritten so the rotation costs almost nothing: $\mathbf{x}W = (\mathbf{x}Q)(Q^\top W)$, and $Q^\top W$ is computed once, ahead of time. QuaRot uses Hadamard rotations this way to run Llama models with 4-bit weights, activations and KV cache; SpinQuant learns the rotations instead. See rotations in quantization.

import numpy as np
def R(theta):
    c, s = np.cos(theta), np.sin(theta)
    return np.array([[c, -s], [s, c]])
 
Q = R(0.7)
np.allclose(Q.T @ Q, np.eye(2))          # True
a, b = np.array([2.2, 0.6]), np.array([0.6, 1.8])
np.isclose((Q @ a) @ (Q @ b), a @ b)     # True
 
# RoPE on a (T, d) matrix of queries, pairs (0,1), (2,3), ...
def rope(x, base=10000.0):
    T, d = x.shape
    theta = base ** (-np.arange(0, d, 2) / d)          # (d/2,)
    ang = np.arange(T)[:, None] * theta[None, :]       # (T, d/2)
    c, s = np.cos(ang), np.sin(ang)
    x1, x2 = x[:, 0::2], x[:, 1::2]
    out = np.empty_like(x)
    out[:, 0::2] = x1 * c - x2 * s
    out[:, 1::2] = x1 * s + x2 * c
    return out
 
from scipy.linalg import hadamard
H = hadamard(8) / np.sqrt(8)
np.allclose(H @ H.T, np.eye(8))          # True