Lengths, distances and normalisation

A norm measures how long a vector is. The usual one is the straight-line length; another adds up the sizes of the numbers. Dividing a vector by its length keeps its direction and makes its length 1. Language models do this to every hidden state, many times per token, with LayerNorm or RMSNorm.

Key idea: L2 norm (straight-line length), L1 norm (city blocks), distance = norm of the difference, unit vectors, RMS; cosine = dot product of unit vectors; LayerNorm and RMSNorm rescale hidden states
Real world: A taxi in Manhattan: the straight-line distance to a place 3 blocks east and 4 north is 5 blocks (L2), but the taxi has to drive 7 (L1).
In an LLM: The hidden state of “ cat” in GPT-2 small has length about 6 after the embedding and about 245 after 11 layers. Before every attention and MLP sublayer, LayerNorm rescales a copy of it to length √768 ≈ 27.7 (before its learned gain), so each layer sees numbers of a familiar size.

How long is a vector?

The length of a vector is called its , written $\lVert\mathbf{x}\rVert$ . The ordinary one, the L2 norm, is Pythagoras’ theorem in any number of dimensions:

\lVert\mathbf{x}\rVert_2 = \sqrt{x_1^2 + x_2^2 + \dots + x_n^2} = \sqrt{\mathbf{x}\cdot\mathbf{x}}

The L1 norm adds up the sizes of the numbers instead, as if you could only move along the axes:

\lVert\mathbf{x}\rVert_1 = |x_1| + |x_2| + \dots + |x_n|

The distance between two points is the norm of their difference, $\lVert\mathbf{a} - \mathbf{b}\rVert$ . And the RMS (root mean square) is the L2 norm divided by $\sqrt{n}$ : the typical size of one number in the vector, which doesn’t grow just because the vector has more numbers.

|a| = √(3.0² + 2.0²) = √13.00 = 3.61

L2 length3.61

L1 length5.00

RMS = L2 ÷ √22.55

The ordinary length: Pythagoras, 3.61. The dashed circle holds every vector with the same length; the shaded disc is everything of length 1 or less.

Drag the arrow. The grey shapes are every point at distance 1 from the origin, measured the same way.

example · a taxi in Manhattan

A restaurant is 3 blocks east and 4 blocks north. As the crow flies it is $\sqrt{3^2 + 4^2} = 5$ blocks away: the L2 distance. A taxi has to follow the streets, $3 + 4 = 7$ blocks: the L1 distance. Every route that only goes east and north is 7 blocks; no route is shorter than 5. In maps, “distance” usually means L2; in a grid city, the meter runs on L1.

Unit vectors and cosine

Divide a non-zero vector by its length and you get a : same direction, length exactly 1. This is normalising it. It throws away “how much” and keeps “which way”.

That explains cosine similarity: it is the dot product of the two unit vectors,

\cos\theta = \frac{\mathbf{a}}{\lVert\mathbf{a}\rVert}\cdot\frac{\mathbf{b}}{\lVert\mathbf{b}\rVert}.

The plain dot product mixes direction and length: a long vector scores high against everything roughly aligned with it. Whether that is what you want depends on the job. Search systems often normalise so that long documents don’t win just for being long. Attention uses the raw dot product, so a key can “shout” by being long; some recent models (OLMo 2, Gemma 3) normalise queries and keys first, called QK-norm, to keep scores in check.

Normalisation inside a transformer

A transformer adds every layer’s output to a running vector per token, the residual stream. With every addition the vector tends to grow. Real numbers from GPT-2 small:

Loading GPT-2 data…

A layer that sometimes receives numbers of size 0.2 and sometimes 30 is hard to train. So before each attention and MLP sublayer, the model takes a normalised copy:

LayerNorm (GPT-2, BERT): subtract the mean of the vector’s numbers, divide by their standard deviation, multiply by a learned gain and add a learned bias.
RMSNorm (Llama, Mistral, Qwen, Gemma): skip the mean, just divide by the RMS and multiply by a learned gain. Cheaper, and works as well.

\operatorname{RMSNorm}(\mathbf{x}) = \frac{\mathbf{x}}{\sqrt{\tfrac{1}{n}\sum_i x_i^2 + \varepsilon}} \odot \mathbf{g}

The small $\varepsilon$ avoids dividing by zero, and $\odot$ means multiplying position by position. Scale the input up or down and watch the output stay put:

multiply input by×1

input x

output = x ÷ RMS

input L2 length3.85

RMS1.36

output RMS1.00

output L2 = √82.83

RMS (root mean square) is the L2 length divided by √8: the typical size of one number, here 1.36. Dividing by it rescales the vector to RMS 1 whatever its size, so a layer that receives it always sees numbers of a familiar scale. The direction, and so the pattern of the bars, is kept. Both then multiply by a learned gain per position (all 1 here).

An 8-number hidden state. Slide its size up and down: the input bars grow, shrink and run off the chart (▲), while the normalised output does not move at all.

example · adjusting the volume before recording

A sound engineer sets the input gain so that a whisper and a shout both reach the mixing desk at a sensible level. The content (the shape of the sound wave) is unchanged; only its loudness is. RMSNorm is the same: “RMS” is literally the measure audio meters use for loudness. Without it, the next stage either clips the loud parts or loses the quiet ones in the noise.

example · embedding lengths in GPT-2

The 50,257 token embeddings in GPT-2 small have L2 norms between 2.45 and 6.32, 3.96 on average. “ the” has length 2.67 and “ queen” 3.78. Because the very first thing each block does is a LayerNorm, these differences in length matter much less than differences in direction. That is one reason cosine, not raw distance, is the usual way to compare embeddings. See normalization.

going deeper

The family of norms

For $p \ge 1$ , $\lVert\mathbf{x}\rVert_p = \big(\sum_i |x_i|^p\big)^{1/p}$ . $p = 2$ is Euclidean length, $p = 1$ the city-block length, and $p \to \infty$ gives $\max_i |x_i|$ , the largest single number (used in quantization to pick a scale). Every norm satisfies $\lVert c\mathbf{x}\rVert = |c|\lVert\mathbf{x}\rVert$ and the triangle inequality $\lVert\mathbf{a} + \mathbf{b}\rVert \le \lVert\mathbf{a}\rVert + \lVert\mathbf{b}\rVert$ .

For $\mathbf{x} \in \mathbb{R}^n$ : $\lVert\mathbf{x}\rVert_2 \le \lVert\mathbf{x}\rVert_1 \le \sqrt{n}\,\lVert\mathbf{x}\rVert_2$ .

LayerNorm in full, with learned $\mathbf{g}, \mathbf{b}$ :

\operatorname{LN}(\mathbf{x}) = \frac{\mathbf{x} - \mu}{\sqrt{\sigma^2 + \varepsilon}} \odot \mathbf{g} + \mathbf{b}, \quad \mu = \tfrac{1}{n}\textstyle\sum_i x_i,\ \sigma^2 = \tfrac{1}{n}\sum_i (x_i - \mu)^2.

In code

import torch
x = torch.tensor([3.0, 4.0])
x.norm()                  # tensor(5.)   L2
x.norm(p=1)               # tensor(7.)   L1
x / x.norm()              # tensor([0.6, 0.8])  unit vector
 
def rms_norm(x, g, eps=1e-6):
    return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps) * g
 
h = torch.randn(10, 768) * 50
torch.nn.functional.layer_norm(h, (768,)).norm(dim=-1)   # all ≈ 27.7 = √768

In papers

LayerNorm: Ba et al., 2016. RMSNorm: Zhang and Sennrich, 2019.
Normalising before each sublayer (pre-LN) and why it trains better: Xiong et al., 2020.
Huge activations on the first token: Sun et al., 2024; attention sinks: Xiao et al., 2023.
QK-norm: Dehghani et al., 2023; used in OLMo 2.