Positions

Attention on its own can’t tell word order: shuffle the tokens and every token’s output stays the same, only moved. So models add information about position. GPT-2 learns one vector per position; the original transformer used sine waves of different speeds; most modern models rotate queries and keys instead (RoPE).

Key idea: Self-attention is permutation-equivariant; fix it by adding a position vector to each token (learned, as in GPT-2, or sinusoidal) or by encoding relative position inside attention (RoPE, ALiBi)
Real world: Scrabble tiles in a bag. The bag holds the same letters whether they spell “listen” or “silent”. To read a word you need the tiles laid out in order on the board.
In an LLM: “dog bites man” and “man bites dog” have exactly the same tokens. GPT-2 tells them apart by adding a learned vector for position 0, 1, 2 to each token’s embedding before the first layer.

Attention ignores order

Look again at what attention computes. Each token’s output is a weighted average of the other tokens’ values, with weights from query–key dot products. Nothing in that recipe refers to where a token is. If you shuffle the input rows, every query, key and value is shuffled the same way, the scores are shuffled, and each token’s output row is exactly what it was before, just in a new place. Mathematicians call this permutation equivariance; for a language model it means the input is a bag of words.

See it with the toy attention head:

add position vectorscausal mask

follow:

output, original order5 × 3

q1q2q3The0.540.220.18cat0.240.590.16sat0.360.320.23on0.530.210.19the0.540.220.18

output, shuffled input5 × 3

q1q2q3The0.540.220.18cat0.240.590.16sat0.360.320.23on0.530.210.19the0.540.220.18

“cat” row changed by0.00

largest change, any token0.00

Both sides are the same sentence, so of course they match. Press Shuffle.

The toy attention head from the attention pages, run on “The cat sat on the” and on a shuffled copy. Shuffle, then compare the output row of the same token in both (click a token to follow it). Then switch on position vectors.

The causal mask does break the symmetry a little, because a token only sees the tokens before it, so moving it changes what it can see. Models trained with no position information at all (“NoPE”) can learn some sense of position this way. But it is indirect, and every major model adds a explicitly.

Option 1: learn a vector per position

The simplest fix: a second embedding table, with one row per position instead of one per token. GPT-2 has 1,024 positions, so its position table is 1,024 × 768. The input to the first layer is the sum of the two lookups:

x_t = E_{\text{token}}[\,\text{id}_t\,] + E_{\text{pos}}[\,t\,]

Now “ cat” at position 1 and “ cat” at position 5 are different vectors, and so are their queries and keys. Here is GPT-2’s real position table:

Loading GPT-2 data…

GPT-2 was never told that position 21 follows position 20; it learned 1,024 independent vectors. Training still made them smooth: neighbouring positions are very similar, and similarity fades with distance. Some columns drift slowly with position, like waves.

Learned positions have one hard limit: there is no row for position 1,024. GPT-2 simply cannot read longer texts, and positions near the end of the table, seen less often in training, are less well trained.

example · seat numbers

At a concert, everyone holds a ticket with a seat number. The people are the same whichever seats they sit in, but the numbers make the order clear. A learned position table is a sheet of seat labels that the model designs for itself; it just has no labels beyond the last row of the hall.

example · “dog bites man”

Without positions, GPT-2 would see the same three vectors for “dog bites man” and “man bites dog”, and would have to predict the same next word after both. With position vectors added, “ dog” at position 0 differs from “ dog” at position 2, so the attention heads can work out who did what to whom.

Option 2: sine waves

The original transformer (Vaswani et al., 2017) used fixed vectors instead of learned ones. Each position becomes $d$ numbers: pairs of sine and cosine at $d/2$ different speeds. Each pair is like a clock hand turning at its own rate as the position grows. The fastest hand turns 1 radian per position; each next one is slower, down to one that barely moves over thousands of positions. Together, the angles of all the hands identify the position, like the hour, minute and second hands of a clock.

position5

dims 0,1every 6

dims 2,3every 11

dims 4,5every 20

dims 6,7every 35

dims 8,9every 63

dims 10,11every 112

dims 12,13every 199

dims 14,15every 353

sincossincossincossincossincossincossincossincospos 5−1.00.30.3−0.91.00.00.80.60.50.90.31.00.21.00.11.0

dot product of position 5’s encoding with every position (scaled to 1 at itself)

The fastest pair turns 1 radian per position, so it repeats every 6.3 positions; the slowest turns 0.018 radians per position. No two positions share all eight angles. The similarity curve peaks at position 5, and its shape is the same wherever you put it: it depends only on the distance between positions, the property RoPE builds on. Base 100 spreads the eight speeds over the 64 positions shown, so similarity falls off clearly with distance.

Each position becomes 16 numbers: 8 pairs of (sine, cosine) at 8 speeds. Drag the position and watch the hands turn: the fast ones spin, the slow ones barely move. Together, the angles pin down the position, like the hands of a clock.

Two properties made this attractive:

It is defined for every position, even ones longer than any training text (though models did not actually work well beyond their training length).
Moving by a fixed offset is a rotation. Shifting from position $p$ to $p + k$ turns every pair by an angle that depends only on $k$ , so the dot product between two position vectors depends only on how far apart they are. The similarity curve in the widget has the same shape wherever you put it.

Option 3: rotate the queries and keys

That second property is the seed of RoPE (rotary position embedding), which almost every model since 2023 uses: Llama, Qwen, Gemma, DeepSeek, Mistral. Instead of adding a position vector to the input, RoPE rotates each query and key by angles that depend on its position. Then the attention score between two tokens depends only on their relative distance, which is what language mostly cares about. It has its own page: RoPE: positions as rotations.

Another relative scheme, ALiBi (Press et al., 2022), adds no vectors at all. It subtracts a penalty from each attention score in proportion to the distance between the two tokens, so distant tokens get less attention by default. It was used in BLOOM and MPT.

Method	Where position enters	Used in
Learned absolute	Added to the input, one vector per position	GPT-2, GPT-3, BERT
Sinusoidal	Added to the input, fixed waves	Original transformer (2017)
RoPE	Rotates queries and keys in every layer	Llama, Qwen, Gemma, DeepSeek, Mistral
ALiBi	Distance penalty on attention scores	BLOOM, MPT
None (NoPE)	Only through the causal mask	Some layers of some models; research

going deeper

Permutation equivariance, formally

Let $P$ be a permutation matrix and $X$ the $T \times d$ input. Without a mask,

\operatorname{softmax}\!\Big(\frac{(PX W_Q)(PX W_K)^\top}{\sqrt{d_k}}\Big) P X W_V = P \operatorname{softmax}\!\Big(\frac{X W_Q (X W_K)^\top}{\sqrt{d_k}}\Big) P^\top P X W_V = P\, \operatorname{Attention}(X)

because $P^\top P = I$ and the row-wise softmax commutes with reordering rows and columns. The MLP and layer norm act on each row separately, so the whole unmasked transformer is equivariant too.

The sinusoidal formula

For position $p$ and pair $i = 0, \dots, d/2 - 1$ :

\text{PE}(p, 2i) = \sin\!\big(p\, \omega_i\big), \qquad \text{PE}(p, 2i+1) = \cos\!\big(p\, \omega_i\big), \qquad \omega_i = 10000^{-2i/d}

Then $\text{PE}(p) \cdot \text{PE}(q) = \sum_i \cos\big((p - q)\,\omega_i\big)$ , which depends only on $p - q$ .

In papers

Sinusoidal encoding: Vaswani et al., 2017, section 3.5.
Learned positions in GPT-2: Radford et al., 2019.
RoPE: Su et al., 2021.
ALiBi: Press, Smith and Lewis, 2022.
Language models without positional encodings still learn position: Haviv et al., 2022.