RoPE: positions as rotations

Key idea

Split q and k into pairs, rotate pair i by position × θᵢ; the score q·k then depends only on the offset; extend context by slowing the slow pairs (PI, NTK, YaRN)

Real world

Two people on a merry-go-round. Wherever the ride has turned to, the angle between them stays the same. You can tell how far apart they sit without knowing where the ride started.

In an LLM

In Llama 3, “it” at position 812 and “cat” at position 805 get the same position effect on their attention score as the same two words at positions 12 and 5: an offset of 7 either way.

The problem RoPE solves

Attention on its own has no idea of order: it compares every query with every key by a dot product, and shuffling the tokens just shuffles the scores. GPT-2 fixed this the way the positions lesson describes: it learned one vector per position (1,024 of them) and added it to the token’s embedding.

That has two weaknesses. The model can’t read past position 1,024, because there is no vector for position 1,025. And what usually matters for language is relative position (“the word just before me”, “the noun three words back”), but an added absolute vector makes the model work that out for itself.

Rotary position embedding, or RoPE (rotary position embedding) (Su et al., 2021), builds the relative part into the arithmetic. Since 2023 it has been the default: Llama, Mistral, Qwen, DeepSeek, Gemma and gpt-oss all use it.

Rotate, don’t add

RoPE leaves the embeddings alone. Instead, after a head has made its query $q$ and key $k$ (see queries, keys and values), it:

1. Splits each vector into pairs of numbers: $(q_1, q_2), (q_3, q_4), \dots$ A head of 128 numbers has 64 pairs.
2. Treats each pair as an arrow in a flat plane.
3. Turns the arrow by an angle equal to the token’s position × that pair’s frequency $\theta_i$.

Turning an arrow by angle $\alpha$ is multiplication by a rotation matrix:

Values are not rotated. Only queries and keys are, because only they meet in the score.

Why only the offset survives

The dot product of two arrows is their lengths times the cosine of the angle between them. If the query sits at position $m$ and the key at position $n$, the query was turned by $m\theta$ and the key by $n\theta$. The angle between them changed by $(m - n)\theta$, and nothing else changed. So the score depends on the content of the two tokens and on the offset $m - n$, never on $m$ or $n$ alone.

Try it. Set any two positions, then shift both tokens along together:

The curve on the right is the score for every offset. Shifting both tokens moves you along nothing: you stay on the same dot. Changing only one position slides you along the curve.

Many pairs, many speeds

If every pair turned at the same speed, the score would repeat every full turn and the model could not tell an offset of 3 from an offset of 3 + one full turn. So each pair gets its own frequency, spread over a huge range:

where $d$ is the head size and $b$ is the base, 10,000 in the original paper. The first pair turns 1 radian per token (a full turn every 6.3 tokens); the last turns about $1/b$ radians per token and needs tens of thousands of tokens to go round.

This is the same trick as the hands of a clock: the second hand is precise over short times, the hour hand tells you roughly where you are in the day. Fast pairs separate neighbours; slow pairs measure long distances. Models with long contexts raise the base so the slowest hands turn more slowly still: Llama 3 uses 500,000, Qwen3 1,000,000.

Reading past the training length

A model trained on 4,096-token texts has only ever seen the angles that positions 0 to 4,095 produce. Fast pairs went round hundreds of times, so every angle is familiar. The slowest pairs only covered part of a turn. Feed the model 16,000 tokens and those pairs point at angles it has never seen, and the output falls apart.

Three fixes, each a way of squeezing angles back into the familiar range:

• Position interpolation (Chen et al., 2023): divide every position by the stretch factor. Everything fits, but neighbours now look closer together than in training, which blurs the fast pairs.
• NTK-aware scaling (an online post by “bloc97”, 2023): raise the base instead. The fastest pair keeps its speed and the slowest pair is fully interpolated, with a smooth slide between.
• YaRN (Peng et al., 2023): decide per pair. Pairs that turned many times in training are left alone, pairs that never finished a turn are interpolated, those in between are blended. It also sharpens the softmax slightly, because attention spreads thinner over a longer text.

All three need little or no fine-tuning, which is why context extension became cheap. Real recipes: DeepSeek-V3 was pre-trained at 4K tokens and stretched to 128K with YaRN; Qwen3 trains at 32K and offers YaRN for 128K; Llama 3.1 went from 8K to 128K with its own per-frequency rule in the same spirit.

What it costs

Nothing in parameters: the angles are fixed, not learned. The rotation is a few multiply-adds per number, fused into the attention kernel. RoPE is applied after the query and key are made and before the score, so it also works with a KV cache: each key is rotated once, by its own position, and stored that way.

import torch
 
def rope(x, pos, base=10000.0):          # x: (T, d), pos: (T,)
    d = x.shape[-1]
    theta = base ** (-torch.arange(0, d, 2) / d)       # (d/2,)
    ang = pos[:, None] * theta[None, :]                # (T, d/2)
    cos, sin = ang.cos(), ang.sin()
    x1, x2 = x[..., 0::2], x[..., 1::2]                # the pairs
    return torch.stack([x1 * cos - x2 * sin,
                        x1 * sin + x2 * cos], dim=-1).flatten(-2)