Real attention patterns

Key idea

Trained heads specialise: previous-token heads, induction heads that continue repeated text, pronoun heads, and first-token “sinks”

Real world

Watching where people’s eyes go while they read. Some glance back at the last word, some jump back to a name mentioned earlier, and when a line needs no checking their eyes rest on the start of the page.

In an LLM

Give GPT-2 “The quick brown fox jumps over the lazy dog.” twice. On the second “ fox”, head 5.5 puts 96% of its attention on “ jumps”, the word that followed “fox” the first time, and GPT-2 predicts “ jumps” next.

How to read an attention pattern

Heads are named layer.head, and both numbers start at 0, so “4.3” is head 3 (the fourth head) in layer 4 (the fifth of the 12 layers).

Each head in each layer produces a $T \times T$ matrix of weights. Row $i$ shows how token $i$ splits its attention over tokens $1$ to $i$. Two ways to draw it:

• Lines: queries on the left, keys on the right, a line for each weight, thicker for more. This is the style of the bertviz tool.
• Heatmap: the matrix itself, darker for more weight, empty above the diagonal because of the causal mask.

A pattern shows where a head reads, not what it does with it. The value vectors and $W_O$ decide what is carried. So a pattern is a strong clue, not a full explanation.

The explorer

All the numbers below come from running GPT-2 small (12 layers, 12 heads each) on four prompts. The grid on the left scores every head on the repeated fox sentence, by how much of its attention goes to the previous token, the first token, or the induction target. Click around.

Heads with a known job

Previous-token head: 4.11

Head 11 in layer 4 gives over 99% of every token’s attention to the token just before it, on every prompt. It does no “understanding” at all. What it does is useful plumbing: it copies information about the previous token into each position, so that later heads can ask “which token came before me?” using only the current position. That is the first half of an induction head.

Induction heads: 5.1, 5.5, 7.10

Language repeats itself: names, phrases, code, quotes. An Induction head exploits that. At the current token, say “fox”, it looks for earlier positions whose previous token was also “fox”, and attends to them. The token at such a position is the one that came after “fox” last time, “jumps”, which is a good prediction for what comes next.

It needs two heads working together, in different layers:

1. A Previous-token head (like 4.11) writes into each position “the token before me was X”.
2. The induction head’s query asks “who was preceded by my current token?”, and its keys answer using what step 1 wrote.

Here is the idea with exact rules instead of learned weights. Press play, then try the other sentences, including one where nothing matches:

Now the real thing. These are GPT-2’s own heads on the fox sentence typed twice:

The effect on predictions is large. The first time through the sentence, GPT-2’s top guess for the next token is right only once in ten tries (“ over” → “ the”). The second time, it is right eight times out of nine, often with over 90% confidence: after the second “ lazy” it gives “ dog” 99%, where the first time it guessed a comma at 6%.

Induction heads are a big part of how models learn from the prompt (in-context learning). Anthropic researchers found that they appear suddenly during training, in a sharp “phase change”, at the same moment the model gets much better at using long contexts.

Pronoun head: 4.3

On “The cat sat on the mat because it was tired”, head 4.3 sends 84% of the attention of “it” to “cat” and 4% to “mat”. This is the example from the intuition page. We have checked only this one sentence, so it is safer to say “4.3 does this here” than “4.3 is the pronoun head”.

Name movers: 9.6 and 9.9

In “When Mary and John went to the store, John gave a drink to”, GPT-2 predicts “ Mary” (45%). At the final “ to”, head 9.6 gives 49% of its attention to “ Mary”, and only 12% and 6% to the two mentions of “ John”; head 9.9 gives 36% to “ Mary”. A 2022 study traced the whole circuit for this kind of sentence and called these name mover heads: they copy the name that hasn’t been repeated to the last position. Other heads, earlier in the network, work out which name is the repeated one.

Attention sinks

Pick a head at random and the most common thing you will see is not a clever pattern: most of the weight goes to the very first token, whatever the query. Heads 4.10, 8.11 and 9.1 send it about 87% of their attention on average, and on the fox sentence 95 of GPT-2’s 144 heads send it more than half on average.

The first token is rarely the most informative word, so why? The best current explanation:

• Softmax weights must add up to 1. A head has no way to say “nothing here is relevant”; the weight has to go somewhere.
• The first token is visible to every other token (thanks to the causal mask), so it is a convenient, always-available place to park unwanted weight. Models learn to give it a small value vector, so attending to it adds almost nothing: a “no-op”. In heads 4.10, 8.11 and 9.1 of GPT-2, what the first token passes on (its value times $W_O$) is about a tenth of the size of what any other token in the cat sentence would pass on.
• The first token’s vector becomes very large inside the model, which helps it attract this attention. In GPT-2 on the cat sentence, by layer 6 its vector has a length of about 3,000, against roughly 70 to 100 for every other token.

This is called an Attention sink. It matters in practice. The StreamingLLM paper found that if you drop the first tokens from the KV cache to save memory on a long conversation, quality collapses; keep just those few tokens and it recovers. Newer models sometimes add a dedicated sink token, or a way for softmax to output “nothing”, for this reason.

A word of caution

Patterns are suggestive, but:

• A head can attend strongly to a token and still pass on almost nothing, if its value vectors are small (sinks are the extreme case).
• Many heads mix several behaviours, and change them depending on the prompt.
• Labels like “pronoun head” come from a few examples. Proper evidence comes from interventions: change the head’s output and see what breaks. That is the subject of interpretability.