Looking inside: circuits and features

Interpretability asks what a trained network is actually computing. Here we take apart one real circuit in GPT-2 small, see why single neurons are hard to read (superposition), and how sparse autoencoders and attribution graphs pull readable features and causal paths out of large models.

Key idea: Circuits (induction heads), the logit lens, superposition, sparse autoencoders and features, attribution graphs
Real world: An engineer reverse-engineering a rival’s chip: trace which wires connect to which, guess what a block does, then cut one wire and check that the behaviour breaks the way the guess predicts.
In an LLM: In GPT-2 small, head 4.11 tags every token with the token before it, and heads 5.1, 5.5 and 7.10 use those tags to continue repeated text. Switch off GPT-2’s six strongest induction heads and its accuracy on a repeated random list falls from 62% to 10%.

Why look inside

Every number in a trained model was set by gradient descent, not by a person. Nobody wrote a rule saying “if a phrase repeats, copy what came next”. Yet GPT-2 does exactly that. tries to find how: which heads, neurons and directions carry out a behaviour, written down precisely enough that you can predict what happens when you change them.

There are three reasons people care. Understanding (what has the model actually learned?), debugging (why did it get this wrong?) and safety (can we check that a model isn’t doing something we don’t want, without trusting its output?).

This page goes from the smallest, best-understood example to the methods used on frontier models today.

A real circuit: induction heads

A is a few model parts that together do one job. The most famous one in transformers is the induction circuit (Elhage et al., 2021; Olsson et al., 2022). Its job: if the text so far contains “… A B … A”, predict “B”. When a phrase repeats, continue it the way it went last time.

No single head can do this. Attention compares a query at one position with keys at others, but the information it needs, “which token came before that one?”, isn’t in any key to begin with. So two heads in different layers work together:

A previous-token head (head 11 in layer 4, written 4.11) puts almost all of every token’s attention on the token just before it. Its output writes “the token before me was X” into each position’s residual stream.
An (heads 5.1, 5.5 and 7.10 in GPT-2 small) runs later. At the current token, its query says “I am A”. The keys it compares against now include the note written in step 1, so the key at the token after the earlier A says “the token before me was A”. That matches. The head attends there, and its value copies that token (B) forward, pushing the output towards B.

Here it is with GPT-2’s real attention, on a sentence written twice. Press play, then click tokens in the second row to move the query.

Loading GPT-2 data…

The induction circuit in GPT-2 small.

The trick is that layer 5 reads something layer 4 wrote. Researchers call this K-composition: the keys of the later head are built partly from the output of the earlier head. It is a small example of how layers build on each other through the residual stream.

example · finding your place in a song

You hear a song you know, and the radio cuts out after “…and I will always”. You can carry on: “love you”. You don’t store a rule for every song. You find the last time you heard “always” in that song and say what came after it. That is the induction move: find where the current word appeared before, and copy its successor.

example · a repeated code snippet

A programmer pastes a function, then starts typing a modified copy: def area(width, height): return width *. An LLM completes height with high confidence. The earlier copy contains “width *” followed by “height”, and induction heads find it. This is a big part of why LLMs are good at continuing lists, tables and code that follow a pattern already in the prompt.

Testing it: switch heads off

Seeing a pattern in attention isn’t proof that the model uses it. The test is an ablation: replace a head’s output with its average output (as if it saw nothing in particular) and measure what breaks. Below, GPT-2 reads 12 random words and then the same 12 again. The first time, it can’t guess anything. The second time it should be able to, purely by induction.

Loading GPT-2 data…

Knocking out induction heads.

Two lessons from this. Switching off all six induction heads (6 of GPT-2’s 144) breaks the repeat almost completely, which is strong evidence that they do the job. But switching off any one barely matters, because the others cover for it. Real circuits are often redundant like this, which is one reason they are hard to find.

Induction heads matter beyond copying. Olsson et al. found that they form suddenly during training, at the same moment the model gets much better at using earlier context (in-context learning), and argued they are a major part of how in-context learning works.

The logit lens: reading the residual stream early

A quick, cheap tool: take the residual stream after any layer and push it straight through the final norm and the unembedding, as if the model stopped there. This is the (nostalgebraist, 2020). In step 4 of the circuit above, the right answer “fox” only becomes the top guess after layer 7, right after the induction heads in layers 5 to 7 have done their work. Before that, the residual stream still looks like the current word (“brown”, “-ish”).

The logit lens is rough. Early layers aren’t trying to produce output yet, so their readings can be junk; the tuned lens (Belrose et al., 2023) learns a small correction per layer and reads more reliably.

Why neurons are hard to read: superposition

The obvious way to understand a network is to look at one neuron at a time. That mostly fails. Many neurons are : one neuron in a small language model fired for academic citations, English dialogue, HTTP requests and Korean text (Bricken et al., 2023).

The leading explanation is (Elhage et al., 2022). A model wants to track far more concepts than it has dimensions: every name, place, language, coding pattern and grammatical state. If concepts are stored as directions, you can fit more directions than dimensions, as long as they are only roughly at right angles. The cost is interference: reading one direction picks up a little of the others. That cost is small when the concepts are rarely present at the same time, because then they rarely collide.

The widget is a trained toy model from that paper: it squeezes 3 to 5 features into just 2 numbers and must get them back out. Each feature matters 30% less than the one before. Drag the slider from “on 100% of the time” to “on 1% of the time”.

each feature is on5% of the time

switch features on, then read each one back out

features stored5 of 5 in 2 dims

arrangementa pentagon

signal lost on average10%

5 features share 2 dimensions (a pentagon). Read-outs are clean for the features you switched on: the ReLU and a small negative bias cut off the small overlaps. Switch on two neighbouring features to see what goes wrong.

A tiny model trained to squeeze 5 features into 2 numbers and get them back out (read-out = ReLU(direction · hidden + bias)). Feature 1 matters most, each next one 30% less. The arrangement for each setting was really trained, not drawn by hand.

The trained arrangement changes with sparsity, as the paper found. When features are common, the model stores only the two most important at right angles and drops the rest. As they become rarer it packs in more: three, then four as two opposite pairs, and finally all five as a pentagon. A ReLU and a small negative bias filter out the small overlaps when only one feature is on. Switch on two neighbours in the pentagon and you see the cost.

example · sharing a coat hook

An office has 10 hooks and 30 staff. If everyone came in every day, the hooks would be hopeless. But each person is in one day a week, so roughly 6 coats are there on a typical day, and sharing hooks almost never causes a pile-up. Rarity is what makes sharing cheap. That is the same deal superposition makes with features.

example · a direction in GPT-2

GPT-2’s residual stream has 768 numbers per token, but the model clearly tracks many more than 768 things: the topic, whether we are inside a quotation, the subject of the sentence, the language, the tense. Most of those are absent from any one token. Superposition lets each have its own direction while sharing the 768 dimensions.

Sparse autoencoders: pulling the features back out

If concepts are directions in superposition, you want to find those directions. A (SAE) learns them. It is trained on millions of activation vectors from one layer and learns a big dictionary of directions, many more than the layer has dimensions. For each vector:

Encode: take the dot product with every dictionary direction, add a bias and keep only the large results (a ReLU or a threshold). Most come out as zero.
Decode: add up the few active directions, each scaled by its activation, to rebuild the vector.

Training rewards a good rebuild while using as few active directions as possible. Each direction that survives is a : if it is a good one, it is active on one recognisable thing and nothing else.

Here is a hand-built SAE with 12 named features and 8-number vectors. Pick a phrase: its vector is a mix of a few features. The SAE takes the mix apart again.

The model reads “…we drove across the Golden Gate”. At the last token, its residual vector is a mix of these features:

residual vector x1 × 8

d1d2d3d4d5d6d7d8−0.51−0.34−0.16−0.700.37−0.30−0.98−0.47

→

rebuilt from active features1 × 8

d1d2d3d4d5d6d7d8−0.52−0.35−0.19−0.830.50−0.39−1.24−0.52

the SAE’s features: dot product with x, kept only if it beats the threshold (click to add or remove a feature)

● really in the mix■ found■ false alarm│ threshold

threshold0.30

features in the mix2

features active2 of 12

false alarms0

rebuild error22%

The SAE recovers exactly the 2 features in the mix and nothing else. The 8 numbers looked meaningless, but they are 2 features added together. The rebuild is not perfect because the leakage between features shifts the measured strengths a little.

A toy sparse autoencoder with 12 features and an 8-number vector. Real ones read vectors of thousands of numbers with millions of features, but the arithmetic is the same: one dot product per feature, then a threshold.

The two limits of the method are both visible. With too low a threshold, overlaps between directions make features fire that aren’t there. And when four features are on at once (“THE DOG DIED.”), their overlaps add up enough to cause a false alarm even at the default threshold. SAEs work because, in real models, only a small fraction of features is active on any one token.

example · naming the notes in a chord

A piano chord is one sound wave, but a trained ear hears “C, E, G”. The wave is the activation vector; the 88 keys are the dictionary; naming the notes is the encoding. It is easy when three keys are pressed and hopeless when forty are, which is why sparsity matters.

example · the Golden Gate Bridge feature

Anthropic trained SAEs with up to about 34 million features on a middle layer of Claude 3 Sonnet (Templeton et al., 2024). Features covered cities, people, code bugs, deception, sycophancy and more, and many responded to the same concept in several languages and in images. One fired on the Golden Gate Bridge. Clamping it to 10 times its usual maximum made the model bring the bridge into almost every answer, and even claim to be the bridge. Anthropic ran this as “Golden Gate Claude”, a 24-hour public demo in May 2024. Changing one feature changing behaviour this specifically is evidence that the feature is real, not just a pattern in the data.

OpenAI trained a 16-million-feature SAE on GPT-4 (Gao et al., 2024) and Google DeepMind released open SAEs for every layer of Gemma 2 2B and 9B (and some of 27B) (Gemma Scope, Lieberum et al., 2024), so anyone can now browse features of real models, for example on Neuronpedia.

Circuit tracing and attribution graphs

Features tell you what is represented at one layer. To see how an answer is computed, you want the links between features across layers. Anthropic’s 2025 circuit tracing work (Ameisen et al., 2025; Lindsey et al., 2025) does this in three steps:

Train a cross-layer transcoder: like an SAE, but its features read the residual stream at one layer and write to the MLP outputs of all later layers. The one used for Claude 3.5 Haiku had 30 million features.
For one prompt, swap the model’s MLPs for these features (a replacement model). Whatever the features fail to explain is kept as “error nodes”, so the output still matches exactly.
Draw an : features as nodes, and an edge wherever one feature directly pushed another up or down on this prompt. Prune it to the strongest paths and group similar features.

Then test the graph the same way we tested induction heads: change a node and see whether the output changes as the graph predicts.

Two hops inside one forward pass: “Dallas” features activate “Texas” features, and “capital” activates “say a capital”. Texas and say-a-capital together push up “say Austin”. The dashed arrow is a shortcut the paper also found: Dallas straight to Austin. Try an intervention.

A simplified attribution graph for Claude 3.5 Haiku completing “Fact: the capital of the state containing Dallas is” with “Austin”, redrawn from Lindsey et al. (2025). Each box is a group of features; each arrow means “this pushed that up”. The interventions and their results are the ones reported in the paper.

The paper used graphs like this to show, among other things, that the model plans rhymes before writing a line of poetry, that it shares features across languages, and that its explanation of how it did arithmetic doesn’t match the mechanism it actually used. The tools were open-sourced in May 2025 for open models such as Gemma 2 and Llama 3.2.

Caveats

Interpretability results are real but partial. Keep these in mind when you read headlines.

Coverage is small. The authors of the circuit tracing work say their graphs give satisfying insight for about a quarter of the prompts they tried. Error nodes and missing attention mechanisms hide the rest.
A replacement model is not the model. SAEs and transcoders don’t rebuild activations perfectly, and the leftover error can hide important computation.
Features depend on the dictionary. A bigger SAE splits one feature into several finer ones (“bridge” into particular bridges). There isn’t one true list of features.
Labels are human guesses. A feature called “deception” is named from the examples that activate it most, which may not cover what it does on other inputs.
Redundancy fools ablations. As in the knockout widget, removing one part may change nothing because backup parts take over. GPT-2 even has backup heads that switch on when the main ones are removed (Wang et al., 2022).
Mixed practical results. Google DeepMind reported in 2025 that SAE features did worse than simple linear probes at detecting harmful intent, and shifted effort away from SAEs (Smith et al., 2025).

What is solid: models do contain readable, causally important structure. Small circuits like induction heads are well understood, and you can find and change interpretable features in frontier models. A complete account of how a large model answers a question is still out of reach.

going deeper

The induction circuit in equations

With $x_t$ the residual stream at position $t$ , the previous-token head writes $o_t \approx W_O^{(4.11)} W_V^{(4.11)} x_{t-1}$ , a transformed copy of the previous token. The induction head’s score between query $t$ and key $s$ is

\text{score}(t, s) = x_t^\top W_Q^\top W_K \,(x_s + o_s) \approx \underbrace{x_t^\top W_Q^\top W_K\, W_O^{(4.11)} W_V^{(4.11)}\, x_{s-1}}_{\text{large when token}_{s-1} = \text{token}_t}

so the key at $s$ matches when the token before $s$ equals the current token. This is K-composition. The induction head’s OV circuit then copies token $_s$ : its $W_U W_O W_V W_E$ has large positive diagonal entries.

The SAE objective

For activations $x \in \mathbb{R}^{d}$ and a dictionary of $F \gg d$ features:

f = \operatorname{ReLU}(W_{\text{enc}} x + b_{\text{enc}}), \qquad \hat x = W_{\text{dec}} f + b_{\text{dec}}, \qquad \mathcal{L} = \lVert x - \hat x \rVert^2 + \lambda \sum_i f_i \lVert W_{\text{dec},i} \rVert

The second term pushes most $f_i$ to zero. Variants replace it with keeping only the top $k$ features (Gao et al.) or a learned threshold, JumpReLU (Rajamanoharan et al., 2024), which the widget uses with a single threshold.

Superposition toy model

Elhage et al. train $x' = \operatorname{ReLU}(W^\top W x + b)$ with $W \in \mathbb{R}^{2 \times n}$ , inputs $x_i = 0$ with probability $S$ and uniform on $[0, 1]$ otherwise, and loss $\sum_i I_i (x_i - x'_i)^2$ with importance $I_i = 0.7^i$ here. The widget’s arrangements were trained this way (Adam, 6,000 steps, best of 6 seeds).

How the data was made

Attention and logit lens: GPT-2 small on the fox sentence, from the shared course data.
Knockout: each head’s output (its slice of the input to the attention output projection) replaced by its mean over about 500 tokens of ordinary text. Probabilities are for the true next word; the average is over 32 random repeated sequences of 12 tokens. Induction heads were ranked by their attention to “the token after my previous occurrence” on those sequences: 5.5, 6.9, 7.10, 5.1, 7.2, 10.7.

Papers

A mathematical framework for transformer circuits: Elhage et al., 2021
In-context learning and induction heads: Olsson et al., 2022
Interpretability in the wild (the IOI circuit in GPT-2 small): Wang et al., 2022
Toy models of superposition: Elhage et al., 2022
Towards monosemanticity: Bricken et al., 2023
Scaling monosemanticity: Templeton et al., 2024
Scaling and evaluating sparse autoencoders: Gao et al., 2024
Gemma Scope: Lieberum et al., 2024
Circuit tracing (methods): Ameisen et al., 2025; On the biology of a large language model: Lindsey et al., 2025
Logit lens: nostalgebraist, 2020; tuned lens: Belrose et al., 2023