The map

A transformer language model is a short pipeline. Text becomes token IDs, IDs become vectors, a stack of identical blocks refines the vectors, and the last vector is turned into a probability for every token. Here is the whole thing on one diagram, with the shape of the data at every step.

Key idea: Tokens → embeddings (+ positions) → N × [norm, attention, add; norm, MLP, add] → final norm → unembedding → softmax
Real world: A car assembly line. The car rolls along one track, and at each station a worker takes a look, does one job and bolts something on. The car is never taken apart; each station adds to it.
In an LLM: In GPT-2 small, each token becomes a list of 768 numbers that rolls up through 12 identical blocks. Each block reads the list, computes something and adds it back. At the top, the list for the last token is turned into 50,257 scores, one per possible next token.

The pipeline

The previous page treated the model as a box that reads tokens and outputs probabilities for the next one. Inside the box there are only six kinds of part:

Tokenizer. Cuts the text into tokens and replaces each one with an ID number.
Embedding. Looks up a list of numbers (a vector) for each ID, and adds information about its position.
Blocks. A has two layers. Attention lets each token gather information from the tokens before it. The MLP (a small two-layer network) then works on each token by itself. Each layer’s output is added back to its input. The stack repeats the same design 12 times in GPT-2 small and 32 times in Llama 3 8B, each copy with its own weights.
Normalisation. Before each layer, and once more at the end, every vector is rescaled to a standard size.
Unembedding. Turns the final vector of each token into one score per vocabulary token.
Softmax. Turns those scores into probabilities.

Click any box to see what it does, how much data flows in and out, and how many parameters it holds. Switch to Llama 3 8B to see what a 2024 model changed. Then press Follow one token to watch the real numbers for “ the” in “The cat sat on the” travel up through GPT-2.

GPT-2 small · Attention

shapeT × 768 → T × 768

parameters (each of 12 blocks)2,362,368

The only place where tokens exchange information. 12 heads of 64 dimensions each; every head lets each token look back at the tokens before it and copy a weighted mix of what they carry.

Lesson: Attention →

A decoder-only transformer, the design behind GPT-2 and Llama (and, in broad outline, most large language models). T is the number of tokens. Click any box for its job and shape, or follow one real token through GPT-2.

Reading the shapes

Every arrow is labelled with the shape of the data on it, written rows × columns. $T$ is the number of tokens in the text. The data is a with one row per token:

Before the embedding it is just $T$ whole numbers, the token IDs.
From the embedding to the final norm it is $T \times d$ : each token carries a vector of $d$ numbers, the model width ( $d$ = 768 in GPT-2 small, 4,096 in Llama 3 8B).
After the unembedding it is $T \times V$ : one score for each of the $V$ tokens in the vocabulary, at every position.

The striking thing is that the shape does not change inside the stack. Every block takes $T \times d$ in and gives $T \times d$ out. That is what makes the blocks stackable, and it is what the vertical line on the left of the diagram stands for: the , one vector per token that each layer reads from and adds to. Attention and the MLP sit on side branches. Their outputs are added to the stream, never substituted for it.

example · an assembly line

On a car assembly line the chassis rolls down one track. At each station someone looks at it, decides what it needs and bolts it on: doors, then seats, then wheels. No station rebuilds the car. If a station does nothing, the car rolls through unchanged. Because every station receives the same kind of object, you can add, remove or reorder stations without redesigning the line.

example · “ the” in GPT-2

In “The cat sat on the”, the last token “ the” starts as row 262 of GPT-2’s embedding table. After the position vector is added, reading a prediction straight out of it gives “ destro”: so far the vector only says which token this is and where. As it goes up the stack, attention layers copy in information from “cat”, “sat” and “on”, and the MLPs add what they know. Read out after block 7, the vector still predicts “ same”. After block 8 it predicts “ floor”, and at the top GPT-2 gives “ floor”, “ bed” and “ couch” 5 to 8% each.

GPT-2 small and Llama 3 8B

The diagram is the same for almost every large language model since 2018. The numbers grow, and a few parts were swapped for better versions:

	GPT-2 small (2019)	Llama 3 8B (2024)
Model width $d$	768	4,096
Blocks	12	32
Attention heads	12 (64 dimensions each)	32 query heads sharing 8 key/value heads (128 dims)
MLP width	3,072, GELU	14,336, SwiGLU
Vocabulary $V$	50,257	128,256
Context length	1,024	8,192
Positions	learned vector added at the bottom	RoPE, rotation inside attention
Normalisation	LayerNorm	RMSNorm
Parameters	124 million	8.0 billion

Each change has its own page: RoPE, normalisation, the MLP, grouped-query attention and the modern recipe.

Where the parameters live

A model’s size is the number of learned numbers in it, its parameters. Here is where every one of GPT-2 small’s 124,439,808 parameters sits, and Llama 3 8B’s 8,030,261,248:

total124,439,808

one block7,087,872

MLP56,669,184

12 × (768 × 3,072 + 3,072 × 768 + biases). The MLPs hold 45.5%, twice as much as attention: each one has two 768 × 3,072 matrices, against attention’s 768 × 2,304 plus 768 × 768.

Every parameter of the model, grouped by job. Click a row or a segment. Numbers are exact counts from the published weights.

Two things stand out. In a small model the vocabulary table is a big share: almost a third of GPT-2 small is the 50,257 × 768 embedding. And in every model the MLPs hold more than attention, roughly two parameters for every one in attention in GPT-2, and four for one in Llama 3 8B. Attention moves information between tokens. The MLPs hold most of the knowledge.

How the course follows the map

The rest of the course walks up the diagram from the bottom:

Tokenization (this section next): how text becomes IDs. Start with why tokens.
Inputs: embeddings and positions.
Attention: from the intuition to queries, keys and values and multiple heads.
The transformer block: the residual stream, normalisation, the MLP, the whole block and the unembedding.
Training and inference: how the parameters are learned, and what it costs to run the model.

going deeper

The forward pass in equations

For token IDs $t_1, \dots, t_T$ , with embedding matrix $W_E \in \mathbb{R}^{V \times d}$ and position matrix $W_P$ (GPT-2):

x^{(0)}_i = W_E[t_i] + W_P[i]

Each block $\ell = 1, \dots, L$ is pre-norm:

\begin{aligned} h &= x^{(\ell-1)} + \operatorname{Attn}_\ell\big(\operatorname{Norm}(x^{(\ell-1)})\big) \\ x^{(\ell)} &= h + \operatorname{MLP}_\ell\big(\operatorname{Norm}(h)\big) \end{aligned}

and the output is

p(\cdot \mid t_{\le i}) = \operatorname{softmax}\big(\operatorname{Norm}(x^{(L)}_i)\, W_U\big), \qquad W_U \in \mathbb{R}^{d \times V}

GPT-2 ties $W_U = W_E^\top$ ; Llama 3 learns a separate $W_U$ . Llama replaces $W_P$ with RoPE inside $\operatorname{Attn}$ .

Parameter counts

GPT-2 small, per block: attention $768 \cdot 2304 + 2304 + 768 \cdot 768 + 768 = 2{,}362{,}368$ ; MLP $768 \cdot 3072 + 3072 + 3072 \cdot 768 + 768 = 4{,}722{,}432$ ; two LayerNorms $2 \cdot 1536$ . Total per block 7,087,872; times 12, plus embeddings ( $38{,}597{,}376 + 786{,}432$ ) and the final LayerNorm (1,536): 124,439,808.

Llama 3 8B, per block: attention $2 \cdot 4096^2 + 2 \cdot 4096 \cdot 1024 = 41{,}943{,}040$ ; MLP $3 \cdot 4096 \cdot 14336 = 176{,}160{,}768$ ; norms $2 \cdot 4096$ . Times 32, plus two $128{,}256 \times 4{,}096$ tables and the final norm: 8,030,261,248.

In code

def forward(ids):                          # ids: (T,)
    x = wte[ids] + wpe[range(len(ids))]    # (T, d)
    for block in blocks:                   # 12 in GPT-2 small
        x = x + block.attn(block.ln1(x))   # (T, d)
        x = x + block.mlp(block.ln2(x))    # (T, d)
    logits = ln_f(x) @ wte.T               # (T, V), tied weights
    return logits.softmax(-1)

In papers

The transformer: Vaswani et al., 2017. It had an encoder and a decoder; GPT-style models keep only the decoder.
GPT-2: Radford et al., 2019. Moved the norm before each layer (pre-norm) and added the final norm.
Llama 3: Grattafiori et al., 2024. RMSNorm (Zhang and Sennrich, 2019), SwiGLU (Shazeer, 2020), RoPE (Su et al., 2021) and grouped-query attention (Ainslie et al., 2023).
Reading predictions out of intermediate layers, as in “Follow one token”, is the logit lens: nostalgebraist, 2020.