The transformer at a glance

One page to come back to. Every component of a decoder-only transformer with its formula, its shape, its parameter count and the lesson that explains it, plus GPT-2 small and Llama 3 8B side by side and a tracer for every shape in a forward pass.

Key idea: Notation, every component with formula, shape and parameters, two real models compared, and a shape tracer
Real world: The exploded diagram at the back of a bicycle manual: every part, its size and its part number, so you can find it again without reading the whole manual.
In an LLM: A paper says “d_model 4096, 32 layers, GQA with 8 KV heads, d_ff 14336”. With this page you can work out every matrix shape and that the model has about 8 billion parameters.

Notation

Symbol	Meaning	GPT-2 small	Llama 3 8B
$T$	tokens in the sequence (context length used)	up to 1,024	up to 8,192
$d_{\text{model}}$	width of the residual stream: numbers per token	768	4,096
$h$	attention heads per layer	12	32
$h_{\text{kv}}$	key/value heads (fewer than $h$ with GQA)	12	8
$d_{\text{head}}$	numbers per head, usually $d_{\text{model}} / h$	64	128
$d_{\text{ff}}$	hidden width of the MLP	3,072	14,336
$L$	number of blocks (layers)	12	32
$V$	vocabulary size	50,257	128,256
$N$	total parameters	124M	8.03B

Matrices act on rows: $X$ has one row per token, so $X$ is $T \times d_{\text{model}}$ and a linear layer is $X W$ with $W$ of shape $d_{\text{in}} \times d_{\text{out}}$ .

Every component

Parameter counts are for GPT-2 small, including biases; per block where marked.

Component	Formula	Output shape	Parameters (GPT-2 small)	Lesson
Tokenizer	text → ids	$T$	0 (a merge table)	BPE
Token embedding	$X = W_E[\text{ids}]$	$T \times d_{\text{model}}$	$V d_{\text{model}}$ = 38.6M	embeddings
Learned positions	$X \mathrel{+}= W_{\text{pos}}[0..T{-}1]$	$T \times d_{\text{model}}$	$1024\, d_{\text{model}}$ = 786K	positions
RoPE (Llama)	rotate pairs of $q, k$ by angle $\propto$ position	unchanged	0	RoPE
LayerNorm / RMSNorm	$\gamma \odot \frac{x - \mu}{\sigma} + \beta$ / $\gamma \odot \frac{x}{\text{rms}(x)}$	$T \times d_{\text{model}}$	$2 d_{\text{model}}$ / $d_{\text{model}}$	normalisation
Q, K, V projections	$Q = \hat X W_Q$ , $K = \hat X W_K$ , $V = \hat X W_V$	$h \times T \times d_{\text{head}}$	$3 d_{\text{model}}^2 + 3 d_{\text{model}}$ = 1.77M per block	Q, K, V
Scores	$S = Q K^\top / \sqrt{d_{\text{head}}}$	$h \times T \times T$	0	scaling
Causal mask + softmax	$A = \operatorname{softmax}(S + M)$	$h \times T \times T$	0	causal mask
Mix values	$O = A V$	$h \times T \times d_{\text{head}}$	0	attention intuition
Output projection	$\operatorname{concat}(O)\, W_O$	$T \times d_{\text{model}}$	$d_{\text{model}}^2 + d_{\text{model}}$ = 591K per block	multi-head
Residual add	$x \leftarrow x + f(\operatorname{norm}(x))$	$T \times d_{\text{model}}$	0	residual stream
MLP (GELU)	$\operatorname{GELU}(\hat X W_1 + b_1) W_2 + b_2$	$T \times d_{\text{ff}}$ → $T \times d_{\text{model}}$	$2 d_{\text{model}} d_{\text{ff}} + d_{\text{ff}} + d_{\text{model}}$ = 4.72M per block	MLP
MLP (SwiGLU, Llama)	$(\operatorname{SiLU}(\hat X W_g) \odot \hat X W_u) W_d$	same	$3 d_{\text{model}} d_{\text{ff}}$	MLP
One block	attention + MLP, each with norm and residual	$T \times d_{\text{model}}$	7,087,872	the block
Final norm	$\operatorname{norm}(x)$	$T \times d_{\text{model}}$	$2 d_{\text{model}}$	normalisation
Unembedding	$\text{logits} = \hat X W_U$	$T \times V$	0 in GPT-2 (tied: $W_U = W_E^\top$ ); $d_{\text{model}} V$ otherwise	unembedding
Next-token probabilities	$p = \operatorname{softmax}(\text{logits}_{T})$	$V$	0	what an LLM does
Training loss	$-\frac{1}{T}\sum_t \log p(x_{t+1} \mid x_{\le t})$	scalar	0	training objective
KV cache	store $K, V$ for every layer	$2 \times L \times h_{\text{kv}} \times T \times d_{\text{head}}$	0 (memory, not weights)	KV cache
MoE router	$\operatorname{top-}k(\operatorname{softmax}(\hat x W_r))$	$E$ scores per token	$d_{\text{model}} E$	mixture of experts

Totals. GPT-2 small: $38{,}597{,}376 + 786{,}432 + 12 \times 7{,}087{,}872 + 1{,}536 = 124{,}439{,}808$ . Llama 3 8B: two $128{,}256 \times 4{,}096$ matrices (embedding and unembedding), 32 blocks of 218,112,000 and a final norm: $8{,}030{,}261{,}248$ .

GPT-2 small and Llama 3 8B

	GPT-2 small (2019)	Llama 3 8B (2024)
Parameters	124M	8.03B
Blocks $L$	12	32
$d_{\text{model}}$	768	4,096
Heads $h$ / KV heads	12 / 12	32 / 8 (grouped-query attention)
$d_{\text{head}}$	64	128
$d_{\text{ff}}$	3,072 (4×)	14,336 (3.5×, three matrices)
Vocabulary $V$	50,257	128,256
Context	1,024	8,192 (128K in Llama 3.1)
Positions	learned, added to the input	RoPE, $\theta$ = 500,000
Normalisation	LayerNorm, before each sublayer	RMSNorm, before each sublayer
MLP activation	GELU	SwiGLU
Biases	yes	no
Embedding and unembedding	tied (shared)	separate
Parameters per block	7.09M	218M
KV cache per token (16-bit)	$2 \times 12 \times 12 \times 64 \times 2$ bytes = 36.9 kB	$2 \times 32 \times 8 \times 128 \times 2$ bytes = 131 kB
Forward pass per token	≈ $2N$ = 0.25 GFLOPs	≈ $2N$ = 16 GFLOPs

The modern recipe explains each of the changes in the right-hand column, and what is new what came after.

Trace every shape

Pick a model and a number of tokens, then step through one forward pass. Click any row to jump to it.

tokens T5

token ids

tokenizer

[T] = [5] · 5 numbers

parameters per block7.1M

blocks12

total parameters124,439,808

The text as 5 integers, each between 0 and 50,256.

Every step of one forward pass. Shapes use the notation table above: T tokens, d_model, h heads of d_head, d_ff, vocabulary V. Rows marked with the amber bar repeat once per block. Parameter counts are exact and add up to the published totals.

Rules of thumb

Parameters per block ≈ $12\, d_{\text{model}}^2$ for a GPT-style block ( $4 d^2$ attention + $8 d^2$ MLP). GPT-2 small: $12 \times 768^2 = 7.08$ M, against 7.09M exactly.
Compute ≈ $2N$ FLOPs per token for a forward pass and $6N$ for training (forward plus backward), plus attention’s $T^2$ term for long contexts. See cost.
Memory for weights = $N$ × bytes per parameter: 8B parameters in 16-bit is 16 GB; see model memory.
KV cache = $2 \times L \times h_{\text{kv}} \times d_{\text{head}} \times$ bytes, per token. It is what limits long context and many users at once.
Attention scores grow with $T^2$ ; everything else grows with $T$ .

Vocabulary in one line each

Token: a chunk of text with an ID; the model only ever sees IDs.
Embedding: the learned vector for each token ID.
Residual stream: the running vector per token that every layer reads from and adds to.
Attention head: moves information between positions, weighted by query–key matches.
MLP: processes each position on its own; stores much of the model’s knowledge.
Logits: one raw score per vocabulary token; softmax turns them into probabilities.
KV cache: saved keys and values, so generation doesn’t recompute the past.
Induction head: a head that continues repeated patterns (interpretability).