The whole block

Put the pieces together: LayerNorm, attention, add; LayerNorm, MLP, add. That is one transformer block, and a model is a stack of identical blocks between an embedding table and an output layer. Here is one block computed number by number, and an exact parameter count for real models.

Key idea: x ← x + Attn(LN(x)); x ← x + MLP(LN(x)); stack N times; count parameters ≈ 12·N·d² + vocab·d
Real world: An assembly line of identical workstations. Each station has the same two tools, a “look around” tool and a “think about it” tool, and adds its work to the product on the belt. A bigger factory just has more stations and wider belts.
In an LLM: GPT-2 small is 12 identical blocks with width 768: 124 million parameters. Llama 3 70B is 80 blocks with width 8,192: 70.6 billion. Same recipe, different sizes.

One block, start to finish

You have met every part. A pre-norm , as used by GPT-2 and Llama, is:

\begin{aligned} x' &= x + \operatorname{Attention}\big(\operatorname{LN}_1(x)\big)\\ x_{\text{out}} &= x' + \operatorname{MLP}\big(\operatorname{LN}_2(x')\big) \end{aligned}

where $x$ is the whole sequence, one row per token. Six steps:

LayerNorm each token’s row (a copy; $x$ itself is kept). Normalization.
Attention on the normalised copy: queries, keys, values, scores, mask, softmax, mix, then an output matrix $W_O$ that maps the heads’ results back to the model width. Queries, keys and values, multi-head.
Add the attention output to $x$ . The residual stream.
LayerNorm again.
MLP on each token separately: widen, activate, narrow. The MLP.
Add the MLP output.

The shape on the main path never changes: $T$ tokens × $d$ numbers in, $T \times d$ out. That is what lets blocks stack.

input

→→→→→→

follow:

x5 × 4

thingact.smallfirstThe0.10.01.01.0cat1.00.10.00.0sat0.21.00.00.0on0.00.30.80.0the0.10.01.00.0

−2.02.0

The block’s input: 5 tokens × 4 numbers. Click a token to follow its row. Every step below keeps the shape 5 × 4 on the main path.

One complete pre-norm transformer block on the toy sentence. Every number is computed live: the attention weights come from the toy head, the MLP weights from the MLP page. Press play or click a stage.

The toy uses width 4 so you can read every number. GPT-2 small does the same with width 768, 12 heads of size 64 and an MLP of width 3072, in each of its 12 blocks.

The whole model

A full decoder-only transformer is:

Embed: look up each token’s row in the embedding table, add position information. Embeddings, positions.
Blocks: run the $T \times d$ matrix through $N$ identical blocks (each with its own weights).
Final norm and unembed: normalise, multiply by the output matrix to get one score per vocabulary token. Unembedding.

Model	Blocks $N$	Width $d$	Heads	MLP width	Vocabulary
GPT-2 small	12	768	12	3,072	50,257
GPT-2 XL	48	1,600	25	6,400	50,257
Llama 3 8B	32	4,096	32 (8 KV)	14,336	128,256
Llama 3 70B	80	8,192	64 (8 KV)	28,672	128,256

Counting the parameters

Every parameter lives in one of five places: the embedding table, the attention matrices, the MLP matrices, the norms, and (unless it is tied to the embedding) the output matrix. With full multi-head attention and a 4× MLP, each block has about $4d^2 + 8d^2 = 12d^2$ weights, so

\text{parameters} \approx 12\,N\,d^2 + \text{vocab} \times d.

For GPT-2 small: $12 \times 12 \times 768^2 = 84.9\text{M}$ in the blocks plus $50{,}257 \times 768 = 38.6\text{M}$ in the embeddings, 123.5M; the biases, norms and position table bring it to exactly 124,439,808. Try the presets:

d_model768layers12heads12d_ff3072vocab50,257

tied embeddings

32%23%46%

embeddings 39.4Mattention 28.3MMLPs 56.7Mnorms 38.4K

where the formula comes from

embeddings = vocab × d + positions (1024) × dattention / layer = 2d² (Q, O) + 2 × d × kv·d/heads (K, V) + biasesMLP / layer = 2 × d × d_ff + biasesnorms = 2 per layer × 2d (gain, bias) + a final oneoutput layer = 0 (shares the embedding matrix)

total parameters124,439,808

per block7.1M

in bf16 (2 bytes each)249 MB

GPT-2 small: 124,439,808 parameters, exactly the 124,439,808 in the published checkpoint (Hugging Face gpt2). The MLPs are 46% of it.

Every weight matrix and vector in the model, counted from the configuration. Pick a preset (the total is checked against the published checkpoint) or move the sliders to design your own.

Every preset matches the published checkpoint to the last parameter. Two things to notice:

Blocks grow with $d^2$ , embeddings with $d$ . In GPT-2 small the token and position tables are 32% of the model; in Llama 3 70B the embedding and output matrices together are only 3%. Small models spend a surprising share on their vocabulary.
Bytes follow directly. Stored in bf16 (2 bytes per parameter) Llama 3 8B needs about 16 GB just for its weights, before the KV cache and activations. How much memory a model needs works through this, and how quantization shrinks it.

example · an assembly line

A car factory has a line of identical workstations. At each one, a worker looks at the car and at the cars next to it on the line (attention), thinks about what this car needs (the MLP), and bolts on a part (the add). Nothing is ever removed. A bigger factory doesn’t invent new kinds of station; it builds more of them and makes the line wider. Doubling the width of every station roughly quadruples the cost of each one, which is the $d^2$ .

example · GPT-2 small to Llama 3 70B

GPT-2 small (2019) and Llama 3 70B (2024) are built from the same block. Llama swaps LayerNorm for RMSNorm, GELU for SwiGLU, learned positions for RoPE, and shares keys and values across groups of heads. It is 6.7 times deeper and 10.7 times wider, and 10.7² × 6.7 ≈ 760: almost all of the 567-fold growth in parameters (124M to 70.6B) comes from width squared times depth.

What one block costs to run

Each parameter in a block is used once per token in one multiply and one add, so a forward pass costs about $2 \times \text{parameters}$ floating-point operations per token, plus the attention scores that grow with the context length. What a forward pass costs works this out properly.

going deeper

The complete count

With $h$ query heads, $h_{kv}$ key–value heads, head size $d_h = d/h$ , MLP width $d_{ff}$ and vocabulary $V$ :

\begin{aligned} \text{attention} &= 2d^2 + 2\,d\,h_{kv} d_h \;(+\text{biases})\\ \text{MLP} &= 2\,d\,d_{ff} \;(\text{GELU, + biases}) \quad\text{or}\quad 3\,d\,d_{ff} \;(\text{SwiGLU})\\ \text{norms} &= 2 \text{ per block} \times d \;(\text{RMSNorm}) \text{ or } 2d \;(\text{LayerNorm})\\ \text{total} &= V d \;(+ P d \text{ learned positions}) + N(\text{attention} + \text{MLP} + \text{norms}) + \text{final norm} \;(+ V d \text{ if untied}) \end{aligned}

Check it yourself

from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("gpt2")
print(sum(p.numel() for p in m.parameters()))   # 124439808 (the tied output matrix is counted once)

In papers

The original Transformer: Vaswani et al., 2017.
GPT-2 (pre-norm, learned positions): Radford et al., 2019.
Llama 3 (RMSNorm, SwiGLU, RoPE, GQA): Grattafiori et al., 2024.
Parameter and compute counts: Kaplan et al., 2020, section 2.1.