The modern recipe: GPT-2 to 2026

Key idea

Same skeleton; swapped parts: LayerNorm → RMSNorm (pre-norm), GELU → SwiGLU, learned positions → RoPE (→ none), MHA → GQA / MLA / hybrid, dense → MoE, biases removed, QK-norm added, untied embeddings, 50K → 130–160K vocabularies, 1K → 128K–1M context

Real world

A 1960s car and a 2026 car both have four wheels, an engine, brakes and a steering wheel. Almost every part has been redesigned (fuel injection, disc brakes, airbags), but a mechanic from 1965 would still recognise the layout.

In an LLM

GPT-2 XL (2019): 1.5B parameters, 1,024-token context, every parameter used for every token. Kimi K3 (2026): 2.8T parameters, 104B used per token, 1M-token context. Both turn a list of tokens into next-token probabilities through a stack of blocks.

Same skeleton

Everything in the block lesson still holds. A token becomes a vector; a stack of blocks updates it on the residual stream; each block mixes information between tokens (attention or something like it) and then processes each token on its own (a feed-forward network); a final layer turns the vector into logits. Training is still next-token prediction.

What changed is the parts. Compare any two of five landmark open models:

The changes, one by one

Normalisation: RMSNorm, before each sub-layer. GPT-2 had already moved LayerNorm to the start of each sub-layer (“pre-norm”), which trains more stably than the 2017 design. Later models switched to RMSNorm, which only rescales the vector by its root mean square and has no mean or bias to learn. See normalization.

Activation: SwiGLU. The MLP in GPT-2 was “expand, apply GELU, shrink”. Since PaLM and Llama it has been SwiGLU (Shazeer, 2020): two expansions, one passed through a smooth gate (SiLU) and multiplied by the other. It has three weight matrices instead of two, so the hidden size is usually made smaller to compensate: 8/3 of the width in the original Llama, 3.5 times in Llama 3 8B (14,336 for a width of 4,096), against GPT-2’s 4 times. Kimi K3 bounds both branches with tanh to stop rare huge activations.

Positions: RoPE, then nothing at all. GPT-2 learned one vector per position, up to 1,024. RoPE rotates queries and keys instead, so scores depend on distance and can be stretched for longer contexts. Kimi K3 goes further: its full-attention layers use no position signal at all, and its linear-attention layers, which read tokens in order, supply it.

Attention: share or compress the keys and values. Every GPT-2 head had its own keys and values. GQA and MLA share or compress them, cutting the KV cache 4 to 70 times. Hybrids make most layers linear attention with a fixed-size state.

Feed-forward: mixture of experts. Since DeepSeek-V3, most large open models replace the MLP with many experts and a router. Kimi K3 picks 16 of 896 per token.

No biases. GPT-2 added a learned bias vector after every matrix multiplication. Modern models drop them all (Qwen kept them on queries, keys and values through Qwen2.5 and dropped them in Qwen3). They cost little but add nothing at this scale, and removing them helps stability.

QK-norm. Qwen3 normalises every query and key before the dot product (Dehghani et al., 2023), which stops attention scores from blowing up mid-training; Kimi K3’s linear layers normalise them too.

Untied embeddings. GPT-2 used one matrix both to look up tokens and to score them at the output. Large models now keep two; small ones (Qwen3 up to 4B, Llama 3.2 1B and 3B) still tie them to save memory.

Bigger vocabularies. 50,257 tokens in GPT-2; 128,256 in Llama 3; 151,669 in Qwen3; 163,840 in Kimi K3. Fewer tokens per text, especially for code and non-English languages. See vocabulary size.

Longer contexts. 1,024 tokens in GPT-2; 128K became normal in 2024; 1M in DeepSeek-V4 and Kimi K3 in 2026. Every other change on this list (RoPE scaling, smaller KV caches, FlashAttention, hybrids) was needed to get there.

What has not changed

• Next-token prediction with a softmax over the vocabulary, trained with cross-entropy.
• The residual stream: blocks still mostly add their output to a running vector. (Kimi K3’s Attention Residuals and DeepSeek-V4’s hyper-connections are the first big changes here in years; it is too early to say whether they become standard.)
• Softmax attention in at least some layers. Even the most aggressive hybrids keep a full attention layer every few, for exact recall.
• Byte-pair-encoded tokens. Bigger vocabularies, same idea (see beyond tokens for the experiments that try to drop them).

Where it is heading

Three trends run through 2025–26:

1. Sparser everything. More and smaller experts (a few percent of weights active per token), sparse attention that reads a few thousand keys out of a million (DeepSeek), and linear layers with fixed state.
2. Hybrids. Qwen3.5 and Kimi K3 put linear attention in three of every four layers. DeepSeek-V4 instead keeps compressed sparse attention everywhere. Which approach wins is still open.
3. Designed for low precision. Models are trained or fine-tuned to run in 8-bit or 4-bit (Kimi K3 ships with 4-bit MXFP4 weights and 8-bit activations; DeepSeek-V4 with 4-bit experts and 8-bit elsewhere), because at trillions of parameters memory is the limit. See quantization.