What is new: the frontier in late 2026 · Transformers

The transformer of 2017 is still the core of every frontier model, but almost every part around it has changed. This page surveys what changed in 2024–2026, with sources, and says which ideas are now standard and which are still experiments.

Key idea: Million-token context, hybrid and sparse attention, very sparse MoE, multi-token prediction, reasoning and looped depth, diffusion and byte-level models, new optimisers
Real world: Cars from 1990 and 2026 both have four wheels, an engine and a steering wheel. Underneath, the fuel injection, the brakes and the electronics have been replaced several times over.
In an LLM: GPT-2 (2019) has 12 identical layers of full attention and a dense MLP. Qwen3.5 (2026) has 60 layers, three in every four of which use linear attention, and each layer picks 10 of 512 experts, so only 17B of its 397B parameters work on any one token.

How to read this page

Each section is one direction of change: what it is, who uses it, a source, and a status label.

Established: used in several leading production models.
Spreading: in production at one or two labs, others are trying it.
Experimental: promising papers and small models, not yet in frontier production models (as far as public information shows).

Closed labs (OpenAI, Anthropic, Google) publish few architecture details, so most specifics below come from open-weight models and papers. “Frontier” here means the models available in October 2026. Here is the whole period at a glance:

Jun 2017The transformer

“Attention is all you need”: the architecture this course is about.

Sep 2022Superposition and induction heads

Anthropic’s toy models of superposition and the induction-head paper.

Sep 2023Attention sinks

StreamingLLM explains why heads dump attention on the first token, and keeps it to run on endless streams.

Feb 2024Gemini 1.5: a million tokens

Google announces a model that takes a 1M-token context, later standard in its API.

Apr 2024Multi-token prediction

Meta shows predicting several tokens at once helps on code and allows up to 3× faster inference.

May 2024Scaling monosemanticity

Millions of readable features found in Claude 3 Sonnet; Golden Gate Claude.

Sep 2024OpenAI o1

The first widely used reasoning model: accuracy grows with thinking time.

Dec 2024DeepSeek-V3

671B parameters, 37B active, 256 small experts, one extra predicted token.

Dec 2024Muon optimiser

Keller Jordan’s orthogonalised-momentum optimiser, first used in speed-run records.

Dec 2024Byte Latent Transformer

Meta’s model that reads raw bytes in patches, with no tokenizer.

Jan 2025DeepSeek-R1

Reasoning learned by reinforcement learning, with open weights.

Feb 2025Recurrent depth and LLaDA

Huginn loops a block to think in latent space; LLaDA is an 8B diffusion language model.

Feb 2025Native sparse attention

DeepSeek’s NSA: learned sparse attention trained from the start.

Mar 2025Circuit tracing

Attribution graphs for Claude 3.5 Haiku: multi-step reasoning, planning rhymes.

Apr 2025Llama 4

Meta’s first MoE models; Scout advertises a 10M-token context.

May 2025Gemini Diffusion

Google’s experimental diffusion text model, about 1,500 tokens per second.

Jul 2025Kimi K2 and MuonClip

1T parameters trained with a Muon variant on 15.5T tokens with no loss spikes.

Aug 2025gpt-oss

OpenAI’s open-weight MoE models: 128 experts, 4 active, alternating window and full attention.

Sep 2025DeepSeek Sparse Attention

V3.2-Exp reads only the top 2,048 tokens per query; API prices cut by more than half.

Sep 2025Qwen3-Next

3 linear-attention (Gated DeltaNet) layers per full-attention layer, 512 experts.

Oct 2025Kimi Linear; MiniMax M2

Kimi’s linear hybrid decodes up to 6× faster at 1M tokens; MiniMax goes back to full attention.

Oct 2025Tiny recursive model

A 7M-parameter looped network scores 45% on the ARC-AGI-1 puzzles.

Feb 2026Qwen3.5 and GLM-5

Qwen3.5: 397B total, 17B active, 3:1 linear hybrid. GLM-5 adopts DeepSeek Sparse Attention.

Apr 2026DeepSeek-V4

1.6T parameters, 1M context with compressed sparse attention: 10% of V3.2’s cache at 1M tokens.

Jun 2026MiniMax-M3

MiniMax adopts learned block-sparse attention for a 1M-token context.

Jul 2026Kimi K3

2.8T parameters, 104B active, 896 experts, 69 linear + 24 full-attention layers.

A selection of the developments on this page, with the month each paper or model appeared. Filter by kind.

1. Long context: a million tokens, used unevenly

Established. Gemini 1.5 introduced a 1-million-token context window in 2024 (Gemini Team, 2024). In 2026 it is common for open models too: DeepSeek-V4, Kimi K3 and MiniMax-M3 all advertise 1M tokens, and Llama 4 Scout advertises 10M.

Most models are not trained on million-token texts from the start. They are trained shorter and then extended: the RoPE rotation frequencies are rescaled so that long positions look like positions the model has seen. Position Interpolation (Chen et al., 2023) squeezes positions linearly; YaRN (Peng et al., 2023) rescales different frequencies by different amounts and needs about 10× fewer training tokens. Qwen3.5, for example, supports 262,144 tokens natively, and its model card describes YaRN scaling (factor 4) to reach about 1 million.

Long context exposed an odd habit. Many heads put most of their attention on the very first token, an , because softmax weights must add up to 1 and a head with nothing to look at needs somewhere to put them (Xiao et al., 2023). You saw this in the induction circuit. Models now handle it on purpose: gpt-oss adds a learned “sink” term to every head’s softmax, and Qwen’s gated attention (Qiu et al., 2025, a NeurIPS 2025 best paper) multiplies each head’s output by a learned gate, which removes the sinks.

The catch: advertised context is not usable context. On the NoLiMa benchmark, where the question and the answer don’t share obvious keywords, 11 of 13 models scored below half of their short-context score at just 32K tokens (Modarressi et al., 2025). Chroma’s “context rot” study of 18 models found performance falls steadily as input grows, even on simple tasks (Hong et al., 2025).

example · a very long meeting

You can sit through an eight-hour meeting, but if someone asks at the end what was decided about the budget in hour two, you will probably miss a detail, especially if it was phrased differently from the question. Being in the room for the whole meeting (context length) isn’t the same as remembering it well (effective context).

2. Attention that doesn’t read everything

Full attention reads every earlier token in every layer. At a million tokens, that cost (and the KV cache) dominates. Three answers are now in production, and they are often mixed within one model:

Sliding windows (established): most layers only look at the last few thousand tokens; a few full layers keep the long view. Gemma 3 uses five 1,024-token window layers per full layer (Gemma Team, 2025).
Linear-attention hybrids (spreading): most layers are replaced by layers that keep a fixed-size state instead of a growing cache (see beyond attention). Qwen3-Next and Qwen3.5 use three Gated DeltaNet layers per full-attention layer; Kimi Linear (Kimi Team, 2025) and Kimi K3 use the same 3:1 idea with their own “Kimi Delta Attention”; the Kimi Linear paper reports up to 6× the decoding throughput at 1M tokens. NVIDIA’s Nemotron models do it with Mamba layers.
Learned sparse attention (spreading): keep the full cache, but let a small, cheap scorer pick which tokens each query reads. DeepSeek’s NSA (Yuan et al., 2025) and DeepSeek Sparse Attention (in V3.2, where each query reads 2,048 tokens) led the way; GLM-5, MiniMax-M3 and DeepSeek-V4 use variants. DeepSeek reports that V4-Pro needs 10% of V3.2’s KV cache and 27% of its compute per token at 1M context (DeepSeek-AI, 2026).

full attention × 15: reads every earlier tokenlinear attention (DeltaNet) × 45: reads a fixed-size state

context1M tokens

layers whose cache grows15 of 60

cache vs all-full25%

tokens read per query vs all-full25%

2026. Three Gated DeltaNet layers (linear attention with a fixed-size state) for every gated full-attention layer: 45 + 15. Kimi K3 uses a similar 3:1 mix (69 linear + 24 full).

Each square is one layer, in order from the first (top left) to the last. Colour shows how that layer’s attention looks back. Pick a model and a context length. Cache sizes are relative to a model of the same shape with full attention in every layer; the small fixed-size state of linear layers is ignored.

It isn’t settled. MiniMax shipped a linear hybrid (MiniMax-01, 2025), went back to full attention for M2 because the hybrid did worse on multi-step reasoning at scale, then moved to learned sparse attention for M3 in 2026. Pick “gpt-oss-120b” or “Gemma 3” above: the simple local-plus-global recipe is still widely used.

3. Mixture of experts: more experts, smaller, fewer active

Established. Almost every large open model since 2025 is a mixture of experts, and Google’s Gemini 3 model card says it is one too. The trend is towards many small (“fine-grained”) experts plus one or two shared experts every token uses (Dai et al., 2024):

Model	Experts per layer (active + shared)	Total / active parameters	Active share
DeepSeek-V3 (Dec 2024)	256 (8 + 1)	671B / 37B	5.5%
gpt-oss-120b (Aug 2025)	128 (4)	117B / 5.1B	4.4%
Kimi K2 (Jul 2025)	384 (8 + 1)	1T / 32B	3.2%
Qwen3.5 (Feb 2026)	512 (10 + 1)	397B / 17B	4.3%
Kimi K3 (Jul 2026)	896 (16 + 2)	2.8T / 104B	3.7%
DeepSeek-V4-Pro (Apr 2026)	384 (6 + 1)	1.6T / 49B	3.1%

Balancing the load across experts used to need an extra loss term that slightly hurt quality. DeepSeek’s auxiliary-loss-free method instead nudges a per-expert bias up or down depending on how busy the expert has been (Wang et al., 2024), and has been widely copied.

4. Multi-token prediction

Established in open models. trains the model to predict several future tokens at each position, using small extra heads on top of the shared network (Gloeckle et al., 2024). It gives a richer training signal, and at inference the extra heads give free drafts for speculative decoding: the next forward pass checks the drafts and keeps the ones that were right. DeepSeek-V3 predicts one extra token, which is accepted 85 to 90% of the time, for about 1.8× faster generation (DeepSeek-AI, 2024). Qwen3.5, GLM-5, Kimi K3 and DeepSeek-V4 all have MTP layers.

extra heads k1guess accepted85%

pass 0 of 11

the text, built pass by pass

Press play.

tokens so far0

forward passes0

tokens per pass, this run–

expected1.85

With 1 extra head and 85% of guesses right, each pass produces 1.85 tokens on average instead of 1. This is DeepSeek-V3’s setting: it reports 85 to 90% acceptance and about 1.8× faster generation.

Each forward pass produces the next token for sure (dark) plus k cheap guesses from the extra prediction heads. Guesses are checked by the next pass for free: green ones are kept, the first wrong one (red) and everything after it is thrown away.

example · finishing someone’s sentence

A friend who knows you well often says the next two or three words with you. When they are right, the conversation speeds up; when they are wrong, you just correct them and carry on. Nothing is lost except their guess. Speculative decoding with MTP heads works the same way: wrong drafts cost a little compute, never a wrong answer.

5. Thinking longer: reasoning models

Established. Since OpenAI’s o1 (September 2024), frontier models are trained with reinforcement learning to write a long chain of reasoning before answering, and accuracy keeps rising with both training compute and the thinking time allowed. This is : spend more computation per question, not more parameters. DeepSeek-R1 (DeepSeek-AI, 2025) showed it could be learned by RL from a base model: in R1-Zero’s training, its score on AIME 2024 maths problems rose from 15.6% to 77.9%. By 2026 thinking modes, thinking budgets and effort levels are standard features. This changed training more than architecture; see reinforcement learning for the method.

6. Thinking in loops: recurrent depth

Experimental. Reasoning models think in words. An alternative is to think in vectors: run the same layers several times on each token. A model like Huginn (Geiping et al., 2025) has a prelude, a core block it can repeat any number of times, and a coda. Its 3.5 billion parameters, looped enough, use about as much compute per token as a 50-billion-parameter model. Related work: Coconut feeds the last hidden state back in as the next input instead of writing a token (Hao et al., 2024); ByteDance’s Ouro (Zhu et al., 2025) trained looped models on 7.7T tokens; and a 7-million-parameter Tiny Recursive Model reached 45% on the ARC-AGI-1 puzzles (Jolicoeur-Martineau, 2025).

loops r4

0 of 20 layers run

different layers (weights)8

layers run per token20

compute vs one pass of 82.5×

With r = 4, every token runs through 20 layers but the model only stores 8. Huginn was trained with a random number of loops (32 on average), so at test time you can choose r: more loops for harder questions.

The shape of Huginn (Geiping et al., 2025): 2 prelude layers, a core of 4 layers that runs r times, then 2 coda layers. The same 8 layers of weights are used whatever r is. Press play to watch one token go through.

As far as public information shows, no frontier model uses looped depth yet. The open questions are how well it scales and how to train it stably.

7. Not left to right: diffusion language models

Experimental, with niche products. A starts from fully masked text and fills in many tokens per step, refining the whole answer over a few rounds, the same idea as diffusion image models. LLaDA (Nie et al., 2025) trained an 8B diffusion model from scratch that is competitive with Llama 3 8B. Inception’s Mercury (Inception Labs, 2025) and Google’s Gemini Diffusion demo generate over 1,000 tokens per second. Quality still trails the best autoregressive models on knowledge and reasoning tests: Google lists Gemini Diffusion at 40.4% on GPQA Diamond versus 56.5% for Gemini 2.0 Flash-Lite. Block diffusion (Arriola et al., 2025) mixes the two: left to right between blocks, diffusion inside each block.

8. Training tricks: Muon, QK-norm, gates and new residuals

QK-norm (established): normalise queries and keys before the dot product, so attention scores can’t blow up during training. Used in OLMo 2, Gemma 3 and Qwen3 among others.
(spreading): an optimiser that orthogonalises each update to a weight matrix, so no single direction dominates (Jordan, 2024). Moonshot showed it scales, at about 2× the compute efficiency of AdamW (Liu et al., 2025), and trained the 1T-parameter Kimi K2 with it; DeepSeek-V4 uses it too.
Gated attention outputs (spreading): a sigmoid gate on each head’s output (Qwen3-Next, Qwen3.5, Kimi K3), the same trick that removes attention sinks.
New residual connections (experimental in 2026): the plain “add each layer’s output” residual stream is being redesigned. DeepSeek-V4 uses manifold-constrained hyper-connections, several mixed residual streams (DeepSeek-AI, 2025); Kimi K3 uses “attention residuals”, where each layer attends over earlier layers’ outputs instead of just summing them.

9. No tokenizer

Experimental. Tokenizers cause quirks, so researchers keep trying to remove them (see beyond tokens). Meta’s Byte Latent Transformer (Pagnoni et al., 2024) groups raw bytes into patches, with bigger patches where the next byte is easy to predict, and matched tokenized models at up to 8B parameters. H-Net (Hwang, Wang and Gu, 2025) learns where to cut the byte stream as part of training. No frontier model is tokenizer-free yet.

example · the overall picture in one model

DeepSeek-V4-Pro (April 2026) puts most of this page in one place: a mixture of experts with 1.6T parameters and 49B active, compressed sparse attention for a 1M-token context, multi-token prediction, the Muon optimiser and a redesigned residual stream, plus a reasoning mode. Underneath, each layer still does what this course described: attention to mix information between tokens, then an MLP (here, experts) at each token, both added to a residual stream.

What hasn’t changed

Next-token prediction with a cross-entropy loss, token embeddings, attention with queries, keys and values, RoPE, RMSNorm before each sublayer, SwiGLU MLPs and the residual stream are all still there in almost every 2026 model. The modern recipe page describes that common core; this page is what is being built on top of it.

going deeper

Status summary

Direction	Status (late 2026)	Representative source
1M-token context with RoPE scaling	Established	YaRN
Sliding window + global layers	Established	Gemma 3
Linear-attention hybrids (3:1)	Spreading	Kimi Linear
Learned sparse attention	Spreading	NSA, DeepSeek-V3.2
Fine-grained MoE + shared experts	Established	DeepSeekMoE
Multi-token prediction	Established in open models	Gloeckle et al.
RL-trained reasoning	Established	DeepSeek-R1
Recurrent depth / looping	Experimental	Geiping et al.
Diffusion LMs	Experimental	LLaDA
Muon optimiser	Spreading	Liu et al.
Byte-level models	Experimental	BLT

Why speculative decoding gives (1 − aᵏ⁺¹)/(1 − a) tokens per pass

With $k$ draft tokens each accepted with probability $a$ (independently, stopping at the first rejection), a pass yields the one certain token plus $j$ accepted drafts with probability $a^j(1-a)$ for $j < k$ , or $a^k$ for all $k$ . The expected count is

\mathbb{E}[\text{tokens}] = 1 + a + a^2 + \dots + a^k = \frac{1 - a^{k+1}}{1 - a}.

For DeepSeek-V3 ( $k = 1$ , $a \approx 0.85$ – $0.9$ ) this is 1.85–1.9 tokens per pass; the reported 1.8× speed-up is a little lower because the extra head costs some compute.

More sources

Long context benchmarks: RULER, NoLiMa, Context rot
Model reports: DeepSeek-V3, Kimi K2, gpt-oss, Qwen3, GLM-5, DeepSeek-V4, MiniMax-01, Qwen3.5 model card, Kimi K3 model card, MiniMax-M3 model card
MiniMax on returning to full attention: blog, Oct 2025
Auxiliary-loss-free load balancing: Wang et al., 2024
Latent reasoning: Coconut, Ouro, HRM, TRM, Mixture-of-Recursions
Diffusion: Mercury, Gemini Diffusion, Dream 7B, Block diffusion
Normalisation: OLMo 2 (QK-norm, reordered norms)