Beyond attention

Key idea

Drop the softmax and attention becomes a running sum of key × value outer products: a fixed-size state. Add gates (Mamba, Mamba-2) to forget, a delta rule (DeltaNet) to overwrite, and keep a few full attention layers for exact recall

Real world

Taking notes in a notebook with a fixed number of pages instead of keeping a recording of the whole lecture. Cheap to carry and quick to consult, but if the lecture is long you have to summarise, and some details are lost.

In an LLM

Qwen3.5-397B-A17B (2026) has 60 layers: 45 are Gated DeltaNet layers that each keep a fixed state per head, and only 15 are full attention layers with a KV cache. It reads 262,144 tokens natively.

The cost that won’t go away

Attention compares each new token with every earlier one, and keeps every earlier key and value in the KV cache. Token number 100,000 costs a hundred thousand comparisons, and the cache keeps growing. The tricks on the previous page make this cheaper but do not change its shape.

Recurrent networks (RNNs), which came before transformers, had the opposite profile: they read text one token at a time into a fixed-size memory, so every token costs the same. Their problem was training: they could not be parallelised across a sequence, and they forgot. The ideas on this page aim for both: train in parallel like a transformer, generate with a fixed memory like an RNN.

Linear attention: a running sum

Write one attention output without the softmax, just a weighted sum of values with weights $q \cdot k_j$:

The bracket does not depend on the query. Call it the state $S_t$: a $d \times d$ matrix, the sum of the outer products $k_j v_j^\top$ of every key with its value. Each new token just adds its own outer product, $S_t = S_{t-1} + k_t v_t^\top$, and reads with $q_t^\top S_t$. That is Linear attention (Katharopoulos et al., 2020). Cost per token: constant. Memory: one $d \times d$ matrix per head, however long the text.

The softmax is what made attention sharp: it can put 99% of its weight on one key. Without it, every stored pair leaks a little into every read. The state is like a single sheet on which every fact is written on top of the others.

Three ways to update the state, all in use:

• Add (linear attention): $S_t = S_{t-1} + k_t v_t^\top$. Never forgets, never corrects.
• Decay (gated linear attention, Mamba-2): $S_t = \alpha_t S_{t-1} + k_t v_t^\top$, with a forget factor $\alpha_t$ between 0 and 1 computed from the current token. Old facts fade so new ones fit.
• Delta rule (DeltaNet, Schlag et al., 2021; Yang et al., 2024): first read what the state already says for this key, $S_{t-1}^\top k_t$, and write only the difference. If a fact changes, the old value is replaced instead of piled on.

Gated DeltaNet (Yang, Kautz and Hatamizadeh, 2024) combines the last two: decay to forget, delta rule to overwrite. Kimi Delta Attention (KDA, in Kimi Linear and Kimi K3) refines the gate so each channel of the state can forget at its own speed.

State space models and Mamba

State space models (State space model (SSM)), such as S4 (Gu et al., 2021), came from signal processing. They run a linear recurrence

with learned $a$, $b$, $c$. Because $a$ and $b$ are the same for every token, the whole sequence can be computed in parallel, but the model can’t decide to remember an important token or skip a filler word.

Mamba (Gu and Dao, 2023) made the recurrence selective: $a_t$, $b_t$ and $c_t$ are computed from the current token. A full stop can wipe the state; a filler word can pass through without writing. A hardware-aware “parallel scan” keeps training fast. Mamba-2 (Dao and Gu, 2024) simplified the gate to one number per head per token and showed the result is exactly a masked attention with decay weights, so it can be computed either as a recurrence (for generation) or as matrix multiplications (for training).

What they trade away

A fixed state means fixed capacity. A $d \times d$ state can keep about $d$ unrelated facts cleanly; beyond that, they interfere. Softmax attention has no such limit, because it keeps everything, and that is exactly what it pays for in memory.

This shows up most in recall tasks: “what was the code mentioned earlier?”, copying a long passage, answering from a document. Studies found pure recurrent models fall behind transformers of the same size here (Zoology, Arora et al., 2023; Jelassi et al., 2024), even when they match them on ordinary next-token loss.

Hybrids: the 2025–26 answer

If most layers can be cheap and a few keep exact lookup, you get most of both. Recent models do exactly that:

• Jamba (AI21, 2024, paper): one attention layer for every seven Mamba layers, with MoE; 52B total, 12B active, 256K context.
• Qwen3-Next-80B-A3B (2025) and Qwen3.5-397B-A17B (2026): three Gated DeltaNet layers for every gated attention layer, 262,144-token native context.
• Kimi K3 (2026): 69 KDA layers and 24 MLA layers (3 to 1, plus one final MLA layer), 2.8T parameters, 1M-token context. Its attention layers use no position encoding at all: the recurrent KDA layers already know the order.
• Nemotron-H (NVIDIA, 2025, paper): mostly Mamba-2 layers with a few attention layers.

Not everyone went this way. MiniMax’s M1 (2025) used a hybrid of linear “lightning” attention and softmax attention, but its M2 (late 2025) went back to full attention in every layer, and DeepSeek-V4 kept softmax attention in every layer, compressing and sparsifying it instead (see efficient attention).