Entropy collapse

RL tends to make a model ever more certain. Once it stops trying alternatives, learning stalls. This page is about keeping just enough randomness.

Key idea: Policy entropy, why RL drains it, and the fixes
Real world: A new employee finds one way of doing a task that works, and from then on always does it that way. They never find out whether a better way exists, and they stop improving.
In an LLM: After a few hundred steps, all eight answers in a GRPO group start the same way and follow the same template. Groups stop having any spread, advantages go to zero, and the reward curve flattens.

What entropy measures

is how spread out a probability distribution is. Zero means one option has all the probability. It's highest when every option is equally likely. For a language model it is measured per token, then averaged over the answer.

example · two distributions

A fair coin:

-(0.5 \ln 0.5 + 0.5 \ln 0.5) = 0.69

A model choosing the next token, 97% on one and 1% on each of three others:

-(0.97 \ln 0.97 + 3 \times 0.01 \ln 0.01) = 0.17

The second has four options, but it almost always picks the same one.

Why RL drains it

Policy gradient makes rewarded answers likelier. The answers sampled most often are the ones the model already prefers, so they collect the most reward and grow further. Rare alternatives are rarely sampled, so they rarely get the chance to be rewarded. Left alone, the distribution narrows every step.

Cui et al. (2025) measured this across many runs and found performance closely tracking entropy: almost all of the gain happens while entropy is being used up, and when it's gone, the curve flattens at a predictable ceiling. In other words, entropy is the budget RL spends on improvement.

Try it

entropy bonus0.00

final success rate87%

final entropy1%

runs that found the best approach10 / 20

entropysuccess ratebest approach (95%)

Policy over the eight approaches, ordered by how often each one succeeds

at the startafter training (average over runs)

Entropy falls to almost zero within about 50 steps. Each run locks onto whatever approach got lucky early, and only 10 of 20 find the best one. After the collapse, success stops improving: there is nothing left to explore.

20 training runs on one problem with eight possible approaches, averaged. Entropy is scaled so 100% means all eight are equally likely.

Fixes used in practice

Clip-higher (DAPO): a higher upper clip lets rare tokens that worked grow faster.
Train on the forks: Wang et al. found that only about 20% of tokens in a reasoning chain have high entropy. These are the decision points, like choosing a method or deciding to double-check. Updating only those tokens matched or beat updating all of them.
Covariance-based clipping (Clip-Cov, KL-Cov): limit updates on the specific tokens that drive entropy down fastest, the confident ones with large advantages.
Adaptive entropy bonus: add a small extra reward for keeping choices spread out, but only when entropy falls below a target.
Temperature: sample rollouts at temperature 1 or a little above.

example · forks on a drive

Most of a car journey is forced: you follow the road. The few decisions that matter happen at junctions. Practising the whole route equally is wasteful; practising the junctions is what changes where you end up.

example · forks in an answer

In 17 × 6 = 17 × 5 + 17 = 85 + 17 = 102, almost every token follows from the one before. The model has a real choice at the start (split as 17 × 5 + 17, or as 10 × 6 + 7 × 6?) and maybe at the end (stop, or write "Let me check"). Those are the high-entropy tokens, and they're where RL's signal is most useful.

going deeper

Definitions

H(\pi(\cdot \mid s)) = -\sum_a \pi(a \mid s) \log \pi(a \mid s)

Cui et al. show that for a softmax policy trained with (natural) policy gradient, the change in entropy at a state is approximately

\Delta H \approx -\eta \cdot \operatorname{Cov}_{a \sim \pi}\big(\log \pi(a \mid s),\ A(s, a)\big)

Entropy falls when high-probability actions also have high advantage, which is the normal case once a model is decent. They also fit $R \approx -a\, e^{H} + b$ between reward $R$ and entropy $H$ across training.

Papers

The Entropy Mechanism of RL for Reasoning Language Models (Cui et al., May 2025)
Beyond the 80/20 Rule: High-Entropy Minority Tokens (Wang et al., Jun 2025)
DAPO (clip-higher)