Entropy

How uncertain a distribution is, measured as the average surprise of its outcomes. Zero means certain; higher means more spread out.

Key idea: Surprise, and entropy as average surprise
Real world: The weather in a desert, sunny 99% of days, tells you almost nothing new each morning. The weather in a city where rain and sun are equally likely keeps you guessing. The second has higher entropy.
In an LLM: After "The capital of France is", the next token is nearly certain: low entropy. After "My favourite food is", many tokens are plausible: high entropy.

Surprise

How surprised should you be by an outcome? The rarer it is, the more surprising. A natural measure:

\text{surprise} = -\log_2 p

Probability	Surprise (bits)
1 (certain)	0
0.5 (coin flip)	1
0.25	2
0.01	6.6

A bit is the surprise of one fair coin flip: the answer to one yes/no question with even odds.

Entropy is average surprise

Entropy is the surprise of each outcome, weighted by how often it happens:

H = \sum_x p(x) \times (-\log_2 p(x))

Rare outcomes are very surprising, but they rarely happen, so they contribute little. Entropy is high only when many outcomes are reasonably likely.

Try it

Weights

A8.0B1.0C0.5D0.5

Probability, surprise (bits), probability × surprise

A80%0.32= 0.26

B10%3.32= 0.33

C5%4.32= 0.22

D5%4.32= 0.22

entropy1.02 bits

maximum for 4 outcomes2.00 bits

in nats (×0.693)0.71

The dashed line is the average height: the entropy. A rare outcome makes a tall, thin column; a common one a wide, short column. Entropy is high only when many columns are both wide and tall.

Four possible outcomes. Each column is as wide as the outcome’s probability and as tall as its surprise, −log₂(probability). Entropy is the shaded area: the surprise averaged over how often each outcome happens.

In the chart, each outcome is a column as wide as its probability and as tall as its surprise, so its area is probability × surprise and the total area is the entropy. A rare outcome is a tall, thin sliver: very surprising, but it adds little.

example · desert vs coin

Fair coin: $0.5 \times 1 + 0.5 \times 1 = 1$ bit.
Desert, 99% sun: $0.99 \times 0.0145 + 0.01 \times 6.64 \approx 0.08$ bits.

A rainy day in the desert is very surprising (6.6 bits), but it happens 1% of the time, so the average is tiny.

example · two next-token distributions

"The capital of France is": Paris 90%, two others 5% each. Entropy $= 0.9 \times 0.15 + 2 \times 0.05 \times 4.32 \approx 0.57$ bits.
"My favourite food is": ten foods, 10% each. Entropy $= \log_2 10 \approx 3.32$ bits.

A model sampling from the second distribution produces varied text; from the first, nearly always the same token.

Two facts worth remembering

Entropy is 0 when one outcome is certain.
For $K$ outcomes it is largest when all are equally likely, at $\log_2 K$ bits. Four outcomes: 2 bits. 100,000 tokens: about 16.6 bits.

ML code usually uses natural logs, which gives entropy in nats instead of bits. 1 bit = 0.693 nats; the idea is identical.

Where this shows up

RL tends to drive a model's entropy toward zero, which stops it exploring: entropy collapse.
Temperature raises or lowers entropy at sampling time: softmax.
The next two pages build directly on it: cross-entropy, KL divergence.

going deeper

Definition

H(p) = -\sum_x p(x) \log p(x) \qquad 0 \le H(p) \le \log K