Cross-entropy

The average surprise when reality follows one distribution but you predict with another. It is the loss every language model is trained on.

Key idea: Cross-entropy, the log loss, and perplexity
Real world: A weather forecaster is scored on every forecast. Saying 90% rain when it rains costs little; saying 10% rain when it rains costs a lot. Being confidently wrong is punished hardest.
In an LLM: Pre-training and fine-tuning both minimise cross-entropy: at every position, the loss is minus the log-probability the model gave to the token that actually came next.

Surprise under the wrong model

Entropy is your average surprise when you know the true probabilities. But you usually don't: you have a model's guess. Cross-entropy is your average surprise when outcomes come from the true distribution P but you measure surprise using your model Q:

H(P, Q) = \sum_x P(x) \times (-\log Q(x))

If Q matches P, cross-entropy equals entropy. If Q is wrong, it's higher. It can never be lower.

example · a forecaster's score

It rains. The forecaster's surprise, $-\log_2(\text{the probability they gave rain})$ :

Forecast for rain	Score (bits)
90%	0.15
50%	1.00
10%	3.32
1%	6.64

Averaged over a year of forecasts, this is their cross-entropy. The only way to score well is to give honest probabilities: hedging everything at 50% costs 1 bit every day, and overconfidence is disastrous on the days you're wrong.

The training loss of a language model

In training data, each position has one actual next token. The true distribution P is then "100% on that token", and cross-entropy simplifies to a single term:

\text{loss} = -\log Q(\text{actual next token})

example · one position

Text: "The cat sat on the mat". At the last position the model gave mat probability 0.4.

Loss: $-\ln 0.4 = 0.92$
If the model had said 0.9: $-\ln 0.9 = 0.11$
If it had said 0.01: $-\ln 0.01 = 4.61$

Training lowers the average of this number over billions of positions. That is all "next-token prediction" means.

Perplexity is $e^{\text{loss}}$ : a loss of 0.92 is a perplexity of 2.5, "as unsure as choosing evenly among 2.5 options". Lower is better.

Try it

Model weights

mat25floor25sofa25roof25

Data (grey) vs model (purple)

mat

floor

sofa

roof

04 bits

entropy of the data (can’t be removed)KL divergence (the model’s fault)

cross-entropy (loss)2.000 bits

entropy of the data1.490 bits

gap = KL divergence0.510 bits

Cross-entropy can never drop below the data’s entropy. The extra is the KL divergence: what you pay for the model being wrong. Try matching the grey bars.

“The cat sat on the ___”. Grey: how often each word really follows, in the data. Purple: your model’s prediction. Adjust the model.

Entropy + something

The widget’s stacked bar shows the key identity:

\text{cross-entropy} = \text{entropy of the data} + \text{KL divergence}

The entropy part is the data's own unpredictability: no model can remove it. The rest is what you lose by having the wrong model, and it has its own name: KL divergence, on the next page.

Where this shows up

Supervised fine-tuning and distillation minimise cross-entropy: distillation, training an LLM with RL.
Policy gradient is cross-entropy on the model's own samples, weighted by advantage: policy gradient.

going deeper

Definitions

H(P, Q) = -\sum_x P(x) \log Q(x) = H(P) + \mathrm{KL}(P \,\|\, Q)

For a dataset of tokens $y_1, \dots, y_N$ with contexts $c_t$ , the language-modelling loss is