KL divergence

How much extra surprise you suffer by using one distribution when reality follows another. It measures how different two distributions are, but not symmetrically.

Key idea: KL divergence, why it's not symmetric, and forward vs reverse KL
Real world: Someone from a desert city moves to London still expecting almost no rain. They're surprised far more often than a Londoner would be. KL measures that average extra surprise.
In an LLM: RL training adds a KL penalty that measures how far the model has drifted from where it started, so it can't wander off into reward hacking.

Definition in words

From the last page: cross-entropy = entropy + KL. So

\mathrm{KL}(P \,\|\, Q) = \text{cross-entropy}(P, Q) - \text{entropy}(P)

It's the extra surprise from predicting with Q when outcomes really follow P. Written out, it's the average of the log-ratio:

\mathrm{KL}(P \,\|\, Q) = \sum_x P(x) \log \frac{P(x)}{Q(x)}

Three properties:

Never negative. You can't do better than knowing the true distribution.
Zero only when P and Q are identical.
Not symmetric. $\mathrm{KL}(P \| Q)$ and $\mathrm{KL}(Q \| P)$ are usually different, so it isn't a distance in the everyday sense.

example · London and a desert city

Chance of rain on a given day: London 40%, the desert city 2%.

A desert resident in London: reality is London (P), their expectation is desert (Q).

$0.4 \ln \frac{0.4}{0.02} + 0.6 \ln \frac{0.6}{0.98} = 1.20 - 0.29 = 0.90$ nats
A Londoner in the desert: reality is desert, expectation London.

$0.02 \ln \frac{0.02}{0.4} + 0.98 \ln \frac{0.98}{0.6} = -0.06 + 0.48 = 0.42$ nats

The desert resident in London is surprised over twice as much: they think rain is nearly impossible, and it keeps happening. The Londoner just finds the desert sunnier than expected.

example · a KL penalty on one token

The model before RL (the reference) gave a token probability 0.2. After some training it gives 0.5. One sample of the KL contribution is the log-ratio $\ln \frac{0.5}{0.2} = 0.92$ . Summed over the tokens of an answer and averaged over many answers, this estimates how far the model has moved. RL subtracts β times it from the reward.

Try it

Q centre5.5Q width1.00

P, the targetQ, the fit

KL(P‖Q), forward2.44

KL(Q‖P), reverse2.36

Press the two buttons. Forward KL punishes Q for missing anywhere P goes, so it spreads out to cover both peaks. Reverse KL punishes Q for going where P doesn’t, so it commits to one peak.

P (grey) has two peaks. Q (purple) has one; move and widen it. KL(P‖Q) asks how badly Q explains P’s outcomes; KL(Q‖P) asks the reverse. Both in nats.

Forward and reverse KL

The widget shows why the direction matters when one distribution is being fitted to another:

Forward KL, $\mathrm{KL}(P \| Q)$ : averaged over P's outcomes. If P has an outcome and Q gives it nearly zero probability, the log-ratio explodes. So Q spreads out to cover everything P does, even if that means putting probability in between. This is what ordinary training with cross-entropy does.
Reverse KL, $\mathrm{KL}(Q \| P)$ : averaged over Q's outcomes. Q is only punished where it puts probability, so it can safely ignore some of P's peaks and commit to one. In the widget, the best reverse-KL fit sits on one peak and pays only $\ln 2 \approx 0.69$ for ignoring the other half of P.

RL's KL penalty and on-policy distillation both use the reverse direction, measured on the model's own samples. That's one reason RL-trained models tend to settle on one way of answering rather than reproducing all the variety of the original model.

Where this shows up

The KL penalty against a reference model: KL and reward hacking.
DPO's loss is derived from the KL-penalised objective: DPO.
On-policy distillation minimises reverse KL to a teacher: distillation.
PPO and GRPO's clipping is a cheap way to keep each update's KL small: PPO.

going deeper

Definitions

\mathrm{KL}(P \,\|\, Q) = \sum_x P(x) \log \frac{P(x)}{Q(x)} = \mathbb{E}_{x \sim P}\Big[\log \frac{P(x)}{Q(x)}\Big] \ge 0

The inequality (Gibbs' inequality) follows from $\log t \le t - 1$ . If $Q(x) = 0$ anywhere $P(x) > 0$ , forward KL is infinite.

In practice KL is estimated from samples, using one of several estimators; see estimating KL.

Definition in words

Try it

Forward and reverse KL

Where this shows up

Definitions

Related