Divergences beyond KL

KL is one of many ways to measure how different two distributions are. Forward or reverse KL, total variation, Jensen–Shannon, χ²: which one a method minimises decides how its model behaves.

Key idea: Forward and reverse KL, total variation, Jensen–Shannon, Hellinger, χ², the f-divergence family, and Pinsker's inequality
Real world: Two maps of the same city can be compared in many ways: the largest single disagreement, the average disagreement, or how lost you'd get using one when the other is correct. Each answers a different question, and they rank maps differently.
In an LLM: A small student model learns from a large teacher. Training on the teacher's samples minimises forward KL and makes the student spread its bets. Training on the student's own samples, graded by the teacher, minimises reverse KL and makes it commit. Same teacher, different students.

Why more than one

KL divergence measures the extra surprise from predicting with $Q$ when reality follows $P$ . It's not symmetric, and it can be infinite. Other measures fix those properties, and some show up directly in LLM methods:

Divergence	Range	Symmetric?	Where you meet it
Forward KL $\mathrm{KL}(P \Vert Q)$	0 to ∞	no	Maximum likelihood, SFT, cross-entropy training
Reverse KL $\mathrm{KL}(Q \Vert P)$	0 to ∞	no	RLHF's KL penalty, on-policy distillation, variational inference
Total variation (TV)	0 to 1	yes	Speculative decoding acceptance rate, theory bounds
Jensen–Shannon (JS)	0 to log 2	yes	GANs; generalised JSD in GKD
Hellinger	0 to 1	yes	Statistical theory; robust to tiny probabilities
χ²	0 to ∞	no	Variance of importance weights

Forward versus reverse KL

The two directions of KL weight mistakes differently (KL divergence has the interactive version):

Forward, $\mathrm{KL}(P \| Q)$ , averages over what $P$ does. If $P$ does something and $Q$ gives it almost no probability, the penalty is huge. Minimising it makes $Q$ cover everything $P$ does, even at the cost of putting probability on things $P$ never does. Called mass-covering or mean-seeking.
Reverse, $\mathrm{KL}(Q \| P)$ , averages over what $Q$ does. $Q$ is only penalised for going where $P$ doesn't. Minimising it makes $Q$ pick one thing $P$ does and do it well, ignoring the rest. Called mode-seeking.

example · packing for a trip

You don't know whether it'll be hot or cold. A forward-KL packer brings clothes for both and some in-between layers nobody needs. A reverse-KL packer bets on one climate and packs perfectly for it, and has nothing if they're wrong.

example · two ways to distil

Supervised distillation trains the student on text the teacher wrote: cross-entropy on teacher samples, which is forward KL. Students spread out, and can produce odd mixtures of the teacher's styles. On-policy distillation (MiniLLM, GKD) lets the student write and the teacher grade each token, which is reverse KL. Students commit to the teacher's main modes and stay on their own distribution, which helps on long generations.

Bounded divergences

KL can be infinite: if $Q$ gives zero probability to something $P$ does, $\mathrm{KL}(P \| Q) = \infty$ . That makes it brittle when probabilities are tiny. Bounded alternatives:

Total variation: half the summed absolute difference in probabilities. The largest amount by which $P$ and $Q$ can disagree on the probability of any event. In speculative decoding, the draft acceptance rate is exactly $1 - \mathrm{TV}$ .
Jensen–Shannon: the average KL from each to their 50/50 mixture. Always finite, symmetric, and at most $\log 2$ .

Pinsker's inequality connects them: $\mathrm{TV} \le \sqrt{\mathrm{KL}/2}$ . A small KL guarantees a small TV. That's how papers turn "the policy stayed within KL budget ε" into "no event's probability changed by more than $\sqrt{\varepsilon/2}$ ."

Try it

how different the student is0.40student’s P(Lyon)0.080

teacher Pstudent Q

Five ways to measure the gap (bar length: value ÷ the measure’s maximum)

forward KL(P‖Q) unbounded; bar drawn up to 30.079

reverse KL(Q‖P) unbounded; bar drawn up to 30.079

Jensen–Shannon at most log 2 = 0.690.020

total variation at most 10.185

squared Hellinger at most 10.020

Pinsker’s inequality ties them together: total variation ≤ √(KL/2), here 0.185 ≤ 0.199. Now push the student’s P(Lyon) towards 0 and watch the chart: the dot is where you are, and the left end is P(Lyon) = 1e-8.

A teacher P and a student Q over the next word after “The capital of France is”. Move the student away, or make it nearly rule out “Lyon”.

example · a weather forecast that rules things out

A forecaster who says "0% chance of snow" and is then wrong once has, by log-loss, an infinitely bad record, a forward-KL penalty. Judged by how far their percentages were off on average (a TV-like measure), one miss costs a bounded amount. The scoring rule decides whether forecasters can ever afford to say "never".

example · why models avoid exact zeros

Language models never assign a token exactly zero probability: softmax can't. That's partly why forward-KL training is stable at all. But truncated sampling (top-p, top-k) does create exact zeros, so a trainer that computes KL to a truncated sampler's distribution can hit infinities. Clipping log-ratios or masking such tokens is the practical fix.

Where this shows up

Distillation: forward KL (SFT on teacher data), reverse KL (on-policy), and mixtures: distillation.
RLHF: the KL penalty is reverse KL from the reference, which is why the optimum is a tilted distribution.
Speculative decoding speed is $1 - \mathrm{TV}$ : rejection sampling.
Importance-sampling variance is a χ² divergence: importance sampling in depth.

going deeper

f-divergences

For a convex $f$ with $f(1) = 0$ :

D_f(P \,\|\, Q) = \sum_x Q(x)\, f\Big(\frac{P(x)}{Q(x)}\Big)

$f(t)$	Divergence
$t \log t$	$\mathrm{KL}(P \Vert Q)$
$-\log t$	$\mathrm{KL}(Q \Vert P)$
$\tfrac12 \lvert t - 1 \rvert$	total variation
$(t - 1)^2$	$\chi^2(P \Vert Q)$
$(\sqrt t - 1)^2$	$2 \times$ squared Hellinger

The widget's Hellinger row uses $H^2 = 1 - \sum_x \sqrt{P(x) Q(x)}$ .

Jensen–Shannon and GKD

\mathrm{JS}(P, Q) = \tfrac12 \mathrm{KL}(P \,\|\, M) + \tfrac12 \mathrm{KL}(Q \,\|\, M), \qquad M = \tfrac12 (P + Q)

GKD uses a generalised JSD with mixture $M = \beta P + (1 - \beta) Q$ , which interpolates between forward KL ( $\beta \to 0$ ) and reverse KL ( $\beta \to 1$ ), up to scaling.

Pinsker

\mathrm{TV}(P, Q) = \tfrac12 \sum_x \lvert P(x) - Q(x) \rvert \ \le\ \sqrt{\tfrac12 \mathrm{KL}(P \,\|\, Q)}

and the same holds with the KL reversed.

Distillation objectives

With teacher $p_T$ and student $q_\theta$ :

\text{SFT on teacher samples: } \min_\theta\ \mathbb{E}_{y \sim p_T}\big[-\log q_\theta(y)\big] \ \Leftrightarrow\ \min_\theta \mathrm{KL}(p_T \,\|\, q_\theta) \qquad \text{on-policy: } \min_\theta \mathrm{KL}(q_\theta \,\|\, p_T)

Papers

f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization (Nowozin et al., 2016): the f-divergence family in ML
MiniLLM: On-Policy Distillation of Large Language Models (Gu et al., 2023): reverse KL for distillation
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (Agarwal et al., 2023): GKD, generalised JSD
Fast Inference from Transformers via Speculative Decoding (Leviathan et al., 2022): acceptance rate and total variation