Likelihood and maximum likelihood

Turn probability around. Instead of asking how likely the data is under a fixed model, ask which model makes the data most likely. Pre-training a language model is exactly this.

Key idea: Likelihood, log-likelihood, maximum likelihood estimation, negative log-likelihood loss, perplexity
Real world: A coin lands heads 7 times in 10. Of all possible biases, a 70% bias makes that result most likely. That’s the maximum-likelihood estimate.
In an LLM: Pre-training adjusts billions of weights to make the actual next tokens in trillions of words as likely as possible. The loss you see in every training log is minus the average log-likelihood per token.

Same formula, different question

So far a model was fixed and we asked how probable an outcome is. Likelihood flips this: the data is fixed (you’ve seen it) and you ask how well each possible model explains it.

For a coin with heads-probability $p$ , seeing 7 heads in 10 flips has probability

L(p) = p^7 (1 - p)^3

Read as a function of the data, this is a probability. Read as a function of $p$ , with the data held fixed, it is the likelihood of $p$ . Same formula, different question.

Maximum likelihood

The maximum-likelihood estimate (MLE) is the parameter value that makes the observed data most probable. For the coin, the peak of $p^7(1-p)^3$ is at $p = 0.7$ : the share of heads you saw. For a categorical distribution, the MLE of each probability is its observed frequency.

Try it

n10k7

guess p0.50

likelihood of each guess p, divided by its peak value

best guess k/n70%

log-likelihood of your guess-6.93

avg loss per sentence0.693

lowest possible loss0.611

The peak is at 70%, the share of times “ you” actually followed. Move your guess there and the average loss (minus the log-likelihood per sentence) falls from 0.693 to its minimum, 0.611. More data makes the peak narrower: the data rules out more guesses.

Data: the word “Thank” was followed by “ you” in k of n sentences. Which probability p for “ you” makes this data most likely?

Logs, again

Likelihoods of many data points are products of many probabilities, which get tiny quickly. Maximising the log-likelihood gives the same answer (log only ever goes up, so it keeps the peak in the same place) and turns the product into a sum:

\log L(p) = 7 \log p + 3 \log(1 - p)

Training frameworks minimise rather than maximise, so they use the negative log-likelihood (NLL). Averaged per data point, the NLL is exactly the cross-entropy between the data and the model. That’s why the standard language-model loss is called both.

example · estimating a coin

Flip a coin 10 times and see 7 heads. The MLE is 0.7. Notice the likelihood curve in the widget is wide: 0.5 is still quite plausible (its likelihood is 44% of the peak). Flip 100 times and see 70 heads, and the curve is much narrower; 0.5 is now very unlikely. The MLE alone doesn’t tell you how sure to be.

example · what pre-training does

In a corpus, “Thank” is followed by “ you” in 7 of every 10 occurrences. A model that only sees this context minimises its loss by predicting “ you” with probability exactly 0.7. With real contexts it is the same principle, applied to every token in the training data at once:

\text{loss} = -\frac{1}{N} \sum_{t=1}^{N} \log \pi_\theta(x_t \mid x_{<t})

Gradient descent moves the weights $\theta$ towards the maximum-likelihood fit. Supervised fine-tuning (SFT) is the same loss on a smaller, curated set of example answers.

Perplexity

Perplexity is the exponential of the average NLL per token:

\text{perplexity} = \exp\Big(-\frac1N \sum_t \log \pi_\theta(x_t \mid x_{<t})\Big)

It has a friendly reading: a perplexity of 10 means the model is, on average, as uncertain as if it were choosing uniformly among 10 tokens. A perfect model would have perplexity 1. Perplexity is only comparable between models that share a tokenizer, because a different tokenizer splits the same text into a different number of tokens.

Likelihood is not probability of the model

$L(0.7) = 0.0022$ is the probability of the data if $p = 0.7$ . It is not the probability that $p = 0.7$ . The likelihood curve doesn’t add up to 1 over $p$ , and it ignores anything you knew beforehand (most coins are fair!). Turning likelihood into a belief about $p$ needs a prior and Bayes’ theorem: the next page.

Likelihoods everywhere

Once you see it, most training objectives are a likelihood of something:

Pre-training and SFT: likelihood of the next token.
Reward models: likelihood of the human preference labels under the Bradley–Terry model, $P(A \text{ preferred}) = \sigma(r_A - r_B)$ .
DPO: the same preference likelihood, with the reward written in terms of the policy: see DPO.
Arena leaderboards: Elo-style ratings are a maximum-likelihood fit of Bradley–Terry to battle outcomes: see comparing models.

Where this shows up

The training loss: cross-entropy.
Distillation minimises a cross-entropy against a teacher instead of the data: distillation.
Policy gradient is “likelihood, weighted by reward”: the score-function trick.

going deeper

Definitions

For i.i.d. data $x_1, \dots, x_n$ and a model $p_\theta$ :

L(\theta) = \prod_{i=1}^{n} p_\theta(x_i) \qquad \ell(\theta) = \log L(\theta) = \sum_{i=1}^{n} \log p_\theta(x_i) \qquad \hat\theta_{\text{MLE}} = \arg\max_\theta \ell(\theta)

$\arg\max$ means “the value of $\theta$ at which the maximum is reached”, not the maximum itself.

For the Bernoulli case with $k$ successes in $n$ trials, setting the derivative of $k\log p + (n-k)\log(1-p)$ to zero gives $\hat p = k/n$ .

MLE minimises KL to the data

Minimising the average NLL is the same as minimising the KL divergence from the data distribution $p_{\text{data}}$ to the model:

\mathbb{E}_{x \sim p_{\text{data}}}\big[-\log p_\theta(x)\big] = H(p_{\text{data}}) + \mathrm{KL}(p_{\text{data}} \,\|\, p_\theta)

The entropy of the data is fixed, so lowering the loss can only lower the KL. This is forward KL, which punishes the model for missing anything that occurs in the data; that’s why pre-trained models spread probability over every plausible continuation. The contrast with reverse KL is on divergences.

Sequence-level form

For a language model on a sequence $x_{1:T}$ , by the chain rule,

\mathcal{L}_{\text{NLL}}(\theta) = -\sum_{t=1}^{T} \log \pi_\theta(x_t \mid x_{<t}) \qquad \mathrm{PPL} = \exp\Big(\frac{1}{T}\mathcal{L}_{\text{NLL}}\Big)

Papers also report bits per byte or bits per character, which divide the NLL (in bits, using $\log_2$ ) by the number of bytes rather than tokens. That makes models with different tokenizers comparable.

In papers

The SFT loss is written $\mathcal{L}_{\text{SFT}} = -\mathbb{E}_{(x, y) \sim \mathcal{D}} \log \pi_\theta(y \mid x)$ , with the prompt $x$ excluded from the loss. The reward-model loss is $-\mathbb{E}_{(x, y_w, y_l)} \log \sigma\big(r(x, y_w) - r(x, y_l)\big)$ , where $y_w$ is the preferred answer and $y_l$ the rejected one. Both are negative log-likelihoods.