The chain rule and autoregressive models

The probability of a sequence is a product of conditionals, one per step. Every GPT-style model is built on this one identity, and its log turns the product into a sum.

Key idea: The chain rule of probability, autoregressive factorisation, sequence log-probabilities, length effects
Real world: The chance of three rainy days in a row is the chance of Monday’s rain, times the chance of Tuesday’s rain given Monday’s, times Wednesday’s given both: $0.3 \times 0.6 \times 0.7 = 0.126$ .
In an LLM: The probability a model writes “Paris.” after “The capital of France is” is $P(\text{Paris} \mid \text{prompt}) \times P(\text{.} \mid \text{prompt, Paris})$ . Every answer’s probability is a long product like this, one factor per token.

From pairs to sequences

The previous page gave the rule for two events: $P(A \text{ and } B) = P(A) \times P(B \mid A)$ . Apply it again and again, and you get the probability of any sequence:

P(x_1, x_2, x_3) = P(x_1) \times P(x_2 \mid x_1) \times P(x_3 \mid x_1, x_2)

This is the chain rule. It is always true, for any sequence of random variables; no assumptions are needed. Each factor conditions on everything that came before.

Try it

step 1

step 2, given Paris

step 3, given Paris.

the sentence so far

The capital of France is Paris.

probability of the text so far, after each token

prompt1.000

␣Paris ×0.820.820

. ×0.700.574

P(path)0.82 × 0.70 = 0.574

log P(path)-0.56

Pick a token in every column. The probability of the whole continuation is the product of the probability of each choice, given the ones before it.

Prompt: “The capital of France is”. Each column is the model’s next-token distribution given the tokens chosen to its left. Click to choose a different token.

This is what a language model is

A GPT-style model is a machine for computing one factor of the chain rule at a time: given the tokens so far, it outputs a distribution over the next token. That’s why these models are called autoregressive: each output becomes part of the input for the next step.

Generating text means sampling one factor at a time: draw a token, append it, ask again.
Scoring text means running the whole sequence through once and multiplying the probabilities the model gave each actual token.
Training means making the product large for real text: see likelihood.

example · three rainy days

$P(\text{rain Mon}) = 0.3$ , $P(\text{rain Tue} \mid \text{rain Mon}) = 0.6$ , $P(\text{rain Wed} \mid \text{rain Mon, Tue}) = 0.7$ . The chance of all three is $0.3 \times 0.6 \times 0.7 = 0.126$ . If the days were unrelated, it would be $0.3^3 = 0.027$ ; rain tends to follow rain, and the conditionals capture that.

example · scoring an answer

A model answers “Paris.” to “The capital of France is”. It gave “ Paris” 0.82 and then “.” 0.70:

P(\text{“ Paris.”} \mid \text{prompt}) = 0.82 \times 0.70 = 0.574

In RL, this number (or rather its log) is what gets pushed up or down for each sampled answer. The policy gradient is computed from exactly these per-token factors.

Work in logs

Products of many numbers below 1 shrink fast. A 1,000-token answer where each token had probability 0.9 has probability $0.9^{1000} \approx 10^{-46}$ , too small for many number formats. Taking logarithms turns the product into a sum:

\log P(x_1, \dots, x_T) = \sum_{t=1}^{T} \log P(x_t \mid x_1, \dots, x_{t-1})

Every paper and every training framework works with these sums of token log-probabilities (logprobs). The logarithms lesson covers why.

Longer answers are less likely

Because each factor is at most 1, a sequence’s probability can only fall as it gets longer. This has real consequences:

Comparing answers of different lengths by raw probability unfairly favours short ones. Beam search and best-of-n rankings often divide the log-probability by the length to compensate.
Sequence-level ratios explode. The ratio of two models’ probabilities for a 1,000-token answer is a product of 1,000 per-token ratios; small per-token differences multiply into huge numbers. This is why PPO and GRPO use per-token ratios, and why GSPO uses the length-normalised version: see importance weights.
Averages per token (log-probability divided by length) are how perplexity is defined, on the likelihood page.

Where this shows up

Policy-gradient methods add up per-token log-probabilities: policy gradient.
Every sampling strategy changes one factor at a time: sampling text.
Diffusion language models replace this left-to-right factorisation: diffusion language models.

going deeper

The identity

For any random variables $x_1, \dots, x_T$ :

p(x_1, \dots, x_T) = \prod_{t=1}^{T} p(x_t \mid x_{<t})

where $x_{<t}$ is shorthand for $x_1, \dots, x_{t-1}$ . A language model with parameters $\theta$ , given a prompt $x$ , defines

\pi_\theta(y \mid x) = \prod_{t=1}^{|y|} \pi_\theta(y_t \mid x, y_{<t}) \qquad \log \pi_\theta(y \mid x) = \sum_{t=1}^{|y|} \log \pi_\theta(y_t \mid x, y_{<t})

$|y|$ is the number of tokens in $y$ . Generation ends when the model samples a special end-of-sequence token, so “stop here” is itself one of the factors.

Why the gradient splits by token

Because the log of the product is a sum, its gradient is a sum of per-token gradients:

\nabla_\theta \log \pi_\theta(y \mid x) = \sum_t \nabla_\theta \log \pi_\theta(y_t \mid x, y_{<t})

This is why RL losses for LLMs are written as sums (or averages) over tokens, and why papers argue about how to average them: per sequence, per token, or per batch. See loss aggregation.

Teacher forcing

During training, every factor conditions on the real previous tokens, not on what the model would have generated. All $T$ factors can then be computed in one parallel pass. At generation time the model conditions on its own samples instead; the gap between the two is called exposure bias, and on-policy methods (RL, on-policy distillation) train on the model’s own samples partly to close it.