Joint, marginal and conditional

Two random things at once. The joint distribution covers both, a marginal forgets one, and a conditional zooms in on the cases where one is known.

Key idea: Joint distributions, marginalising (summing out), conditioning (zoom in and renormalise)
Real world: Of 1,000 days, 120 had a cloudy morning and a rainy afternoon. Knowing the morning is cloudy, the chance of rain is not 155/1,000 but 120/300: you only look at cloudy mornings.
In an LLM: A model’s accuracy is 67% overall, but 60% on maths, 50% on code and 85% on chat. Each of those is a conditional probability: accuracy given the task.

Two things at once

Usually there is more than one random thing going on: the weather in the morning and in the afternoon; the kind of prompt and whether the answer was right. The joint distribution gives the probability of every combination.

A joint distribution of two variables fits in a table. Each cell is “this and that”, and all the cells add up to 1.

Try it

grade →task ↓			total
	0.18 180	0.12 120	0.30
	0.15 150	0.15 150	0.30
	0.34 340	0.06 60	0.40
total	0.67	0.33	1.00

All 6 cells add up to 1. A row total, like P(maths) = 0.30, is a marginal: add across, forgetting the other variable. Click “maths” to see P(grade | maths).

Each cell is how often two things happen together, out of 1,000 cases: the joint distribution. The edges are the marginals. Click a heading to condition on it.

Marginals: forget one variable

To get the distribution of just one variable, add up across the other. This is called marginalising, because people used to write those totals in the margins of the table.

P(\text{maths}) = P(\text{maths and correct}) + P(\text{maths and wrong}) = 0.18 + 0.12 = 0.30

Marginalising is the most common operation in probabilistic machine learning: whenever there is something you can’t see or don’t care about (which reasoning path the model took, which hidden “concept” a prompt is about), you add it up over all its possibilities.

Conditionals: zoom in and renormalise

A conditional probability $P(A \mid B)$ , read “A given B”, is the probability of $A$ among the cases where B happened. Two steps:

Zoom in: throw away every case where $B$ didn’t happen.
Renormalise: divide by what’s left, so the remaining probabilities add up to 1 again.

P(A \mid B) = \frac{P(A \text{ and } B)}{P(B)}

example · clouds and rain

Of 1,000 days, 300 had cloudy mornings and on 120 of those it rained in the afternoon. On the 700 clear mornings, it rained 35 times.

$P(\text{rain}) = 155/1000 = 15.5\%$ — the marginal.
$P(\text{rain} \mid \text{cloudy}) = 120/300 = 40\%$ .
$P(\text{rain} \mid \text{clear}) = 35/700 = 5\%$ .

The forecast for “rain given cloudy” is very different from “rain” on its own, and also very different from $P(\text{cloudy} \mid \text{rain}) = 120/155 = 77\%$ . The order of a conditional matters.

example · accuracy by task

A model was graded on 1,000 prompts: 300 maths, 300 code and 400 chat. It got 180, 150 and 340 of them right.

Overall accuracy, the marginal: $670/1000 = 67\%$ .
$P(\text{correct} \mid \text{maths}) = 180/300 = 60\%$ .
$P(\text{maths} \mid \text{correct}) = 180/670 = 27\%$ : of the right answers, about a quarter were maths.

The overall number depends on the mix of prompts. Change the benchmark to be half maths and the same model scores lower, without changing at all. This is why papers report per-category results, and why comparing scores across benchmarks with different mixes is risky.

Weighted averages of conditionals

You can rebuild a marginal from conditionals: weight each conditional by how common its condition is. This is the law of total probability:

P(\text{correct}) = \underbrace{0.3 \times 0.6}_{\text{maths}} + \underbrace{0.3 \times 0.5}_{\text{code}} + \underbrace{0.4 \times 0.85}_{\text{chat}} = 0.67

model A: % hard80%model B: % hard20%

model Aoverall accuracy 50.0%

easy 90%

hard 40%

model Boverall accuracy 74.0%

easy 85%

hard 30%

easy prompts: A vs B90% vs 85%

hard prompts: A vs B40% vs 30%

overall: A vs B50.0% vs 74.0%

Overall accuracy for A = 20% × 90% + 80% × 40% = 50.0%: the law of total probability, a weighted average of the groups. The dashed line is that average. A is better in both groups, yet B wins overall, because A got far more hard cases. That is Simpson’s paradox: the totals compare the mixes, not the models.

Each wide bar is everything one model (or hospital) was tested on, split by group: the width of a part is how common that group is, the filled height is the success rate inside it. The overall rate is the total filled area.

example · two hospitals

Hospital A saves 95% of mild cases and 70% of severe ones; hospital B saves 90% and 60%. A is better at both. But A is the regional trauma centre and 80% of its patients are severe, while 80% of B’s are mild. Overall, A saves $0.2 \times 0.95 + 0.8 \times 0.70 = 75\%$ and B saves $0.8 \times 0.90 + 0.2 \times 0.60 = 84\%$ . The better hospital has the worse total.

example · two models on different prompt mixes

Model A scores 90% on easy prompts and 40% on hard ones; model B scores 85% and 30%. If A was evaluated on a set that is 80% hard and B on one that is 20% hard, A’s headline number is 50% and B’s is 74%. Leaderboards that mix results from different test sets, or RL papers that change the prompt filter between runs, can produce exactly this.

Everything in a language model is conditional

A language model never outputs $P(\text{word})$ . It outputs $P(\text{next token} \mid \text{everything before it})$ : a different distribution for every context. Prompting, few-shot examples and system prompts all work by changing what is being conditioned on. The chain rule shows how these conditionals combine into the probability of a whole answer.

Where this shows up

A policy is a conditional distribution over actions given the state: the RL loop.
Bayes’ theorem flips the order of a conditional: Bayes’ theorem.
Summing out a hidden reasoning path is marginalising: latent variables.

going deeper

Definitions

For random variables $X$ and $Y$ with joint distribution $p(x, y)$ :

p(x) = \sum_y p(x, y) \qquad p(y \mid x) = \frac{p(x, y)}{p(x)} \qquad p(x, y) = p(x)\, p(y \mid x)

The law of total probability, and its version for expectations (the tower rule):

p(y) = \sum_x p(x)\, p(y \mid x) \qquad \mathbb{E}[Y] = \mathbb{E}_X\big[\mathbb{E}[Y \mid X]\big]

In papers

The objective of nearly every LLM fine-tuning paper is a nested expectation of this kind: average over prompts from a dataset $\mathcal{D}$ , and for each prompt, average over answers from the policy:

J(\theta) = \mathbb{E}_{x \sim \mathcal{D}}\, \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}\big[r(x, y)\big]

Read it inside out: for one prompt $x$ , the inner expectation is the model’s expected reward on that prompt; the outer one averages over prompts. Some papers write the pair as $\mathbb{E}_{(x, y) \sim \mathcal{D}, \pi_\theta}$ or just $\mathbb{E}_{x, y}$ . A semicolon, as in $p(y; \theta)$ , marks parameters rather than conditioning on a random variable: same idea, different status.