Independence

Two things are independent when knowing one tells you nothing about the other. Samples from a model are independent given the prompt, but not across prompts, and that gap explains a lot of surprising results.

Key idea: Independence, conditional independence, i.i.d. samples, and hidden common causes
Real world: Two coin flips are independent. But whether two friends both carry umbrellas is not: both depend on the weather. Given the weather, they become independent again.
In an LLM: Two answers sampled for the same prompt are independent given the prompt. Across a benchmark they aren’t: if the first is right, the question is probably easy, so the second is likely right too. That’s why pass@k grows more slowly than the textbook formula.

What independence means

Two events are independent if knowing one happened doesn’t change the probability of the other:

P(A \mid B) = P(A)

Equivalently, the probability of both is the product of the probabilities: $P(A \text{ and } B) = P(A)\,P(B)$ . That product rule is the reason independence matters so much. It is what lets you write $0.8^{10}$ for “ten failures in a row”, or multiply probabilities across data points.

example · coins and dice

A coin and a die don’t affect each other: $P(\text{heads and a six}) = \tfrac12 \times \tfrac16 = \tfrac1{12}$ . In the dice widget, “first die is even” and “sum is 7” are independent too ( $18/36$ and $6/36$ ; together $3/36 = \tfrac12 \times \tfrac16$ ), even though both involve the first die. Independence is about numbers, not about whether things are “connected”.

example · two different prompts

Sample one answer to a geometry question the model solves 60% of the time, and one to an unrelated history question it solves 50% of the time. Each answer is drawn with its own fresh randomness, so the chance both are right is $0.6 \times 0.5 = 0.3$ . That product is what lets you predict a whole batch: on 10 such unrelated prompts at 50% each, the chance all 10 are right is $0.5^{10}$ , about 0.1%.

Independent given something else

Often two things are linked only through a shared cause. Once you know the cause, the link disappears. This is conditional independence.

example · two umbrellas

On a random day, if your friend brought an umbrella, you probably did too: you both looked at the same sky. Your umbrellas aren’t independent. But given the forecast, your choices have nothing more to do with each other. The weather is a common cause: condition on it, and the dependence goes away.

example · two answers to one prompt

Sample two answers to the same question at temperature 1. Each token is drawn with fresh randomness, so given the prompt, the two answers are independent: $P(\text{both right} \mid \text{question}) = p_q^2$ , where $p_q$ is the model’s pass rate on that question.

But across a benchmark, questions differ. Some have $p_q = 0.95$ , others $p_q = 0.02$ . If you’re told only that the first answer was right, that is evidence you’re on an easy question, and the second answer is more likely to be right too. The question’s difficulty is the common cause.

Try it

The top chart is the spread of difficulty across questions. The bottom chart is pass@k as you sample more answers: dashed if every answer were an independent coin flip at the average pass rate, solid for what you actually get. Make the questions differ more and watch the two pull apart.

average pass rate50%how much questions differ25%

How many questions have each pass rate

always wrongalways right

P(answer right)50%

P(right)² (if independent)25%

P(both right), actually31%

P(2nd right | 1st right)63%

pass@k: chance at least one of k answers is right (k doubles at each step)

if independent: 1 − (1 − 50%)^kactually, averaged over questions

pass@8 if independent99.6%

pass@8, actually92.6%

pass@64, actually99.6%

Questions differ, so a right first answer is evidence the question is easy: the second answer is then right 63% of the time, not 50%. The answers are independent given the question, but not independent overall. That is why sampling more answers helps less than the independent formula promises: hard questions stay hard.

A benchmark of many questions. Each question has its own chance that the model gets it right: easy ones near 100%, hard ones near 0%. We sample two answers to each question.

Why it matters

The textbook formula $1 - (1 - p)^k$ for “at least one of $k$ succeeds” assumes every attempt has the same success chance $p$ . Averaged over a benchmark of easy and hard questions, it overestimates: many tries on an easy question add little, and many tries on a near-impossible question still fail. The real pass@k averages the formula per question:

\text{pass@}k = \text{average over questions of } \big[1 - (1 - p_q)^k\big]

The same effect shows up in:

Majority voting helps less than “independent voters” maths predicts, because a model’s mistakes on a question are correlated: it tends to make the same mistake. See latent variables.
Error bars on evals are too small if you treat related questions (several questions about one document, or several samples per question) as independent. See error bars.
GRPO groups where all 8 answers get the same reward carry no learning signal; on very easy or very hard prompts this happens far more often than independent coin flips at the average pass rate would suggest. See GRPO.

i.i.d.

Papers constantly say samples are i.i.d.: independent and identically distributed. Each sample comes from the same distribution, and none of them affects the others. Most formulas for averages and error bars assume it. When a paper samples several answers per prompt, the answers are i.i.d. within a prompt, but the full set isn’t i.i.d. across the benchmark, which is exactly the situation in the widget.

Where this shows up

Repeated sampling and pass@k: Does RL teach new skills?.
Clustered questions and correct error bars: error bars.
Language models are explicitly not independent across tokens; the chain rule handles the dependence.

going deeper

Definitions

$X$ and $Y$ are independent, written $X \perp Y$ , when

p(x, y) = p(x)\, p(y) \quad \text{for all } x, y

They are conditionally independent given $Z$ , written $X \perp Y \mid Z$ , when

p(x, y \mid z) = p(x \mid z)\, p(y \mid z)

Neither implies the other: independent variables can become dependent once you condition on a common effect (two independent causes of a single alarm become linked once you know the alarm went off; this is “explaining away”).

The hidden-difficulty calculation

Let each question have pass rate $p_q$ , itself random across questions with mean $\mu$ and variance $\sigma^2$ . Then for two answers to the same question:

P(\text{both right}) = \mathbb{E}[p_q^2] = \mu^2 + \sigma^2 \;\ge\; \mu^2

The excess over $\mu^2$ is exactly the variance of difficulty across questions. With a Beta distribution of pass rates (as in the widget), averaging $(1 - p_q)^k$ has a closed form, and $\mathbb{E}[(1-p_q)^k] \ge (1-\mu)^k$ by Jensen’s inequality, so true pass@k is always at most the naive formula.

In papers

The Large Language Monkeys paper (Brown et al., 2024) measures “coverage”, pass@k as $k$ grows into the thousands, and finds it rises roughly log-linearly, well fitted by an “exponentiated power law”, rather than the fast exponential approach to 1 that a single shared $p$ would give. A slow, power-law-like climb is what you get by averaging $1 - (1-p_q)^k$ over a wide spread of per-question difficulties with many very hard questions.