Averages, the central limit theorem and error bars

An average of many random things is less random than any one of them, and close to bell-shaped. That is what lets a benchmark score come with an honest “± something”.

Key idea: Law of large numbers, central limit theorem, standard error, confidence intervals, clustered questions
Real world: A poll of 1,000 voters says 52% support a policy, “± 3 points”. Ask a different 1,000 people and you’d get a slightly different number; the ± says how different.
In an LLM: A model scores 70% on a 200-question benchmark. The ± is about 6 points. A rival scoring 73% might not be better at all.

The law of large numbers

The average of many independent samples gets closer and closer to the expected value. You saw this in expected value and sampling: the running average wobbles a lot at first and then settles. That is the law of large numbers. It says the average converges, but not how fast or how much it wobbles on the way.

How much an average wobbles

The standard deviation of an average of $n$ independent samples is called the standard error (SE):

\text{SE} = \frac{\sigma}{\sqrt{n}}

$\sigma$ is the standard deviation of one sample. The $\sqrt{n}$ is the key: to halve the error you need four times as much data, and to cut it by 10 you need 100 times as much.

For a benchmark where each question is right (1) or wrong (0), one question’s standard deviation is $\sqrt{p(1-p)}$ , so a score $\hat p$ on $n$ questions has

\text{SE} = \sqrt{\frac{\hat p (1 - \hat p)}{n}}

The central limit theorem

The central limit theorem (CLT) says more: whatever the distribution of a single sample (as long as its variance is finite), the average of many samples is approximately Gaussian. Individual answers are 0 or 1, nothing like a bell, but scores of a 200-question benchmark, rerun many times, form a bell curve centred on the true accuracy.

That gives the familiar recipe for a 95% confidence interval: the estimate plus or minus 1.96 standard errors.

\hat p \pm 1.96 \times \text{SE}

Try it

questions200questions come in groups of 10 (one passage each)

scores from the runsbell curve the ± formula predictslast run’s 95% interval

runs60

last score71.5%

± from the formula6.4%

± actually (1.96 × spread)6.3%

intervals containing 70%97%

The scores pile up in a bell curve around 70%, as the central limit theorem says they must, with a spread of ±6.4%. About 95% of the intervals contain the truth. Four times as many questions halves the spread: with 200 questions, a 9.0% gap between two models can be pure noise.

A model whose true accuracy is 70%. Each run asks it a fresh set of questions of the same kind and reports the share it got right.

example · an opinion poll

1,000 voters, 52% in favour. $\text{SE} = \sqrt{0.52 \times 0.48 / 1000} \approx 0.016$ , so the 95% interval is $52\% \pm 3.1\%$ : from about 49% to 55%. The headline “majority in favour” is not actually established.

example · a benchmark score

A model gets 140 of 200 questions right: 70%. $\text{SE} = \sqrt{0.7 \times 0.3 / 200} \approx 0.032$ , so the interval is $70\% \pm 6.4\%$ .

A rival scores 73% on the same benchmark. Is it better? The comparing models page does this properly, but the quick answer is: not on this evidence. A 3-point gap is well within the noise of a 200-question test.

What “95% confidence” means

A 95% confidence interval does not mean “there’s a 95% chance the true value is in this particular interval”. It means the procedure catches the true value 95% of the time: build intervals this way for many evals, and 95 in 100 will contain the truth. The widget counts this. (The Bayesian version, which does say “95% probability the value is here”, is a credible interval: see priors and posteriors.)

When the recipe breaks

The ± formula assumes independent questions and enough of them. Real evals violate both.

Clustered questions. Many benchmarks group several questions around one passage, one codebase or one document. Questions in a cluster succeed or fail together, so they carry less information than the same number of independent questions. Treating them as independent makes the error bars too small, sometimes by a factor of two or three. The fix is a clustered standard error that treats each cluster as one unit. Tick “questions come in groups of 10” in the widget above: the scores spread about 1.6 times wider than the formula says, and only around three quarters of the “95%” intervals contain the truth.

Small benchmarks. With a few dozen questions, or a score near 0% or 100%, the bell-curve approximation is poor: “ $3\% \pm 4\%$ ” would include negative accuracies. Better intervals (Wilson, or a Bayesian Beta posterior) stay inside 0–100%.

Sampling noise inside each question. At temperature above 0, a model’s answer to the same question varies between runs. Sampling several answers per question and averaging reduces this part of the noise, but not the part that comes from which questions happened to be in the benchmark.

Where this shows up

Comparing two models properly, with paired differences: comparing models.
Choosing pass rates and group sizes in RL is a variance question: GRPO.
The Monte Carlo view of every estimate from samples: Monte Carlo.

going deeper

The theorems

Law of large numbers. For i.i.d. $X_1, X_2, \dots$ with mean $\mu$ , the sample mean $\bar X_n \to \mu$ as $n \to \infty$ .

Central limit theorem. If the $X_i$ also have finite variance $\sigma^2$ :

\sqrt{n}\,\big(\bar X_n - \mu\big) \xrightarrow{d} \mathcal{N}(0, \sigma^2) \qquad \text{so approximately} \quad \bar X_n \sim \mathcal{N}\Big(\mu, \frac{\sigma^2}{n}\Big)

The arrow with a $d$ reads “converges in distribution”.

Eval-specific formulas

Following Miller, 2024, for question scores $s_i$ (0/1, or partial credit) on $n$ questions:

\text{SE}_{\text{CLT}} = \sqrt{\frac{\operatorname{Var}(s)}{n}}

With questions grouped into clusters $c$ , the clustered standard error is

\text{SE}_{\text{clustered}} = \sqrt{\text{SE}_{\text{CLT}}^2 + \frac{1}{n^2} \sum_c \sum_{i \ne j \in c} (s_i - \bar s)(s_j - \bar s)}

The extra term is the covariance within clusters; positive correlation inflates the error. If each question’s score is itself an average of $K$ sampled answers, the within-question sampling variance shrinks by $K$ , but the between-question part does not.

The Wilson score interval, better for small $n$ or extreme $\hat p$ , solves $|\hat p - p| = z\sqrt{p(1-p)/n}$ for $p$ instead of plugging in $\hat p$ .

In papers

Miller, 2024 lays out the full statistical recipe for evals: CLT standard errors, clustering, paired differences, and power analysis (how many questions you need to detect a given gap).
Bowyer et al., 2025 argue that CLT intervals are unreliable below a few hundred questions and recommend Bayesian or exact intervals.
Heineman et al., 2025 measure benchmark quality as a signal-to-noise ratio: how far apart models are, relative to how much a single model’s score wobbles between checkpoints.