Comparing models

Model B scores 2 points higher than model A. Is it actually better? Paired differences, clustered questions, the bootstrap, arena ratings, and the traps in reading leaderboards.

Key idea: Paired comparisons, standard errors of a difference, clustering, the bootstrap, Bradley–Terry ratings, and multiple comparisons
Real world: Two runners are timed on different days, in different weather: hard to tell who's faster. Race them side by side on the same day and course, and a small gap becomes clear. Comparing on the same conditions cancels everything they share.
In an LLM: A new checkpoint scores 62.4% on a 500-question benchmark against 60.1% for the old one. With unpaired error bars the difference looks like noise; computed per question, it may be clearly real, or not. Papers increasingly report this properly.

A benchmark score is an estimate

A model's accuracy on a benchmark is a Monte Carlo estimate of its accuracy on the whole population of similar questions. With $n$ questions and accuracy $p$ , its standard error is $\sqrt{p(1-p)/n}$ (error bars). On 500 questions at 60%, that's about 2.2 points, so the 95% interval is roughly ±4.3 points.

Comparing two models, the question is whether their difference is larger than its own noise.

Pairing

Both models answer the same questions. Questions vary hugely in difficulty, so the two models' results are strongly correlated: easy questions are right for both, hard ones wrong for both. Those questions say nothing about which model is better. Only the questions where they disagree carry information.

The paired standard error uses per-question differences, and is often much smaller than treating the two scores as independent:

example · tasting two coffees

To tell whether a new roast is better, have the same 20 people taste both and say which they prefer. People's general taste in coffee cancels out. Asking 20 people about the new roast and 20 different people about the old one needs far more people to see the same difference.

example · a two-point gain

On 1,000 questions, model A gets 60.0% and model B 62.0%. They agree on 880 questions. Of the 120 where they disagree, B wins 70 and A wins 50. Unpaired, the standard error of the difference is about 2.2 points, and the 2-point gain looks like noise. Paired, it's about 1.1 points, and the gain is nearly twice its standard error: close to significant. Same data, opposite conclusions.

Try it

questions500A’s accuracy0.60true gap B − A3.0 ptshow alike their mistakes are0.60

This run, question by question (500 questions)

B rightB wrong

A right247both right: no information30only A right

A wrong61only B right162both wrong: no information

B − A = (B-only − A-only) ÷ questions = (61 − 30) ÷ 500. Only the 91 questions where they disagree move the difference, and only they add noise to it.

measured B − A+6.2 pts

unpaired ±6.1 pts

paired ±3.7 pts

runs that detect B is better: unpaired / paired5% / 33%

Questions differ in difficulty, so two models tend to get the same ones right and wrong. The unpaired interval ignores that and treats the scores as independent; the paired interval works with the per-question differences, and only questions where the models disagree add noise. The more alike their mistakes, the narrower the paired interval, and the more often a real 3.0-point gap is detected. The dashed green line is the true gap; grey is zero.

Model B is truly better than model A by the gap you set. Both answer the same questions. The bars are 95% confidence intervals for B − A from one run of the benchmark.

Other traps

Clustered questions: many benchmarks contain several questions about the same passage or the same program. They're not independent, so the effective number of questions is smaller and naive error bars are too narrow. Use clustered standard errors.
Sampling noise: with temperature sampling, a model's score varies between runs even on fixed questions. Average several samples per question, or report it.
Small benchmarks: with a few hundred questions or fewer, especially near 0% or 100% accuracy, the normal approximation behind "±1.96 SE" is poor. Use exact or Bayesian intervals (a Beta posterior) or the bootstrap.
Many comparisons: test 20 checkpoints against a baseline at the 5% level and you expect one false "win" even if none is better. Correct for multiple comparisons, or pre-register which comparison matters.
Training noise: two RL runs with different random seeds can differ by several points. A single run per method can't separate method from luck.

The bootstrap

When there's no neat formula, resample. Draw $n$ questions with replacement from the benchmark, recompute the metric, and repeat 10,000 times. The spread of the recomputed metrics is the uncertainty. It works for any statistic, such as pass@k, a ranking, or the difference between two models (resample questions, keep the pairing).

Arena ratings

Chatbot Arena compares models by human votes on pairs of answers. Ratings come from the Bradley–Terry model (reward models): each model has a strength $s$ , and model $i$ beats $j$ with probability $\sigma(s_i - s_j)$ . Fitting the strengths is maximum likelihood on the votes, and the bootstrap gives confidence intervals. Two models whose intervals overlap heavily are not reliably ranked, whatever the leaderboard order says.

example · chess ratings

Elo ratings work the same way: a 200-point rating gap means the stronger player wins about 76% of games. A new player's rating is uncertain until they've played enough games, just as a newly added model's arena score is.

example · a 10-point arena gap

Arena scores use the same scale as Elo. A 10-point gap means the higher-rated model wins about 51.4% of head-to-head votes. With a few thousand votes per model, the confidence interval is often ±5 points or more, so a 10-point gap between neighbours is weak evidence.

Where this shows up

Every results table: a paper that reports a one-point gain without error bars is asking you to trust that it isn't noise.
pass@k curves need unbiased estimators and error bars too: Does RL teach new skills?.
Benchmark design: signal-to-noise ratio decides which benchmarks are useful for small-scale experiments.
Bradley–Terry also underlies reward models and DPO: reward models, DPO.

going deeper

Unpaired and paired standard errors

For accuracies $\hat p_A, \hat p_B$ on the same $n$ questions, and per-question differences $d_i = y_{B,i} - y_{A,i}$ :

\text{SE}_\text{unpaired} = \sqrt{\frac{\hat p_A(1 - \hat p_A) + \hat p_B(1 - \hat p_B)}{n}}, \qquad \text{SE}_\text{paired} = \sqrt{\frac{\operatorname{Var}(d)}{n}}

\operatorname{Var}(d) = \operatorname{Var}(y_A) + \operatorname{Var}(y_B) - 2 \operatorname{Cov}(y_A, y_B)

The covariance term is what pairing gains. For binary outcomes, a significance test on the discordant pairs is McNemar's test.

Clustered standard errors

With questions in clusters $c$ (e.g. per passage) and per-question residuals $e_i = d_i - \bar d$ :

\text{SE}_\text{clustered} = \frac{1}{n} \sqrt{\sum_c \Big(\sum_{i \in c} e_i\Big)^2}

Bradley–Terry

P(i \succ j) = \sigma(s_i - s_j) = \frac{1}{1 + e^{-(s_i - s_j)}}, \qquad \text{Elo scale: } P(i \succ j) = \frac{1}{1 + 10^{-(R_i - R_j)/400}}

Papers

Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (Miller, 2024): paired and clustered standard errors
Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints (Bowyer et al., 2025): Bayesian and exact alternatives
Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (Heineman et al., 2025)
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al., 2024)
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)