Expected value and sampling

The average outcome you'd get if you could repeat something forever, and how to estimate it from a handful of samples.

Key idea: Expected value, estimating it by sampling, and why estimates are noisy
Real world: A raffle ticket wins £10 with probability 0.2 and nothing otherwise. On average a ticket is worth £2, even though no single ticket ever pays £2.
In an LLM: A model answers a maths question correctly half the time. Its expected reward on that question is 0.5, and RL tries to raise it. We never compute it exactly; we sample answers and average.

Expected value

The expected value is a weighted average: each outcome's value, multiplied by its probability, all added up.

example · a game at a fair

Pay £1 to play. You win £10 with probability 0.2, otherwise nothing.

\mathbb{E}[\text{winnings}] = 0.2 \times £10 + 0.8 \times £0 = £2

Minus the £1 to play, you gain £1 per game on average. Play once and you either gain £9 or lose £1. Play 1,000 times and you'll be close to £1,000 up.

example · a model's expected reward

Answer type	Reward	Probability
Wrong	0	0.3
Right, wrong format	0.2	0.2
Right	1	0.5

\mathbb{E}[\text{reward}] = 0.3 \times 0 + 0.2 \times 0.2 + 0.5 \times 1 = 0.54

RL's whole goal is to change the model so this number goes up.

Estimating it by sampling

For a real model we don't know the probabilities: there are far too many possible answers. But we can sample: generate answers, score them, and average. The average of many samples gets close to the expected value. This is the law of large numbers.

Try it

samples0

last reward–

running average–

expected value0.54

typical error–

running averageearlier runsexpected value± typical error

Press Sample 1 a few times, then Sample 50.

Each sample is one answer from the model, scored 0, 0.2 or 1. The dashed line is the expected value; the blue funnel is how far an average of that many samples typically lands from it.

How noisy is an estimate?

The error of a sample average shrinks with the square root of the number of samples. Four times as many samples halves the error. In the example above, with 8 samples the estimate is typically off by about 0.16; with 100 samples, by about 0.05. That is the dotted funnel in the widget: press New run a few times and watch every run narrow at that rate.

This is why RL needs tricks to reduce noise, like subtracting a baseline, and why it samples a group of answers per prompt in GRPO.

Where this shows up

The goal of RL is to maximise expected return: the RL loop.
A value is an expected return from a state: value.
The baseline and advantage are built from expected rewards: policy gradient.
Importance sampling estimates one expected value with samples from somewhere else: importance sampling.

going deeper

Definitions

\mathbb{E}[X] = \sum_x p(x)\, x \qquad \operatorname{Var}[X] = \mathbb{E}\big[(X - \mathbb{E}[X])^2\big]