Expectation, variance and covariance

The mean says where a random number sits; the variance says how much it wobbles; the covariance says whether two of them wobble together. GRPO’s advantage formula is these three ideas in one line.

Key idea: Linearity of expectation, variance and standard deviation, z-scores, covariance and correlation
Real world: Two investments both return 5% a year on average. One returns 4–6% every year, the other swings between −20% and +30%. Same mean, very different variance.
In an LLM: GRPO samples 8 answers to a prompt, then turns each reward into an advantage: reward minus the group’s mean, divided by the group’s standard deviation. If all 8 get the same reward, the standard deviation is 0 and the prompt teaches nothing.

Expectation, again

The expected value of a random variable is its probability-weighted average. Its most useful property is that it is linear:

\mathbb{E}[aX + bY] = a\,\mathbb{E}[X] + b\,\mathbb{E}[Y]

This holds always, even when $X$ and $Y$ are dependent. It’s why the expected total length of 100 answers is 100 times the expected length of one, and why the expected reward of a weighted mix of rewards (say, correctness plus 0.1 times a format bonus) is the same weighted mix of their expectations.

Variance: how much it wobbles

The variance is the average squared distance from the mean. The standard deviation (std, $\sigma$ ) is its square root, which brings it back to the original units.

\operatorname{Var}[X] = \mathbb{E}\big[(X - \mu)^2\big] = \mathbb{E}[X^2] - \mu^2 \qquad \sigma = \sqrt{\operatorname{Var}[X]}

example · two investments

Investment A returns 4% or 6% with equal chance: mean 5%, standard deviation 1%. Investment B returns −20% or +30% with equal chance: mean 5%, standard deviation 25%. The expectation alone can’t tell them apart.

example · a 0/1 reward

A reward that is 1 with probability $p$ and 0 otherwise (a Bernoulli variable) has mean $p$ and variance $p(1 - p)$ . That is 0 when $p = 0$ or $p = 1$ , and largest, 0.25, when $p = 0.5$ .

So prompts the model always solves or never solves have zero reward variance. In RL that’s a problem: the learning signal comes from differences between answers, and there are none. Prompts near 50% pass rate carry the most signal, which is why many recipes filter prompts by pass rate: see data and length.

Standardising: z-scores

Subtract the mean and divide by the standard deviation, and any random variable becomes one with mean 0 and standard deviation 1. The result is called a z-score: how many standard deviations above or below average a value is.

GRPO computes exactly this within each group of answers to the same prompt:

A_i = \frac{r_i - \operatorname{mean}(r_1, \dots, r_G)}{\operatorname{std}(r_1, \dots, r_G)}

Try it

divide by the standard deviation

rewards of the 8 answers (click to flip)

advantage = (reward − mean) ÷ std

+1.7-0.6-0.6+1.7-0.6-0.6-0.6-0.6

mean (p)0.250

variance p(1−p)0.188

std0.433

std of a 0/1 reward at each success rate; the dot is this group

2 of 8 right, so the mean is 0.250: each right answer gets +1.73 and each wrong one -0.58. Dividing by the standard deviation (0.43) puts every prompt on the same scale. Flip answers until only one is right and watch its advantage grow. The advantages always add up to 0.

Click an answer to flip it between right (1) and wrong (0). Each answer’s advantage is how far its reward is from the group average.

Two things to notice in the widget. When one answer in eight is right, it gets a big advantage (about +2.6) and each wrong one a small penalty: a rare success is loud. And dividing by the standard deviation gives prompts near 0% or 100% pass rate the same overall weight as prompts near 50%, even though their raw differences are tiny. Dr. GRPO argues this over-weights nearly-solved and nearly-impossible prompts, and drops the division.

Covariance: wobbling together

Covariance measures whether two random variables move together. It is positive when one tends to be above its mean whenever the other is; negative when they move in opposite directions; zero when there is no linear relationship.

\operatorname{Cov}[X, Y] = \mathbb{E}\big[(X - \mu_X)(Y - \mu_Y)\big]

Divide by both standard deviations and you get the correlation, which always lies between −1 and 1.

example · ice cream and sunburn

Ice cream sales and sunburns are positively correlated: both go up on sunny days. Neither causes the other; sunshine is a common cause, as on the independence page. Correlation measures co-movement, not cause.

example · length and reward

Across a batch of answers, response length and reward are often correlated. In early RL training on maths, longer answers tend to be right more often (they show more work), so length and reward have positive covariance, and the model learns to write longer. If wrong answers tend to be longer instead (rambling when lost), the covariance is negative. Papers track this correlation to understand why lengths grow or shrink: see data and length.

The variance of a sum

Variances add for independent variables, but in general covariance appears:

\operatorname{Var}[X + Y] = \operatorname{Var}[X] + \operatorname{Var}[Y] + 2\operatorname{Cov}[X, Y]

Two consequences used constantly in ML:

The average of $n$ independent samples has variance $\sigma^2 / n$ : the noise shrinks with more samples. That’s the subject of error bars.
Subtracting something correlated reduces variance: $\operatorname{Var}[X - Y] = \operatorname{Var}[X] + \operatorname{Var}[Y] - 2\operatorname{Cov}[X, Y]$ . This is the whole idea behind baselines in policy gradients and paired comparisons between models: see the score-function trick and comparing models.

Where this shows up

GRPO and its fixes: GRPO, what GRPO gets wrong.
Baselines reduce the variance of the gradient: policy gradient.
Every error bar is a standard deviation of an average: error bars.

going deeper

Definitions and rules

\operatorname{Var}[aX + b] = a^2 \operatorname{Var}[X] \qquad \operatorname{Var}\Big[\sum_i X_i\Big] = \sum_i \operatorname{Var}[X_i] + 2 \sum_{i<j} \operatorname{Cov}[X_i, X_j]

\rho_{XY} = \frac{\operatorname{Cov}[X, Y]}{\sigma_X\, \sigma_Y} \in [-1, 1]

For $n$ i.i.d. samples the sample mean $\bar X$ and sample variance $s^2$ are

\bar X = \frac1n \sum_i X_i \qquad s^2 = \frac{1}{n - 1} \sum_i (X_i - \bar X)^2

Dividing by $n - 1$ rather than $n$ makes $s^2$ an unbiased estimate of the variance (the sample mean is closer to the samples than the true mean is, so dividing by $n$ comes out slightly too small). GRPO implementations typically use one or the other without comment; for $G = 8$ the difference is about 7% in the std. Bias is defined on the Monte Carlo page.

The law of total variance

When there are two levels of randomness, such as prompts and then answers to each prompt, variance splits into two parts:

\operatorname{Var}[R] = \underbrace{\mathbb{E}_x\big[\operatorname{Var}[R \mid x]\big]}_{\text{within a prompt}} + \underbrace{\operatorname{Var}_x\big[\mathbb{E}[R \mid x]\big]}_{\text{between prompts}}

Subtracting the per-prompt mean (as GRPO and RLOO do) removes the second part from the learning signal: the model is no longer rewarded for having been given an easy prompt. The same split appears in eval methodology as within-question vs between-question variance.

In papers

The GRPO advantage, from DeepSeekMath:

\hat A_{i,t} = \frac{r_i - \operatorname{mean}(\{r_j\}_{j=1}^G)}{\operatorname{std}(\{r_j\}_{j=1}^G)}

The subscript $t$ means the same advantage is given to every token $t$ of answer $i$ . Dr. GRPO removes the std division; DAPO drops groups whose std is zero (“dynamic sampling”) so every batch has signal.