Fixing GRPO

Dr. GRPO, DAPO, GSPO, CISPO and friends. A handful of small changes to GRPO that nearly every recipe since 2025 uses.

Key idea: Loss aggregation, advantage normalisation, clip-higher, filtering, and sequence-level ratios
Real world: Grading on a curve has known flaws. Divide by the class's spread and a quiz where everyone scored 98 to 100 suddenly counts as much as the final exam.
In an LLM: Olmo 3's RL recipe lists the typical set: drop groups with no signal, average the loss over tokens, clip higher than lower, no KL penalty, and no division by the group's standard deviation.

The problems GRPO has

GRPO worked, but several details of the original loss had side effects that only showed up at scale. Each fix below changes one line of the loss.

Problem	Effect	Fix	From
Each answer averaged over its own length	Long wrong answers punished less per token, so lengths grow	Average over all tokens in the batch, or a constant	DAPO, Dr. GRPO
Advantages divided by the group's std	Nearly-solved and nearly-impossible prompts get amplified	Don't divide	Dr. GRPO
Symmetric clip	Rare useful tokens can barely grow, so the model’s choices narrow (entropy collapses)	Higher upper clip, $\varepsilon_\text{high} > \varepsilon_\text{low}$	DAPO
Groups with identical rewards	Zero gradient, wasted compute	Skip them and sample more prompts	DAPO
Answers cut off at the length limit	Good reasoning scored 0 just for being long	Mask them, or a soft length penalty	DAPO
Per-token ratios on long answers	Noisy, unstable, especially for mixture-of-experts models	One ratio per answer, or clip the weight instead	GSPO, CISPO

1. How the loss is averaged

right answer length200 tokwrong answer length1200 tok

push per token, right : wrong6.0 : 1

whole right answer+0.50

whole wrong answer-0.50

Each answer is averaged over its own length, so both blocks have the same area (±0.50), and the long wrong answer is spread thin: each of its tokens is pushed down 6.0× less than a token of the short right answer is pushed up. Rambling dilutes the penalty, so wrong answers drift longer over training.

A group of two answers to one prompt, advantages +1 and −1. Each block is one answer: its width is the answer’s length in tokens, its height is how hard each token is pushed (up for the right answer, down for the wrong one). So its area is the push on the whole answer.

example · the same bias in an exam

An examiner gives each essay one overall mark, then spreads it evenly across the essay's sentences to decide which sentences to praise or criticise. A two-page wrong essay gets the same overall criticism as a half-page wrong essay, but each sentence of the long one gets a quarter of it. Students learn that padding protects bad sentences.

2. Dividing by the spread

example · a prompt the model almost always solves

Eight answers, seven correct: rewards $(1,1,1,1,1,1,1,0)$ . Mean 0.875, standard deviation 0.33.

	Right answers	Wrong answer
Advantage, divided by std (GRPO)	+0.38	−2.65
Advantage, not divided (Dr. GRPO)	+0.13	−0.88

Dividing by a small spread triples the signal from an easy prompt. In a batch, prompts that are almost always solved, or almost never, end up weighted far more than ones that are genuinely uncertain. Dropping the division keeps every prompt on the same scale.

The real-world version is the curve-grading problem in the summary: the narrower the spread, the bigger every small difference looks.

3. Clip higher than lower

ε low0.20ε high0.20

Probability now → highest allowed after this batch (capped by ε high)

·Sothe usual next word0.600 → 0.720

·Therefore0.200 → 0.240

·Waitstarts a self-check0.050 → 0.060

·Alternativelytries another approach0.010 → 0.012

Bar: how much probability the token can gain in this batch.

“Alternatively” can gain+0.0020

batches to grow it from 1% to 10%13

The clip limits the relative change, so the common token can gain 0.12 while “Alternatively” can gain only 0.0020. Batch after batch, the likely tokens get likelier and the rare exploratory ones barely move: entropy collapses. Raise ε high to give them more room.

Four candidate next tokens. How far can one batch move each one before the clip stops it? The clip limits the ratio new ÷ old probability to between 1 − ε low and 1 + ε high.

DAPO uses $\varepsilon_\text{low} = 0.2$ and $\varepsilon_\text{high} = 0.28$ . The asymmetry matters because entropy tends to collapse: likely tokens get likelier while rare exploratory ones, the "Wait" and "Alternatively" that start a self-check or a new approach, can hardly grow.

4. Skip groups with no signal

When every answer in a group scores the same, every advantage is 0 (GRPO). DAPO's dynamic sampling keeps drawing prompts until the batch is full of groups that do have a spread. More on choosing prompts in data and length.

5. Answers that hit the length limit

An answer cut off at the maximum length has no final answer, so it scores 0, even if the reasoning was going well. That punishes thinking. Options: leave truncated answers out of the loss entirely, or apply a soft penalty that grows over the last stretch before the limit. There's a simulation comparing all three in data and length.

example · DAPO's soft length penalty

Maximum length 20,000 tokens, with a 4,000-token warning zone:

Answer length	Extra reward
15,000	0
16,000	0 (zone starts)
18,000	−0.5
20,000	−1

A real-world analogue: a word limit with a sliding deduction, instead of failing anyone who goes one word over.

6. Ratios per answer, not per token

PPO clips each token's ratio separately. On long answers, a handful of tokens can be far off while the rest are fine. The per-token ratios are especially noisy in mixture-of-experts models (built from several specialist sub-networks), because a small weight change can switch which specialists are used.

GSPO (Qwen) computes one ratio per answer: the geometric mean of its token ratios (multiply them all, then take the root that matches the answer’s length). The whole answer is clipped or kept together.
CISPO (MiniMax) clips the importance weight but never zeroes the gradient, so every token keeps contributing a bounded push.
SAPO (Qwen) replaces the hard clip with a smooth gate.

Other baselines you'll meet

RLOO (leave-one-out): each answer's baseline is the mean of the other answers in its group. Unbiased, no critic.
REINFORCE++: normalises advantages over the whole batch instead of per group.
VAPO / VC-PPO: keep PPO's critic and fix how it's trained for long answers. A minority choice; most recipes stay critic-free.

going deeper

DAPO objective

J = \mathbb{E}\Bigg[\frac{1}{\sum_{i=1}^{G} |o_i|} \sum_{i=1}^{G} \sum_{t=1}^{|o_i|} \min\Big(\rho_{i,t} \hat A_i,\ \operatorname{clip}\big(\rho_{i,t}, 1 - \varepsilon_\text{low}, 1 + \varepsilon_\text{high}\big) \hat A_i\Big)\Bigg]

subject to the group containing at least one correct and one incorrect answer, with no KL term.

Dr. GRPO advantage

$\hat A_i = R_i - \operatorname{mean}(R_{1..G})$ , and the per-answer sum is divided by a constant (for example the maximum length) instead of $|o_i|$ .

Overlong penalty

With maximum length $L_\text{max}$ and warning zone $L_\text{cache}$ :

R_\text{length}(y) = \begin{cases} 0 & |y| \le L_\text{max} - L_\text{cache} \\ \dfrac{(L_\text{max} - L_\text{cache}) - |y|}{L_\text{cache}} & L_\text{max} - L_\text{cache} < |y| \le L_\text{max} \\ -1 & |y| > L_\text{max} \end{cases}

GSPO and CISPO

\text{GSPO:}\quad s_i = \Big(\frac{\pi_\theta(y_i \mid x)}{\pi_\text{old}(y_i \mid x)}\Big)^{1/|y_i|} \qquad \text{CISPO:}\quad \operatorname{sg}\big[\operatorname{clip}(\rho_{i,t}, 1 - \varepsilon_\text{low}, 1 + \varepsilon_\text{high})\big]\, \hat A_i \log \pi_\theta(o_{i,t})

where $\operatorname{sg}$ means stop-gradient: the clipped weight scales the update but isn't itself differentiated.

RLOO

A_i = R_i - \frac{1}{G - 1} \sum_{j \ne i} R_j

Papers

Understanding R1-Zero-Like Training (Dr. GRPO, Mar 2025)
DAPO (Mar 2025)
Group Sequence Policy Optimization (GSPO, Jul 2025)
MiniMax-M1 (CISPO, Jun 2025)
Soft Adaptive Policy Optimization (SAPO, Nov 2025)
Back to Basics: REINFORCE-style optimisation (RLOO, 2024), REINFORCE++, VAPO
GDPO (Jan 2026): normalise each reward separately when there are several