- Key idea
- Group-relative advantage: the group is the baseline
- Real world
- Grading on a curve. A 70% is great if the class averaged 50% and poor if it averaged 90%. GRPO scores each answer against the other answers to the same question.
- In an LLM
- DeepSeek-R1-style reasoning training: 8 to 16 samples per maths prompt, reward = right answer plus right format, no critic network. TRL's
GRPOTrainer.
Why a new algorithm
Policy gradient needs a baseline, and PPO gets it from a critic: a second network, roughly the size of the policy, trained alongside it. For a large language model that doubles the memory and adds a model that's hard to train well.
GRPO (Group Relative Policy Optimization, introduced with DeepSeekMath in 2024) drops the critic. The baseline comes from sampling several answers to the same prompt and comparing them.
One GRPO step
- Sample a group of G answers for one prompt.
- Score each one. Here: +1 for the right number, +0.2 for putting it in the answer tags.
- Normalise: advantage = (score − group mean) ÷ group standard deviation (a measure of how spread out the scores are).
- Update as in policy gradient, with PPO's clip and optionally the KL penalty.
Prompt: "What is 17 × 6?". The model writes four answers:
| Answer | Correct? | Tags? | Reward | Advantage |
|---|---|---|---|---|
<answer>102</answer> | ✓ | ✓ | 1.2 | +0.99 |
<answer>112</answer> | ✓ | 0.2 | −0.81 | |
| "It's about 100" | 0 | −1.17 | ||
<answer>102</answer> | ✓ | ✓ | 1.2 | +0.99 |
Mean reward = 0.65, standard deviation ≈ 0.55. The correct answers are made more likely, the wrong ones less, and the answer that ignored the format is pushed down the most.
If all four had been correct, every reward would be 1.2, every advantage 0, and the prompt would teach nothing.
Try it
Why it works
The group mean plays the role of the baseline from policy gradient. It is re-estimated for each prompt from samples you already paid for, so there is no critic to train. Dividing by the spread keeps the update size similar for easy and hard prompts, although that turned out to have side effects.
GRPO as first published has a few such quirks, and nearly every recipe since 2025 changes some of them. The next pages cover what verifiable rewards made possible and then the standard fixes.
Advantage and loss
The example uses the population standard deviation. Variants such as Dr. GRPO and DAPO change the length normalisation and drop the std division; see Fixing GRPO. TRL exposes these through loss_type and scale_rewards; check the docs for your version.
Paper
- DeepSeekMath (Feb 2024), where GRPO was introduced