- Key idea
- Difficulty filtering, curricula, and controlling answer length
- Real world
- A good coach picks drills at the edge of an athlete's ability. Drills they always nail teach nothing; drills they can never complete teach nothing either.
- In an LLM
- Keep prompts the model currently solves somewhere between roughly 10% and 90% of the time, and keep re-checking as it improves.
Most groups can be wasted
A GRPO group only teaches something if its answers score differently. If a prompt’s pass rate (the share of its answers that are correct) is p and you sample a group of 8, the chance every answer gets the same score is .
| Model solves the prompt | Group has no signal |
|---|---|
| 50% of the time | 0.8% |
| 90% | 43% |
| 95% | 66% |
| 99% | 92% |
As training succeeds, more prompts move to the bottom rows. A run that started efficient wastes most of its generation on solved prompts unless the prompt set is refreshed.
A tennis coach notices that their student now returns every serve at 120 km/h. Practising more of those is cheap and pleasant, and it teaches nothing. Moving to 140 km/h, where the student wins about half the points, is uncomfortable and exactly where improvement happens.
Ways to choose prompts
- Offline difficulty filtering: before training, sample each prompt several times with the starting model and keep only the prompts it sometimes solves, not the ones it always or never solves.
- Dynamic sampling (DAPO): during training, drop groups with no spread and draw more prompts until the batch is full.
- Retiring solved prompts: ScaleRL removes prompts once the model's pass rate reaches 90% or more.
- Curricula: start with easier prompts or shorter context limits and raise them as training progresses. Several open recipes raise the maximum answer length in stages.
Length
RL on verifiable rewards tends to make answers longer, because longer reasoning often helps. But not always:
- Overthinking. Reasoning models have been shown to spend hundreds of tokens on "What is 2 + 3?", checking and rechecking a trivial answer.
- Truncation. Answers that hit the limit score 0, which wastes compute and, without care, punishes long reasoning.
An exam asks for an answer "in about 200 words". Students who write 2,000 waste time and often lose marks for clarity, even if the content is right. Students who write 50 rarely cover enough. The limit is part of the task.
L1 trains a model with prompts like "Think for 512 tokens", rewarding correct answers and penalising the gap between the requested and actual length. The trained model can then be steered: the same question asked with a 512-token budget or a 4,000-token budget trades accuracy for cost on demand. Products expose the same idea as a "reasoning effort" or "thinking budget" setting.
Other length controls you'll see:
- Length penalty in the reward, often only applied to correct answers, so the model isn't pushed toward short wrong ones.
- Group-relative length reward: among correct answers in a group, shorter ones get a bonus.
- Long-to-short (Kimi k1.5): after training a long-thinking model, make a shorter-answering model that keeps its accuracy. Methods include length penalties, training on only the shortest correct answers (rejection sampling), and averaging models together (model merging).
L1 reward
where is the number of tokens the model used.
Wasted-group probability
For pass rate and group size , the probability of zero spread is , minimised at . Larger groups reduce it but cost proportionally more generation.
Papers
- Do NOT Think That Much for 2+3=? (overthinking, Dec 2024)
- L1: Controlling How Long a Reasoning Model Thinks (Mar 2025)
- Kimi k1.5 (long-to-short, Jan 2025)
- DAPO (dynamic sampling), The Art of Scaling RL Compute (retiring solved prompts)