Verifiable rewards and reasoning

Let a program check the answer, run GRPO on hard maths and code, and models learn to think longer. This is the recipe behind DeepSeek-R1 and today's reasoning models.

Key idea: RLVR: reinforcement learning with verifiable rewards
Real world: Practising maths from a textbook with the answers at the back. Nobody marks your working; you only check whether your final number matches, and over time you learn the methods that get there.
In an LLM: DeepSeek-R1-Zero: GRPO on a base model with only two rewards, a correct final answer and the right output format. Its AIME 2024 accuracy rose from 15.6% to 71.0%.

What makes a reward verifiable

A verifiable reward is computed by a program rather than a learned model. It is cheap, exact, and much harder to hack than a reward model.

Task	Verifier	Reward
Maths	Parse the final answer, compare with the answer key	1 if equal, else 0
Code	Run hidden unit tests in a sandbox	Fraction of tests passed, or 1 if all pass
Format	A pattern match (regex) for the expected structure, e.g. `<think>…</think>` then the answer	Small bonus
Instruction following	Check constraints: "exactly 3 bullet points", "valid JSON", "under 100 words"	1 per satisfied constraint
Logic puzzles	Check the solution satisfies every rule	1 if valid

What happened with DeepSeek-R1

R1-Zero applied GRPO directly to a base model, with no supervised fine-tuning first. As training went on:

Answers grew from hundreds to thousands of tokens.
Behaviours like re-checking appeared in the text: "Wait, let me verify this step."
Accuracy kept climbing with the length.

The published R1 model added stages around that core to make it readable and general:

Cold start: a first round of supervised fine-tuning (training on written examples) on a few thousand long, readable reasoning examples.
Reasoning RL with verifiable rewards, plus a reward for answering in a consistent language.
Rejection sampling: generate many answers, keep the good ones, fine-tune on them along with general data.
RL for all scenarios: verifiable rewards for reasoning, reward models for helpfulness and harmlessness.

Later work complicated the story. Base models such as Qwen2.5 already show some self-reflection before any RL, and part of the length growth came from a bias in GRPO's loss rather than from better reasoning. The current view: RL with verifiable rewards makes behaviours the base model already has much more common. Whether it also creates new ones is the subject of Does RL teach new skills?.

example · an answer key with mistakes

A student practises from a book whose answer key has typos. Every time they get the right answer, the key says it's wrong. If they trust the key, they learn to make the same mistakes the key does. The quality of what you learn is capped by the quality of the checker.

example · a verifier with false negatives

The answer to a problem is ½. A strict string-match verifier only accepts the exact text 1/2.

Model writes	Correct?	String match	Maths-aware check
`1/2`	✓	1	1
`0.5`	✓	0	1
`\frac{1}{2}`	✓	0	1
`2/4`	✓	0	1

With string matching, three of the four correct answers here get reward 0, so the model is taught that correct answers in those forms are wrong. Real pipelines use symbolic equivalence checkers for maths, and run code against tests instead of comparing text.

Try it

A verifier can fail in two directions. A false negative marks a right answer wrong; a false positive marks a wrong answer right. Both turn straight into advantages, so the model learns from the verifier's mistakes exactly as it learns from its correct verdicts.

This verifier accepts only the text 1/2.

answerreallyrewardadvantage

1/2right1+2.65

0.5false negative: right, scored 0right0-0.38

\frac{1}{2}false negative: right, scored 0right0-0.38

50%false negative: right, scored 0right0-0.38

2/4false negative: right, scored 0right0-0.38

1/3wrong0-0.38

11/2wrong0-0.38

1/3 or 1/2wrong0-0.38

false negatives4

false positives0

right answers pushed down4 of 5

Only one of the five right answers counts. The other four get the same negative advantage as 1/3, so training teaches the model that writing 0.5 or 50% is as bad as getting it wrong. A strict format can be fine if the prompt asks for it; this one didn’t, so the model is being taught noise.

Prompt: “A bag has 2 red and 2 blue marbles. What is the chance the first one drawn is red? Answer in <answer> tags.” Eight sampled answers (the tag contents are shown), scored by a verifier, then turned into GRPO advantages.

Why verifiable rewards took over

Hard to hack: a unit test can't be flattered. It can still be gamed, for example by special-casing the test inputs; see reward design.
No reward model to train or keep in memory.
Scales: millions of maths problems and coding tasks already exist with answers or tests.

The limitation is obvious: most of what people want from a chat model, such as a good explanation, a kind reply or a well-written essay, has no program that can check it. Rubrics and judges are the current attempt to extend RL there.

going deeper

A typical reward function

R(y) = \underbrace{\mathbb{1}[\text{answer}(y) \equiv \text{answer key}]}_{\text{accuracy}} + \lambda\, \underbrace{\mathbb{1}[y \text{ matches the format}]}_{\text{format}}

with λ small (0.1 to 0.2) so the model can't win on format alone. Some recipes make format a gate instead: a badly formatted answer scores 0 regardless.

Papers

DeepSeek-R1 (Jan 2025)
Kimi k1.5 (Jan 2025): RL on long chains of thought, released the same week
Understanding R1-Zero-Like Training (Mar 2025): base models already self-reflect; GRPO's length bias