- Key idea
- Off-policy distillation, rejection sampling, on-policy distillation, and how much RL actually teaches
- Real world
- Learning chess three ways: replay grandmaster games move by move, play your own games and only learn whether you won, or play your own games with a coach annotating every move you made.
- In an LLM
- DeepSeek fine-tuned small models on 800,000 reasoning traces from R1. The distilled small models beat the same small models trained with RL directly.
Three ways to learn from a better model
| Method | Who writes the training text | Signal per token | Chess version |
|---|---|---|---|
| Distillation (SFT) | The teacher | "Write exactly this" | Replay grandmaster games |
| RL | The student | One reward for the whole answer | Play, learn only win or lose |
| On-policy distillation | The student | The teacher's opinion of every token | Play, with a coach marking each of your moves |
Distillation is ordinary fine-tuning on a stronger model's answers. It’s cheap and gives feedback on every token, but it is off-policy: the student imitates text it would never have written, and never sees its own mistakes corrected.
RL trains on the student's own answers, which fixes that, but gives only one number per answer.
On-policy distillation gets both. The student writes an answer; the teacher computes its probability for every token the student wrote; each token is pushed toward what the teacher would have done.
The student solving 17 × 6 writes two tokens worth looking at:
| Token | Student's probability | Teacher's probability | Advantage |
|---|---|---|---|
112 (wrong) | 0.40 | 0.02 | −3.0 |
Let me check | 0.60 | 0.90 | +0.41 |
The wrong token is pushed down hard, the self-check is pushed up a little. No answer key is needed, and every token gets its own signal instead of sharing one reward.
Try it
After your game, a coach goes through it and marks each of your moves: good, dubious, blunder. You learn from positions you reached, which replaying grandmaster games never shows you, and you learn far more per game than from the result alone.
Rejection sampling and expert iteration
A simple middle ground: sample many answers, keep the ones that are correct, and fine-tune on them. Repeat. This is STaR, RAFT or "expert iteration" depending on the paper. It's the baseline that RL has to beat, and it's a standard stage in recipes like DeepSeek-R1.
How much does RL teach?
An outcome reward is roughly one bit of information per answer: right or wrong. Distillation gives the teacher's full preference at every token. Thinking Machines used this argument to explain why LoRA (a small add-on that trains far fewer numbers) matches full fine-tuning for RL even when made very small: there isn't much information to store. A related finding, "RL's razor": because RL changes the model only as much as the reward requires, it forgets less of what the model knew than fine-tuning does.
In frontier recipes
Since late 2025, several labs train separate RL "experts" for maths, code, agents and so on, then merge them into one model with on-policy distillation from all of them. A September 2026 study found that running on-policy distillation first and RL second beats either alone, and beats mixing them: distillation widens what the model can do, and RL then sharpens it. See frontier recipes.
On-policy distillation objective
The student samples ; minimise the per-token reverse KL to the teacher :
A cheap estimator uses only the sampled token: per-token advantage , plugged into the usual policy-gradient loss. Using the full vocabulary distribution is lower-variance and more expensive.
Papers and posts
- MiniLLM and GKD: On-Policy Distillation of Language Models (2023)
- Thinking Machines, On-Policy Distillation and LoRA Without Regret (2025)
- STaR (2022)
- RL's Razor: Why Online RL Forgets Less (Sep 2025)
- Self-Distilled Reasoner (Jan 2026): the model as its own teacher, given a reference solution
- Sequential Beats Joint: On-Policy Distillation and RLVR (Sep 2026)