How frontier models are trained

Modern post-training (everything after the first big round of learning from text) chains supervised fine-tuning, several kinds of RL, and distillation. This page maps the published recipes onto the ideas in this course.

Key idea: Multi-stage post-training, and predictable RL scaling
Real world: Training a doctor: lectures, then supervised practice, then specialist rotations, then independent work under review. Each stage needs the one before it.
In an LLM: DeepSeek-R1: a short supervised cold start, reasoning RL, rejection-sampled fine-tuning, then RL for everything else.

The common shape

Almost every published recipe since 2025 follows a version of this:

Supervised fine-tuning (cold start): teach the format and basic behaviour from curated examples, often long reasoning traces.
Reasoning RL with verifiable rewards: maths, code, logic, using GRPO or a variant.
Agentic RL in tool environments: search, coding, terminals.
General RL for helpfulness and safety, with reward models, judges or rubrics.
Distillation or merging: combine specialist models, or transfer to smaller ones, often by on-policy distillation.

example · training a surgeon

No hospital puts a first-year student straight into the operating theatre to learn by trial and error. They learn the basics from teachers first (SFT), then practise with feedback on outcomes (RL), then learn from senior surgeons reviewing their own operations (on-policy distillation). Trial and error only works once you can occasionally succeed.

Published recipes

All details below are as reported by each lab's technical report.

Model	Released	RL highlights
DeepSeek-R1	Jan 2025	GRPO with rule-based rewards; R1-Zero showed RL alone on a base model works. Tried process rewards and tree search and dropped them.
Kimi k1.5	Jan 2025	Long-context RL with partial rollouts; long-to-short methods for shorter answers.
Qwen3	May 2025	Long-reasoning cold start, reasoning RL, then fused thinking and non-thinking modes; small models by distillation. GSPO came out of this line.
MiniMax-M1	Jun 2025	CISPO; FP32 output head to reduce sampler–trainer mismatch.
Kimi K2	Jul 2025	Verifiable rewards plus a self-critique rubric reward for open-ended tasks; heavy agentic data.
GLM-4.5	Aug 2025	Separate expert RL for reasoning, agents and chat, then distilled into one model; trained with slime.
Olmo 3	Dec 2025	Fully open: SFT, then DPO, then RLVR with a DAPO-style recipe, asynchronous training and truncated importance sampling.
DeepSeek-V3.2	Dec 2025	Large synthetic agentic-task pipeline; unbiased KL, off-policy sequence masking, keep routing.
GLM-5	Feb 2026	Asynchronous agentic RL infrastructure at scale.
DeepSeek-V4	Apr 2026	Domain experts trained with GRPO, merged into one model by on-policy distillation from all of them; generative reward models.
Kimi K3	Jul 2026	RL experts at several reasoning-effort levels, merged by multi-teacher on-policy distillation; very long agentic episodes with persistent sandboxes.

Closed labs (OpenAI, Anthropic, Google) publish far less about their RL, though papers from them on reward hacking and monitoring show it is central to their training too.

RL scaling is predictable

Pre-training has well-known scaling laws. Meta's The Art of Scaling RL Compute (Oct 2025) found that RL has its own: reward against compute follows an S-shaped curve (a sigmoid), so a curve fitted on the early part of a run predicts where a run ten times longer will end up.

ceiling A0.50efficiency B2.5midpoint10^2.3

at 100 GPU-h: yours / ref16% / 12%

at 300,000 GPU-h: yours / ref50% / 60%

reference recipeyour recipe

Your recipe wins a small run and loses a big one. Judged at 100 GPU-hours it looks better; its lower ceiling A only shows at scale. Fitting the whole curve to the small run is what exposes this before you pay for the big one.

Expected reward against training compute, from 1 to 1 million GPU-hours on a log scale. Dashed: a reference recipe (A = 60%, B = 1.5, midpoint 1,000 GPU-hours). Solid: yours. Both start at 10%.

The simulation starts with a recipe that climbs faster than the reference but has a lower ceiling. Compare the two at 100 GPU-hours and at 300,000: the recipe that wins the small run loses the big one.

example · reading the curve

A recipe with ceiling A = 60%, starting reward 10%, efficiency B = 1.5 and midpoint 1,000 GPU-hours (the compute at which half the gain is made):

At 1,000 GPU-hours: $10\% + 50\% \div 2 = 35\%$
At 4,000 GPU-hours: $10\% + 50\% \div (1 + 0.25^{1.5})$ , which is $10\% + 50\% \div 1.125 \approx 54\%$

The last 6 points to the ceiling cost far more than the first 44. Their finding: most recipe tweaks change how fast you climb, not where you end up. That makes it possible to compare recipes on small runs without being fooled by early speed.

going deeper

The ScaleRL curve

R_C = R_0 + \frac{A - R_0}{1 + (C_\text{mid} / C)^B}

$A$ is the asymptotic reward, $B$ the steepness, $C_\text{mid}$ the compute at which half the gain is reached. Their recipe combined asynchronous PipelineRL-style training, the CISPO loss, an FP32 output head, prompt-level loss averaging, batch-level advantage normalisation, zero-variance filtering and retiring solved prompts.

Reports

DeepSeek-R1 · Kimi k1.5 · Qwen3 · MiniMax-M1 · Kimi K2 · GLM-4.5 · Olmo 3 · DeepSeek-V3.2 · GLM-5 · DeepSeek-V4 · Kimi K3 · The Art of Scaling RL Compute

The common shape

Published recipes

RL scaling is predictable

The ScaleRL curve

Reports

Related