- Key idea
- Rollouts, synchronous vs asynchronous RL, and staleness
- Real world
- A kitchen that won't start the next table's order until every dish for the last table is out sits idle waiting for the slowest dish. A kitchen that keeps cooking continuously serves far more, though some dishes were started before the latest change to the menu.
- In an LLM
- In a batch of 512 answers, most finish in 2,000 tokens, but one runs to 32,000. A synchronous trainer waits for that one while every other GPU sits idle.
Where the time goes
A rollout is one sampled answer, or one full agent episode, together with its reward. Producing it means generating tokens one at a time, which is slow, because the hardware spends most of its time reading memory. The gradient step afterwards is comparatively quick. In reasoning and agent RL, generation usually takes most of the real time.
Answer lengths also vary a lot. Most finish quickly; a few think for a very long time. If every training step waits for the slowest answer, most of the hardware is idle most of the time.
Try it
Synchronous vs asynchronous
- Synchronous: generate a batch with the current weights (the model’s learned numbers), train on it, update the weights, repeat. All data is on-policy. Simple, and wasteful.
- Asynchronous: generation and training run at the same time on separate machines. The trainer takes whichever answers are finished; the generators pick up new weights whenever they're ready. Fast, but answers can be a few versions stale: written by older weights than the ones being trained.
Staleness is handled with importance sampling: each token is reweighted by current probability ÷ the probability when it was generated, usually with a cap. Recipes typically allow only a small gap, a handful of versions, though some newer methods tolerate far more.
An answer starts generating with weights version 3. It's long, so by the time it finishes, the trainer has published versions 4 and 5. It is trained on at version 5: two versions stale. If a token had probability 0.20 under version 3 and 0.26 under version 5, its importance weight is 1.3.
Other tricks
- Partial rollouts (Kimi k1.5): pause very long answers at a token budget and continue them in the next iteration, instead of letting them block the batch.
- In-flight weight updates (PipelineRL): push new weights to the generators mid-answer, so one answer can be written by several versions. Staleness stays low without stopping generation.
- Colocated vs disaggregated: colocated means generation and training take turns on the same GPUs (simpler, less memory traffic). Disaggregated means they run on separate pools (needed for asynchronous training and for agents waiting on slow tools).
Frameworks you'll see named
| Framework | From | Notes |
|---|---|---|
| TRL | Hugging Face | Easiest start; GRPOTrainer |
| verl | ByteDance | Widely used for research-scale reasoning RL |
| OpenRLHF | Open source | Ray-based, PPO and GRPO |
| slime | Zhipu (GLM) | Megatron + SGLang; used for GLM models |
| AReaL | Ant / Tsinghua | Fully asynchronous |
| PipelineRL | ServiceNow | In-flight weight updates |
| SkyRL, prime-rl | Berkeley, Prime Intellect | Agent and async focused |
| Tinker | Thinking Machines | Hosted API: you write the loop, they run the GPUs |
The off-policy objective
With rollouts from a behaviour policy (an older version, or the inference engine), the per-token clipped objective becomes
Some systems (AReaL's decoupled PPO) separate the two roles: the trust region is measured against a recent "proximal" policy, while the importance weight corrects for the older behaviour policy.
Papers
- AReaL (May 2025), PipelineRL (Sep 2025)
- Kimi k1.5 (partial rollouts)
- The Art of Scaling RL Compute (asynchronous training in a scaling recipe)