When the sampler and trainer disagree

The fast engine that generates answers and the code that trains on them compute slightly different probabilities for the same tokens. Left alone, this quietly destabilises RL.

Key idea: Training–inference mismatch, and the corrections for it
Real world: A coach reviews match footage that was recorded at a slightly wrong speed. Every movement looks a little different from what the athlete actually did, and advice based on the footage is slightly off.
In an LLM: vLLM says it sampled a token with probability 0.300. The trainer, with exactly the same weights, computes 0.297. Over thousands of tokens and steps, these differences add up and can make training collapse.

Where it comes from

RL for LLMs uses two separate pieces of software. An inference engine (vLLM, SGLang) generates answers as fast as possible. A trainer (PyTorch with FSDP or Megatron) recomputes probabilities and takes gradient steps. They should agree exactly. They don't, because:

Different kernels (the low-level routines that do the maths) add numbers in different orders. Computers round as they go, so a different order gives a slightly different total.
Low precision: bfloat16 (a compact number format) rounds more coarsely, so those differences are larger.
Batch-dependent results: an engine's output for one answer can change depending on what else is in the batch.
Mixture-of-experts routing (each token is sent to a few specialist sub-networks): a tiny difference can flip which experts handle a token, giving a much larger difference in its probability.

The upshot: even "on-policy" RL is slightly off-policy, with the inference engine as the behaviour policy.

example · small per token, large per answer

Suppose each token's log-probability differs between the engines by a random ±0.01. Over a 10,000-token answer, those differences add up like a random walk, to about $0.01 \times \sqrt{10000} = 1$ . The sequence-level ratio between the engines is then typically off by a factor of around $e^1 \approx 2.7$ , one way or the other.

answer length10k tok

typical gap per token±0.0100

typical whole-answer ratio×2.72 or ÷2.72

answers off by 2× or more49%

one answertypical spread

Each token is off by only about 1%, in a random direction. Over 10,000 tokens the gaps add up like a random walk, so a typical answer’s probability differs by a factor of 2.7 between the two engines. The trainer treats these answers as on-policy when they aren’t quite.

Eight answers. Each line adds up, token by token, how far the trainer’s log-probability is from the sampler’s for the same token. The height at the end is the whole answer’s probability ratio between the two. Dashed: the typical spread, growing like the square root of length.

example · footage at the wrong speed

At 2% too fast, a single frame looks fine. Over a 90-minute match, the footage drifts by almost two minutes, and timing advice based on it is wrong. Small, consistent errors compound over long sequences.

Fixes

Fix	What it does
Truncated importance sampling (TIS)	Multiply each token's loss by trainer prob ÷ sampler prob, capped at C. Corrects the drift, bounds the noise.
Masking (MIS, IcePop)	Drop tokens or whole answers whose ratio is outside a trusted range, instead of reweighting them.
Off-policy sequence masking (DeepSeek-V3.2)	Drop answers with negative advantage that have drifted too far. A bad answer the current model would no longer write teaches little and can destabilise training.
FP16 instead of BF16	More digits of precision (mantissa bits) make the two engines agree far more closely, with no algorithm change.
Routing replay (R3, "keep routing")	Record which experts the inference engine used and force the trainer to use the same ones.
Keep the sampling mask	If generation used top-p or top-k (cutting the list of candidate tokens), apply the same truncation when the trainer computes probabilities.
Deterministic, batch-invariant kernels	Make both sides compute identical numbers. Costs some speed.
FP32 output layer (MiniMax-M1)	Compute the final softmax in higher precision, where rounding matters most.

going deeper

TIS

With trainer policy $\pi_\theta$ and sampler probabilities $\mu$ :

\nabla J \approx \sum_t \min\Big(\frac{\pi_\theta(o_t \mid s_t)}{\mu(o_t \mid s_t)},\ C\Big)\, \hat A\, \nabla \log \pi_\theta(o_t \mid s_t)

DeepSeek-V3.2 sequence mask

An answer $i$ is dropped when both

\hat A_i < 0 \quad\text{and}\quad \frac{1}{|o_i|} \sum_t \log \frac{\mu(o_{i,t})}{\pi_\theta(o_{i,t})} > \delta

Papers and posts

Yao et al., Your Efficient RL Framework Secretly Brings You Off-Policy RL Training (blog, Aug 2025): TIS
Defeating the Training-Inference Mismatch via FP16 (Oct 2025)
Stabilizing MoE RL by Aligning Training and Inference Routers (R3, Oct 2025)
Every Step Evolves (Ring-1T, IcePop, Oct 2025)
DeepSeek-V3.2 (Dec 2025): off-policy sequence masking, keep routing, unbiased KL
He et al., Defeating Nondeterminism in LLM Inference (Thinking Machines, Sep 2025)

Where it comes from

Fixes

TIS

DeepSeek-V3.2 sequence mask

Papers and posts

Related