PPO clipping

Reuse each batch of experience for several updates without letting one lucky sample swing the policy too far.

Key idea: The clipped probability ratio
Real world: A robot learning to walk. Collecting experience is slow, so you want to train on each batch several times. But if one stumble happened to go well, you don't want to rebuild the whole gait around it.
In an LLM: Generating answers is the slow part, so trainers take several gradient steps per batch. epsilon=0.2 in TRL's GRPOConfig, cliprange in PPOConfig.

The problem

Plain policy gradient uses each sample once and throws it away. Reusing a batch for several steps is cheaper, but each step pushes the same actions further in the same direction. After a few steps, the policy can move far past what one batch of evidence justifies, and training collapses.

PPO (Proximal Policy Optimization, OpenAI 2017) limits how far each update can go.

The idea

Ratio ρ = the action's probability now ÷ its probability when it was sampled. ρ = 1 means no change yet. This is an importance weight: it corrects for the batch having been sampled by a slightly older policy.
Inside [1 − ε, 1 + ε], usually [0.8, 1.2], the update works like policy gradient.
Once ρ leaves that range in the direction the advantage is pushing, the objective goes flat. A flat objective has zero gradient, so the push stops.

Try it

advantage+0.80ε0.20ρ now1.00

step 0

At ρ = 1.00: objective 0.80, slope 0.80. The gradient keeps pushing this token up. Press Take a gradient step to follow it.

Solid: the clipped objective PPO maximises. Dashed: the unclipped one. Red band: flat, so no gradient.

example · four cases with ε = 0.2

PPO takes the smaller of the unclipped term $\rho A$ and the clipped term $\text{clip}(\rho) \cdot A$ (ρ held inside the range).

Advantage	Ratio ρ	$\rho A$	Clipped	Objective	Still learning?
+2	1.1	2.2	2.2	2.2	Yes, inside the range
+2	1.5	3.0	1.2 × 2 = 2.4	2.4	No, already raised enough
−1	0.5	−0.5	0.8 × −1 = −0.8	−0.8	No, already lowered enough
−1	1.5	−1.5	1.2 × −1 = −1.2	−1.5	Yes, a bad action got more likely, so keep correcting

The clip only stops updates that have already gone far enough in the right direction. Mistakes in the wrong direction are always corrected.

example · one token in a language model

The model answered "What is 17 × 6?" correctly, and the answer got advantage +1. When it was sampled, the token 102 had probability 0.40. The trainer takes several gradient steps on this batch:

Step	P( `102`)	ρ	Gradient?
1	0.40	1.00	Yes
2	0.45	1.13	Yes
3	0.49	1.23	No, past 1.2

From step 3 on, this token stops pushing, while other tokens in the batch that haven't moved as far keep learning. The robot's gait works the same way: once a movement is 20% more likely than when it was tried, that batch has nothing more to say about it.

Where it's used

PPO became the default policy-gradient method because it's simple and stable:

OpenAI Five used it to play Dota 2 at a professional level.
Robotics simulators use it for locomotion and manipulation.
RLHF, the method behind InstructGPT and early ChatGPT, used PPO to tune the model against a learned reward model.

For language models PPO is heavy: the policy, a frozen reference copy, the reward model and a critic as large as the policy all sit in memory at once. DPO removes the reward model and the RL loop; GRPO removes the critic. Both came from wanting PPO's results without its cost.

going deeper

The objective

L(\theta) = \mathbb{E}_t\Big[\min\big(\rho_t A_t,\ \operatorname{clip}(\rho_t, 1-\epsilon, 1+\epsilon)\, A_t\big)\Big] \qquad \rho_t = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_\text{old}}(a_t \mid s_t)}