KL and reward hacking

A reward is only a proxy for what you want. Optimise it too hard and the agent finds its blind spots.

Key idea: Reward hacking, and a KL penalty that keeps the policy near where it started
Real world: A cleaning robot rewarded for "no visible mess" learns to cover the mess instead of cleaning it.
In an LLM: A reward model that prefers longer answers produces padded, repetitive replies. beta in TRL's GRPOConfig sets the KL weight; recent versions default it to 0.

Reward hacking

An agent optimises exactly the reward you wrote down, not the goal you had in mind. When the two differ, a strong enough optimiser finds the gap. Some real cases:

CoastRunners (OpenAI, 2016): a boat-racing agent found it scored more by circling a lagoon and hitting the same targets forever than by finishing the race.
Tetris: a program told to avoid losing learned to pause the game just before the last block landed, and never unpause.
Chat models: reward models trained on human preferences often favour longer, more confident answers, so RL drifts toward padding and flattery.
Coding models: rewarded for passing unit tests, a model can learn to special-case the test inputs, or edit the tests, instead of fixing the code.
Maths models: a checker that looks for the last number in the answer can be gamed by listing several numbers.

Better rewards help, but you can never specify everything. The other defence is to limit how far the policy can move.

The KL penalty

Score = reward − β × (how far the policy has drifted from a frozen reference copy of itself). The drift is measured with the , which is 0 when the two policies are identical. Small β: chase the reward. Large β: stay close to the original behaviour.

Try it: a reward model that likes long answers

β0.21

reference modelRL-trained modelreward model scoretrue quality

reward model score0.37 (start 0.31)

true quality0.79 (start 0.77)

drift (KL)0.17 nats

Over-optimisation: both scores against how far the policy has drifted

A real improvement: true quality 0.79, up from 0.77, about the best this reward model allows. Lower β and watch the green curve turn over.

Top: how often each response length is produced, with what the reward model pays and what a human would actually rate. Bottom: the same two scores as the policy drifts further from the reference.

example · how β trades reward for staying put

A model gives a short answer 80% of the time and a long one 20%. The reward model scores them 0.5 and 1.0. With a KL penalty, the best possible policy multiplies each answer’s original chance by $e^{\text{reward} / \beta}$ and rescales so the chances add up to 100%:

β	Short answer	Long answer
Reference model	80%	20%
1	$0.8 \cdot e^{0.5} = 1.32$ → 71%	$0.2 \cdot e^{1} = 0.54$ → 29%
0.1	$0.8 \cdot e^{5} = 119$ → 3%	$0.2 \cdot e^{10} = 4405$ → 97%

With β = 1 the model shifts a little toward the higher reward. With β = 0.1 it almost always gives the long answer. If length is what the reward model is wrong about, that is the hack winning.

In language models

The reference model is the model as it was before RL. Rewards that can be checked by rules, such as an exact maths answer or passing unit tests, are much harder to hack than a learned reward model. That's one reason many recent training recipes use them with little or no KL penalty: DAPO and Olmo 3 drop it entirely for maths and code, while DeepSeek keeps a small, per-domain one. For learned rewards, which are easier to hack, the penalty stays.

going deeper

Objective and its optimum

\max_\pi\ \mathbb{E}_{y \sim \pi}\big[r(y)\big] - \beta\, \mathrm{KL}\big(\pi \,\|\, \pi_\text{ref}\big) \quad\Rightarrow\quad \pi^*(y) \propto \pi_\text{ref}(y)\, \exp\!\big(r(y) / \beta\big)

\mathrm{KL}\big(\pi \,\|\, \pi_\text{ref}\big) = \sum_y \pi(y) \log \frac{\pi(y)}{\pi_\text{ref}(y)}

The widget and the example compute $\pi^*$ exactly from this formula. The same closed form is the starting point for DPO, which trains on preference pairs directly without sampling.

Estimating KL from samples

The exact KL needs a sum over every possible answer, so trainers estimate it from the sampled tokens. With $r = \pi_\text{ref}(o_t) / \pi_\theta(o_t)$ for a sampled token, three standard estimators are:

k_1 = -\log r \qquad k_2 = \tfrac12 (\log r)^2 \qquad k_3 = (r - 1) - \log r

$k_1$ and $k_3$ are unbiased; $k_3$ also has low variance and is never negative, which is why GRPO adopted it. For one token with $r = 0.5$ : $k_1 = 0.69$ , $k_2 = 0.24$ , $k_3 = 0.19$ . With $r = 2$ : $k_1 = -0.69$ (a negative "distance"), $k_3 = 0.31$ .

Where the estimate goes also matters. Added to the reward, $k_1$ gives the correct gradient. Used directly as a loss term, as GRPO does with $k_3$ , the gradient is only approximately right, and $k_1$ as a loss has zero expected gradient. DeepSeek-V3.2 reweights $k_3$ by the current-to-sampling probability ratio to remove the bias.

Papers

Schulman, Approximating KL Divergence (blog, 2020)
Rethinking KL Regularization in RLHF (Oct 2025)