Reward models and RLHF

When no program can score an answer, learn a scoring function from human comparisons, then run RL against it. This is how the first chat assistants were trained.

Key idea: A reward model trained on pairwise preferences (Bradley–Terry), and the RLHF pipeline
Real world: Chess ratings. Nobody assigns a player a score directly; Elo ratings are inferred from who beat whom, and the rating gap predicts the next result.
In an LLM: People see two answers to the same prompt and pick the better one. A reward model learns to give the preferred answer a higher score, and RL then trains the chat model to write answers that score highly.

Why comparisons

Ask ten people to rate an answer from 1 to 10 and you get ten different scales. Ask them which of two answers is better and they mostly agree. So preference data is collected as pairs: a prompt, a chosen answer, and a rejected answer.

A reward model turns those pairs into a score. It's usually a copy of the language model with its last layer swapped for one that outputs a single number. Training uses the Bradley–Terry model: the chance that A is preferred to B is the of the score gap.

Try it

score of A+1.5score of B+0.5

moves both scores, keeps the gap

Chance a person prefers A = sigmoid(score A − score B)

A73%

B27%

gap+1.0

loss if people chose A0.31

loss if people chose B1.31

A gap of +1.0 predicts 73% for A. If people chose B instead, the loss is 1.31 and training pushes B’s score up and A’s down. Press Both +0.5: the bars don’t move.

A reward model gives each answer a score. Only the difference between two scores matters.

example · Elo ratings

Elo is the same idea for chess. A player rated 400 points above another is expected to score

\frac{1}{1 + 10^{-400/400}} = \frac{1}{1.1} \approx 91\%

against them. After each game, both ratings move toward whatever would have predicted the result better. Nobody ever rates a player in isolation.

example · one reward-model training pair

Prompt: "How do I reverse a list in Python?" A labeller preferred answer A (shows my_list[::-1] and reversed(), explains the difference) over answer B (only says "use a loop").

The reward model currently scores A at 2.1 and B at 0.9.

Predicted chance A is preferred: $\sigma(2.1 - 0.9) = \sigma(1.2) \approx 0.77$
Loss: $-\log 0.77 \approx 0.26$

The gradient raises A's score and lowers B's until the model is confident in the labellers' choice, across millions of such pairs.

The RLHF pipeline

RLHF (reinforcement learning from human feedback) is the three-stage recipe behind InstructGPT and early ChatGPT:

Stage	Data	What it produces
1. Supervised fine-tuning	Human-written example answers	A model that follows instructions roughly
2. Reward model	Human comparisons between model answers	A scorer for any answer
3. RL	Prompts only; the reward model scores the samples	A model tuned to write high-scoring answers

Stage 3 uses PPO with a KL penalty to the stage-1 model. Four models are in memory at once: the policy, the reference copy, the reward model and PPO's critic. Much of what came later, DPO and GRPO included, is about removing some of them.

Over-optimising the reward model

A reward model is a learned approximation of what people want, so it has blind spots. Optimise against it hard enough and the policy finds them. Gao et al. measured this with a large "gold" reward model standing in for humans: as the policy moves away from where it started, the reward model’s own score (the proxy) keeps rising, while the gold score rises, peaks and then falls. Larger reward models push the peak further out but don't remove it. The KL page has a widget where you can watch the two curves part. This is the reason for the KL penalty, early stopping, and the move toward checkable rewards wherever possible.

going deeper

Bradley–Terry loss

P(y_w \succ y_l \mid x) = \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big) \qquad L(\phi) = -\mathbb{E}\big[\log \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big)\big]

Only score differences are identified, so reward models are often normalised to mean 0 on a reference set.

The RLHF reward

In PPO-based RLHF the reward model's score arrives at the last token, and the KL penalty is added per token:

r_t = -\beta \log \frac{\pi_\theta(o_t \mid s_t)}{\pi_\text{ref}(o_t \mid s_t)} + \mathbb{1}[t = T]\, r_\phi(x, y)

Papers

Training language models to follow instructions with human feedback (InstructGPT, 2022)
Scaling Laws for Reward Model Overoptimization (Gao, Schulman, Hilton, 2022)