DPO and preference optimisation

Skip the reward model and the RL loop. Train directly on preference pairs using a loss that already contains the best policy under a KL penalty.

Key idea: Direct Preference Optimisation: the policy is its own reward model
Real world: Learning to cook for a friend. You could build a scoring system for dishes and then optimise against it, or simply make more of what they picked in each taste test and less of what they turned down.
In an LLM: The same pairs used to train a reward model (prompt, chosen answer, rejected answer) train the chat model directly with one supervised-style loss. No sampling, no critic, no reward model.

The trick

The KL page showed that the best policy under a KL penalty has a simple exact formula: $\pi^*(y) \propto \pi_\text{ref}(y)\, e^{r(y)/\beta}$ . Turn it around and the reward can be read off the policy:

r(y) = \beta \log \frac{\pi(y)}{\pi_\text{ref}(y)} + \text{a constant per prompt}

Plug that into the Bradley–Terry loss and the constant cancels, because only reward differences matter. What's left is a loss on the policy alone: make the chosen answer's log-ratio bigger than the rejected answer's.

Try it

β0.10Shared drift: both answers also lose probability each step

log-ratio, chosen+0.0log-ratio, rejected+0.0

step 0

Each answer’s log-ratio. Arrows: how hard the next step pushes it.

implicit reward gap β·(chosen − rejected)+0.00

loss0.693

still pushing50%

Little has changed yet. Press Take a DPO step: the chosen answer is pushed up and the rejected one down.

log-ratio = log π(answer) − log π_ref(answer): how much more (or less) likely training has made this answer than the reference model. 0 means unchanged.

example · one DPO step by hand

β = 0.1. During training, the chosen answer's log-probability rose from −40 to −38 (log-ratio +2), and the rejected answer's fell from −35 to −39 (log-ratio −4).

Implicit reward gap: $0.1 \times (2 - (-4)) = 0.6$
Loss: $-\log \sigma(0.6) \approx 0.44$
The pair still pushes at $\sigma(-0.6) \approx 35\%$ strength: it has learned some of the preference, not all of it.

Set the widget's sliders to +2 and −4 to check these numbers.

example · the same idea in the real world

A taste test with 30 pairs of dishes doesn't tell you a score for any dish. It tells you that your friend liked the lemon chicken more than the curry, and the curry more than the stew. DPO uses exactly this kind of information: never "how good", only "better than".

Variants you'll see in papers

Method	What changes
IPO	Squared loss on the gap instead of log-sigmoid, so it stops pushing once the gap reaches a target. Less overfitting to noisy pairs.
KTO	Needs only "good" or "bad" labels on single answers, not pairs.
SimPO	No reference model: uses the length-normalised log-probability plus a margin.
ORPO	Folds a preference term into the ordinary fine-tuning loss, in one stage.
Online / iterative DPO	Regenerates pairs from the current policy each round, so the data stays on-policy.

Offline vs online

Standard DPO trains on a fixed dataset of answers that some other model wrote. That data is off-policy, and it never shows the model its own new mistakes. Two practical consequences:

Both answers can become less likely. DPO only cares about the gap, so the chosen answer's probability can fall too, as long as the rejected one falls faster. Tick "shared drift" in the widget and take a few steps to see it.
It doesn't explore. It can only prefer among answers someone already wrote.

So DPO remains a cheap, stable stage for style and helpfulness: Olmo 3, for example, runs SFT, then DPO, then RL with verifiable rewards. But the big gains in reasoning since 2024 have come from online RL such as GRPO.

going deeper

The DPO loss

L_\text{DPO} = -\mathbb{E}_{(x, y_w, y_l)}\Big[\log \sigma\Big(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_\text{ref}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_\text{ref}(y_l \mid x)}\Big)\Big]

Its gradient weights each pair by $\sigma(-\text{margin})$ , which is the "still pushing" number in the widget: pairs the model already gets right fade out.

Papers

Direct Preference Optimization (Rafailov et al., 2023)
IPO, KTO, SimPO, ORPO