The sigmoid and log-odds

An S-shaped curve that turns any score into a probability between 0 and 1. It's how a score gap becomes "chance A beats B".

Key idea: Odds, log-odds, the sigmoid, and the logistic loss
Real world: Chess ratings. A player rated 200 points higher is expected to win about 76% of the time; 400 points higher, about 91%. A rating gap becomes a win probability through an S-shaped curve.
In an LLM: A reward model gives two answers scores. The chance a person prefers the first is the sigmoid of the score gap. Reward-model training and DPO are both built on this.

Odds and log-odds

Odds compare the chance something happens with the chance it doesn't: probability 0.8 is odds of 0.8 : 0.2, or 4 to 1.

The log-odds is the log of that ratio: $\ln 4 \approx 1.39$ . Log-odds run from minus infinity (impossible) through 0 (even odds) to plus infinity (certain), which makes them a natural thing for a model to output as a plain score.

The sigmoid

The sigmoid turns log-odds back into a probability:

\sigma(x) = \frac{1}{1 + e^{-x}}

$\sigma(0) = 0.5$ : a gap of zero is a coin flip.
Large positive $x$ : close to 1. Large negative: close to 0.
Each +1 in $x$ multiplies the odds by $e \approx 2.7$ .

Try it

score gap (log-odds)+1.2

P(A wins)76.9%

odds A : B3.32 : 1

loss −ln P(A)0.26

push σ(−gap)0.23

P(A wins) = σ(gap)push from the loss = σ(−gap)

A gap of 0 means a coin flip and a push of 0.5. Each +1 multiplies the odds by e ≈ 2.7. Drag right: as A pulls ahead, the dashed push curve falls toward 0.

The score gap between answer A and answer B goes in; the probability that A wins comes out. The dashed curve is how hard the training loss still pushes when people preferred A.

example · chess ratings

Elo uses the same curve with base 10 and a scale of 400 points:

P(\text{win}) = \frac{1}{1 + 10^{-\text{gap}/400}}

A gap of 200 gives $\frac{1}{1 + 10^{-0.5}} = \frac{1}{1.32} \approx 76\%$ .

example · a reward model's comparison

A reward model scores answer A at 2.1 and answer B at 0.9. The gap is 1.2:

P(\text{A preferred}) = \sigma(1.2) \approx 0.77

If people did prefer A, the training loss is $-\ln 0.77 \approx 0.26$ . Training adjusts the scores to make preferred answers' gaps larger.

Why the loss stops pushing

The loss for a correct comparison is $-\log \sigma(\text{gap})$ . Its gradient is proportional to $\sigma(-\text{gap})$ , the probability the model still gives the wrong answer. At gap 0 that's 0.5; at gap 3 it's 0.05: the dashed curve in the widget. Pairs the model already gets right barely affect training, so learning concentrates on the ones it gets wrong.

Softmax with two options

A sigmoid is a softmax over two choices: with logits $a$ and $b$ , softmax gives the first $\frac{e^a}{e^a + e^b} = \sigma(a - b)$ . Only the gap matters, just as with softmax.

Where this shows up

Bradley–Terry reward models: reward models and RLHF.
DPO's loss is a sigmoid of the implicit reward gap: DPO.

going deeper

Definitions

\sigma(x) = \frac{1}{1 + e^{-x}} \qquad \sigma^{-1}(p) = \ln \frac{p}{1 - p} \qquad \sigma'(x) = \sigma(x)\big(1 - \sigma(x)\big) \qquad \frac{d}{dx}\big[-\log \sigma(x)\big] = -\sigma(-x)