Logarithms

Logarithms turn multiplying into adding, and tiny probabilities into ordinary negative numbers. That's why ML works with log-probabilities almost everywhere.

Key idea: The logarithm, and why models use log-probabilities
Real world: Sound volume in decibels. Every +10 dB is ten times the power. A logarithmic scale fits a whisper and a jet engine on the same ruler.
In an LLM: A 1,000-token answer can have probability around 10⁻⁵⁰. Its log-probability is about −115: an ordinary number you can add, compare and take gradients of.

What a logarithm is

The logarithm answers "what power do I raise the base to, to get this number?"

$\log_{10} 1000 = 3$ because $10^3 = 1000$ .
$\log_2 8 = 3$ because $2^3 = 8$ .
The natural log, $\ln$ , uses base $e \approx 2.718$ . It's the default in ML.

The one property that matters most:

\log(a \times b) = \log a + \log b

Multiplying turns into adding.

Logs of probabilities

Probabilities are between 0 and 1, so their logs are zero or negative:

Probability	ln(probability)
1	0
0.5	−0.69
0.1	−2.30
0.01	−4.61
0.001	−6.91

Each time the probability gets 10 times smaller, the log drops by the same amount, 2.30. Certain is 0; impossible is minus infinity.

example · decibels

A quiet room is about 30 dB and a busy street about 70 dB. The difference of 40 dB means the street carries $10^4 = 10{,}000$ times the sound power. Adding 40 on the log scale is the same as multiplying by 10,000 on the ordinary one.

example · the log-probability of an answer

The model writes three tokens with probabilities 0.5, 0.8 and 0.9.

Probability of the answer: $0.5 \times 0.8 \times 0.9 = 0.36$
Log-probability: $-0.69 - 0.22 - 0.11 = -1.02$ , which is $\ln 0.36$

Same information, but the second is a sum, one term per token. Training code works with these per-token log-probabilities directly.

Try it

tokens n50probability per token0.900

probability of the answer5.15 × 10^-3

log-probability-5.27

per-token log-prob-0.105

Probability (multiply)

Log-probability (add)

probabilitylog-probability

Probability multiplies, log-probability adds

0.900 × 0.900 × … (50 times) = 5.15 × 10^-3

-0.105 + -0.105 + … (50 times) = -5.27

The probability curve dives to almost 0 within a few dozen tokens and then tells you nothing. The log-probability is a straight line: every token adds the same amount. Push n toward 1,000 with a low per-token probability.

An answer of n tokens, each chosen with the same probability. How likely is the whole answer?

The two charts show the same answer both ways. The probability crashes to almost 0 within a few dozen tokens, and below a low enough per-token probability it drops under the smallest number a computer can store (the red line) and becomes exactly 0. The log-probability is a straight line that never runs out of room.

Why ML uses logs everywhere

No underflow. Products of thousands of probabilities become too small for a computer to store. Sums of logs don't.
Sums are easy to work with. The loss of a whole answer is the sum of per-token losses, and gradients of sums are sums of gradients.
Ratios become differences. $\log \frac{a}{b} = \log a - \log b$ . Every probability ratio in RL (PPO, KL, DPO) is computed as a difference of log-probabilities.

Where this shows up

Policy-gradient updates push up $\log \pi$ : policy gradient.
Ratios are computed as exp(logp_new - logp_old): PPO, importance sampling.
Every loss on the next pages is a log: cross-entropy.

going deeper

Rules

\log(ab) = \log a + \log b \qquad \log \frac{a}{b} = \log a - \log b \qquad \log a^n = n \log a \qquad \log_2 x = \frac{\ln x}{\ln 2}