Where floats are dense and sparse

Floats are not evenly spaced. They crowd near zero and spread out as numbers grow, so the relative error stays about the same everywhere. That one fact explains why floats suit neural networks.

Key idea: Representable values, spacing that doubles with each power of two, relative vs absolute error, subnormals, infinity and NaN
Real world: A tape measure for a house is marked in millimetres; a map of a country is marked in kilometres. You need fine marks for small distances and coarse ones for big distances, and the relative precision is similar.
In an LLM: A weight of 0.001 and an activation of 300 can both be stored in bf16 with about 0.4% error. With evenly spaced integers, one of them would be either rounded to zero or clipped.

Only some numbers exist

A format with $n$ bits can store at most $2^n$ different numbers. Every other number has to be rounded to one of them. So the important question about a format is: which numbers can it store, and how far apart are they?

For integers, the answer is simple: evenly spaced, 1 apart. For floats, the answer is the picture on this page.

Try it

The widget uses a made-up tiny float so you can see every single value it can store. Real formats work the same way with many more ticks.

your number8.400

values > 027

largest14

stored as8.000

relative error4.8%

Each grey band runs from one power of two to the next and holds 4 evenly spaced ticks, so every band’s gap is twice the one before. Ticks crowd near zero and spread out further away. Switch to the log scale: every band becomes the same width.

A made-up 6-bit float: 1 sign bit and 5 bits shared between exponent and mantissa (E3M2: 3 exponent, 2 mantissa). Every tick is a number it can store exactly; anything else is rounded to the nearest tick. Drag on the line or use the slider.

What the picture shows

Each exponent value is a band. Within one band, say between 2 and 4, the mantissa bits split the range into equal steps. With 2 mantissa bits there are 4 steps: 2, 2.5, 3, 3.5.

Each band is twice as wide as the last. The next band, 4 to 8, also gets 4 steps, so they are twice as far apart: 4, 5, 6, 7. Then 8, 10, 12, 14. The spacing doubles every time the numbers double.

So the relative error is roughly constant. Rounding 2.6 to 2.5 is a 4% error; rounding 13 to 12 or 14 is about 4–8%. Small numbers get small absolute errors and large numbers get large ones, but as a fraction of the number, the error is similar. On the log scale, every band looks the same width.

example · the tape measure and the map

You measure a shelf to the millimetre and a road trip to the kilometre. Both measurements are about equally precise relative to what you’re measuring (a few parts in a thousand), even though one error is a million times bigger than the other. Floats are a measuring system that automatically switches from millimetres to kilometres as the number grows.

example · weights and activations

Inside an LLM, typical weights are around 0.01–0.1, while some activations reach the hundreds. In bf16, both are stored to about 3 significant figures, because each is placed in its own band. If you instead used one evenly spaced grid fine enough for 0.01, it would need far too many steps to reach 300; one coarse enough for 300 would round every weight to zero. Integer quantization has exactly this problem, which is why it needs scales and groups.

The edges of the line

Near zero: subnormals. The smallest exponent would leave a gap between zero and the first band. Instead, the all-zeros exponent is used for subnormal numbers: evenly spaced values filling that gap, without the hidden leading 1. They are less precise but stop small numbers from suddenly dropping to zero (purple ticks in the widget).

Too small: underflow. Below the smallest subnormal, a number rounds to zero. In fp16 that happens below about $6 \times 10^{-8}$ , which used to wipe out small gradients during training.

Too big: overflow. Above the largest value, IEEE formats give infinity, and arithmetic with infinity soon produces NaN (“not a number”, e.g. $\infty - \infty$ ). One NaN anywhere in a training step usually spreads to everything. fp16 overflows above 65,504, a number that large attention logits or activations can exceed.

Why not just use integers?

A trained network’s numbers span several orders of magnitude, and gradients during training span many more. Floats handle that without any bookkeeping. Integers are cheaper in hardware, but need an extra scale per block of numbers to say where the grid sits. Quantization is mostly the craft of choosing those scales well, starting with what quantization is.

Where this shows up

The bit layout: floating point.
How real formats choose their trade-off: fp32, fp16, bf16, fp8, fp4.
Loss scaling to avoid underflow: training in low precision.

going deeper

Spacing and error

In the band $[2^k, 2^{k+1})$ , a format with $M$ mantissa bits has spacing

\text{ulp}(x) = 2^{\,k - M}

(“unit in the last place”). Rounding to nearest gives an absolute error of at most half that, so a relative error of at most

\frac{|\hat x - x|}{|x|} \le 2^{-(M+1)} = \frac{\epsilon}{2}

where $\epsilon = 2^{-M}$ is machine epsilon: $1.19 \times 10^{-7}$ for fp32, $9.8 \times 10^{-4}$ for fp16 and $7.8 \times 10^{-3}$ for bf16. So the worst-case relative rounding error is half that: about $6 \times 10^{-8}$ , $4.9 \times 10^{-4}$ and $3.9 \times 10^{-3}$ .

Dynamic range

The ratio between the largest and smallest normal values, in powers of two, is roughly $2^{E}$ bands:

\log_2 \frac{x_{\max}}{x_{\min,\text{normal}}} \approx 2^{E} - 2

fp16 ( $E = 5$ ) covers about 30 binades, fp32 and bf16 ( $E = 8$ ) about 254. Adding one exponent bit roughly doubles the number of binades; adding one mantissa bit halves every gap.

Summation

Adding a small number to a large one can lose it entirely: in bf16, $256 + 1 = 256$ , because the spacing at 256 is 2. This is why reductions (sums over thousands of products in a matrix multiply, or accumulating gradients) are done in fp32 even when the inputs are bf16, fp8 or int8.