Floating point: sign, exponent, mantissa

A float is scientific notation in binary. The exponent says roughly how big the number is; the mantissa says exactly where it sits within that size. Click the bits and watch.

Key idea: fp32’s three fields: 1 sign bit, 8 exponent bits, 23 mantissa bits; value = sign × 2^exponent × 1.mantissa
Real world: Scientific notation: Avogadro’s number is 6.022 × 10²³. “10²³” says how big; “6.022” says exactly which number of that size. A float does the same with powers of two.
In an LLM: A weight like 0.0123 is stored in bf16 as about 1.578 × 2⁻⁷: exponent −7 and mantissa 1.57. Quantization formats differ mainly in how many bits they give each part.

Scientific notation, again

Writing very large and very small numbers in ordinary digits is clumsy: 602,200,000,000,000,000,000,000. Scientific notation splits a number into two parts:

6.022 \times 10^{23}

The exponent (23) says the scale: how many places to move the decimal point.
The significand (6.022) says the digits: exactly which number at that scale.

With just four significant digits you can write the mass of an electron or of a galaxy. You lose the digits beyond the fourth, but you never run out of room.

The same idea in binary

decimal scientific notation (powers of 10)

1 × 10^-1

digits between 1 and 10

binary scientific notation (powers of 2)

1.6 × 2^-4

significand between 1 and 2, so it always starts “1.”

In binary the significand is always 1-point-something, so the leading 1 is never stored: fp32 keeps only the part after the point (the mantissa) and the power of two (the exponent). For 0.1 that is 1.6 and -4.

The same number written two ways. Amber is the exponent (the rough size), blue is the significand (the exact digits). A float stores exactly the binary version.

A floating-point number does exactly this with base 2. A 32-bit float (fp32) has three fields:

Field	Bits	Job
Sign	1	0 for positive, 1 for negative
Exponent	8	which power of two: the number is between $2^e$ and $2^{e+1}$
Mantissa (or fraction)	23	where exactly the number sits between those two powers

\text{value} = (-1)^{\text{sign}} \times 2^{\,\text{exponent} - 127} \times 1.\text{mantissa}

Two details make this work:

The bias. The 8 exponent bits store a number from 0 to 255. Subtracting 127 turns that into exponents from −126 to +127 (the two extreme patterns are reserved), so tiny and huge numbers are both possible.
The hidden 1. In binary scientific notation the leading digit is always 1 (just as the first digit in decimal scientific notation is never 0), so it isn’t stored. The 23 mantissa bits are the digits after the binary point, and you get 24 bits of precision for the price of 23.

Try it

number

sign

exponent − 127

128

mantissa: where in between

⅛

1⁄16

2⁻⁵

2⁻⁶

…

putting the three parts together

+1×2^-4×1.6=0.10000000149011612

exponent field 123 − 127 = -4 · mantissa 1.1001100110… in binary

the exponent picks a range; the mantissa picks a point inside it

2^-4 = 0.062560% of the way2^-3 = 0.125

stored value0.10000000149011612

you typed0.1

rounding error1.49e-9

0.1 has no exact binary form (like 1/3 in decimal), so fp32 stores the nearest pattern. Press × 2: the exponent goes up by one and the mantissa bits don’t change.

Red is the sign, amber the exponent, blue the mantissa. The small numbers under the bits are what each bit is worth. Click any bit, or type a number and see which pattern stores it.

Things to try:

Start from 1 and press × 2 a few times. The exponent goes up by one each time, and the mantissa bits don’t change at all. ÷ 2 does the reverse.
Look at the small numbers under the bits. Mantissa bits are worth ½, ¼, ⅛, …: add up the lit ones, put “1 +” in front, and you get the blue number in “putting the three parts together”.
Type 0.1. It can’t be stored exactly, for the same reason 1/3 can’t be written exactly in decimal. fp32 stores 0.100000001490116…
Switch to bf16 and type 0.1 again: only 7 mantissa bits remain, and the error grows to about 0.0001.
Try 65504 in fp16, then press × 2. Its 5 exponent bits run out, and the number becomes infinity.
Flip the last mantissa bit of any number and see how little it changes; flip the top exponent bit and see how much.

Decoding one by hand

example · 5.5 in fp32

5.5 is positive, so the sign bit is 0.
Find the power of two just below it: $2^2 = 4$ . So $5.5 = 1.375 \times 2^2$ .
Exponent: $2 + 127 = 129$ , which is 10000001 in binary.
Mantissa: the part after “1.” is 0.375 = 0.25 + 0.125, i.e. binary .011. So the 23 mantissa bits are 0110000…0.

Reading it back: $(+1) \times 2^{129 - 127} \times 1.375 = 4 \times 1.375 = 5.5$ . Exact, because 5.5 is a sum of powers of two.

example · a weight in bf16

A weight of 0.0123 sits between $2^{-7} = 0.0078$ and $2^{-6} = 0.0156$ , so its exponent is −7 and $0.0123 = 1.574 \times 2^{-7}$ . bf16 has only 7 mantissa bits, so 1.574 must be rounded to a multiple of $1/128$ : $1.578125$ . The stored weight is $1.578125 \times 2^{-7} = 0.012329…$ , off by about 0.24%. That rounding is already a mild quantization; every model trained in bf16 lives with it.

Why the parts behave so differently

The exponent and mantissa do very different jobs, and that explains most of this course:

Exponent bits buy range. Each extra exponent bit roughly squares the largest and smallest numbers you can write. fp32’s 8 exponent bits reach about $3.4 \times 10^{38}$ .
Mantissa bits buy precision. Each extra mantissa bit halves the spacing between neighbouring numbers. With 23 bits, neighbours differ by about one part in 8 million; with bf16’s 7 bits, about one part in 128.

The next page shows what that spacing looks like on a number line: where floats are dense and sparse.

Where this shows up

The spacing of representable values: where floats are dense and sparse.
How fp16, bf16, fp8 and fp4 split their bits: fp32, fp16, bf16, fp8, fp4.
Microscaling formats share one exponent across a block of values: FP8, MXFP4 and NVFP4.

going deeper

IEEE 754

For a format with $E$ exponent bits, $M$ mantissa bits and bias $B = 2^{E-1} - 1$ , a stored exponent field $e$ and mantissa field $m$ give:

v = \begin{cases} (-1)^s \, 2^{\,e - B} \left(1 + \dfrac{m}{2^M}\right) & 0 < e < 2^E - 1 \quad \text{(normal)} \\[2ex] (-1)^s \, 2^{\,1 - B} \, \dfrac{m}{2^M} & e = 0 \quad \text{(subnormal, includes } \pm 0) \\[2ex] \pm\infty \text{ if } m = 0, \text{ else NaN} & e = 2^E - 1 \end{cases}

For fp32, $E = 8$ , $M = 23$ , $B = 127$ : the largest value is $(2 - 2^{-23}) \times 2^{127} \approx 3.40 \times 10^{38}$ , the smallest normal $2^{-126} \approx 1.18 \times 10^{-38}$ , and the smallest subnormal $2^{-149} \approx 1.4 \times 10^{-45}$ .

Machine epsilon, the gap between 1 and the next float, is $2^{-M}$ : about $1.19 \times 10^{-7}$ for fp32 and $7.8 \times 10^{-3}$ for bf16. Rounding to nearest gives a relative error of at most half of that.

Rounding

The default rounding mode is round to nearest, ties to even: when a value is exactly halfway between two floats, pick the one whose last mantissa bit is 0. It avoids the slow upward drift that “round half up” causes when millions of values are summed. Some low-precision training recipes instead use stochastic rounding, which rounds up with probability proportional to the distance, so the expected value is preserved.