PTLearning

fp32, fp16, bf16, fp8 and fp4

The same three fields, different bit budgets. bf16 kept fp32’s range and gave up precision; fp16 did the opposite. fp8 comes in two flavours, and fp4 can only say 15 different numbers.

Key idea
How each format splits its bits between exponent and mantissa, and what that costs in range and precision
Real world
A camera with a fixed number of pixels can capture a wide landscape (range) or zoom in on a face (detail), but not both. Each format picks a point on that trade-off.
In an LLM
Models are trained in bf16, served in bf16 or fp8, and squeezed into 4-bit formats for laptops and phones. Knowing what each one can and can’t represent explains which parts of a model break first.

The family

FormatSignExponentMantissaLargest valueRelative precisionUsed for
fp3218233.4×10383.4 \times 10^{38}~6×10−86 \times 10^{-8}master weights, optimizer states, sums
fp16151065,504~5×10−45 \times 10^{-4}older training and inference
bf161873.4×10383.4 \times 10^{38}~4×10−34 \times 10^{-3}training and serving today
fp8 E4M3143448~6%weights and activations
fp8 E5M215257,344~12%gradients
fp4 E2M11216~25%block formats like MXFP4, NVFP4

“E4M3” just means 4 exponent bits and 3 mantissa bits.

Try it

Move the slider to choose a number and watch which formats store it well, which round it coarsely, and which overflow.

fp3232b
stored: 0.70795 · error ≈ 0the default for training maths
bf1616b
stored: 0.707031 · error 0.13%fp32’s range, far less precision
fp1616b
stored: 0.708008 · error <0.01%more precision, tiny range
fp8 E5M28b
stored: 0.75 · error 5.9%range for gradients
fp8 E4M38b
stored: 0.6875 · error 2.9%precision for weights and activations
fp4 E2M14b
stored: 0.5 · error 29%only 15 distinct values
int88b
stored: 1 · error 41%evenly spaced whole numbers
int44b
stored: 1 · error 41%−8 to 7
10⁻¹⁰10⁻⁵110⁵10¹⁰
fp32 and bf16 go on to 3.4 × 10³⁸ →
Exponent bits buy range (how wide the bar is); mantissa bits buy precision (how small the error is). bf16 and fp16 spend the same 16 bits differently. Integer formats have no exponent: evenly spaced whole numbers, and nothing below 1.
Each row is one format. The coloured strip is its bit layout (red sign, amber exponent, blue mantissa; grey for integers). The bar shows the range of magnitudes it can hold, on a log scale.

bf16: range over precision

bf16 (“brain float”, from Google Brain) is fp32 with the bottom 16 mantissa bits chopped off. It keeps all 8 exponent bits, so it has exactly fp32’s range, but only 7 mantissa bits, so roughly 2–3 significant decimal digits.

fp16 spends the same 16 bits differently: 5 exponent bits and 10 mantissa bits. More precise, but its largest number is 65,504 and anything below about 6×10−86 \times 10^{-8} is zero.

For training, range turned out to matter more. Gradients and activations vary over many orders of magnitude; in fp16 they overflow or underflow unless you carefully rescale them (loss scaling). In bf16 they simply fit. So modern LLMs are trained in bf16, with sums and optimizer states in fp32.

example · a camera’s trade-off

A 12-megapixel camera can photograph a whole mountain range with each pixel covering a metre, or one face with each pixel covering a fraction of a millimetre. Same pixel budget, different trade-off. bf16 is the wide-angle lens; fp16 is the zoom.

example · an fp16 overflow

Some LLMs develop a few enormous activation values in specific channels, sometimes in the thousands, and attention logits can grow large too. Multiply two such numbers and you can exceed 65,504, giving infinity, then NaN, and training or inference silently produces garbage. Several open models released in bf16 produce NaNs when naively run in fp16 for exactly this reason. bf16 would need values beyond 103810^{38} to overflow.

fp8: two flavours

With only 8 bits, you can’t have both. The FP8 standard used by NVIDIA, AMD and Intel defines two:

  • E4M3: more precision, range up to 448. It gives up infinity entirely to gain one extra binade. Used for weights and activations in the forward pass.
  • E5M2: more range, up to 57,344, with IEEE infinity and NaN. Used for gradients, which vary more.

Because fp8’s range is small, every tensor (or block of values) gets its own scale factor so its values land in the representable region. That idea, a cheap format plus a scale, is the same one integer quantization uses: what quantization is.

fp4: fifteen numbers

E2M1 has 16 bit patterns. With a sign bit, those give ±{0,0.5,1,1.5,2,3,4,6}\pm\{0, 0.5, 1, 1.5, 2, 3, 4, 6\}: 15 distinct values (+0 and −0 are both zero). On its own that is useless for weights. With a shared scale per block of 16 or 32 values it becomes MXFP4 and NVFP4, the 4-bit formats of the newest GPUs: see FP8, MXFP4 and NVFP4.

Integers in the same picture

Integer formats (int8, int4) have no exponent at all: all their bits are mantissa, and every value is evenly spaced. In the widget, they store every whole number in range exactly and nothing in between. That makes integer hardware simple and fast, and it is why integer quantization depends entirely on a well-chosen scale.

Where this shows up

going deeper

Exact limits

FormatBiasMaxMin normalMin subnormalϵ=2−M\epsilon = 2^{-M}
fp321273.40×10383.40 \times 10^{38}1.18×10−381.18 \times 10^{-38}1.4×10−451.4 \times 10^{-45}1.19×10−71.19 \times 10^{-7}
fp161565,5046.10×10−56.10 \times 10^{-5}5.96×10−85.96 \times 10^{-8}9.77×10−49.77 \times 10^{-4}
bf161273.39×10383.39 \times 10^{38}1.18×10−381.18 \times 10^{-38}9.2×10−419.2 \times 10^{-41}7.81×10−37.81 \times 10^{-3}
E4M374482−62^{-6}2−92^{-9}0.1250.125
E5M21557,3442−142^{-14}2−162^{-16}0.250.25
E2M11610.50.50.5

E4M3 keeps the all-ones exponent for ordinary numbers and reserves only the single pattern S.1111.111 for NaN, which is why its maximum is 1.75×28=4481.75 \times 2^{8} = 448 rather than 240. Conversions into E4M3 usually saturate (clamp to ±448) instead of producing infinity.

In papers

Micikevicius et al., 2022 define the E4M3 and E5M2 formats and show that, with per-tensor scaling, FP8 training matches bf16 on models up to 175B parameters. DeepSeek-V3 was trained largely in FP8 with fine-grained (block-wise) scaling (DeepSeek-AI, 2024). The OCP Microscaling spec (Rouhani et al., 2023) defines MXFP8, MXFP6 and MXFP4 as blocks of 32 elements sharing a power-of-two scale.

Related

  • floating-point - A float is scientific notation in binary. The exponent says roughly how big the number is; the mantissa says exactly where it sits within that size. Click the bits and watch.
  • float-precision - Floats are not evenly spaced. They crowd near zero and spread out as numbers grow, so the relative error stays about the same everywhere. That one fact explains why floats suit neural networks.
  • fp8-fp4 - New GPUs multiply small floats directly. FP8 is now standard for serving and is used in training; 4-bit floats with a shared scale per small block (MXFP4, NVFP4) are the newest step, with hardware support on NVIDIA Blackwell.