Training in low precision

Training uses low precision too, but it needs extra care, because gradients are tiny and updates are tinier. Mixed precision, loss scaling, BF16, FP8 with fine-grained scales, stochastic rounding, and the newest FP4 recipes.

Key idea: Mixed precision (low-precision compute, FP32 master weights), loss scaling, why BF16 won, FP8 training, stochastic rounding, 8-bit optimizer states, and low-precision RL rollouts
Real world: A bank account that only records whole pounds. Interest of 0.4p a day is rounded to zero every single day and never accumulates, though over a year it should be £1.46. Keep the running balance in pennies and post whole pounds to the statement.
In an LLM: A weight of 1.0 receiving an update of 0.0001 in BF16: the nearest BF16 numbers to 1.0001 are 1.0 and 1.0078, so the update rounds away to nothing, every step. That’s why the optimizer keeps an FP32 “master” copy of every weight, even when all the multiplies run in BF16 or FP8.

Why training is harder than inference

Inference only needs the forward pass to be roughly right. Training also needs:

Gradients, which are often tiny: many are smaller than $2^{-24}$ , the smallest number FP16 can store.
Updates, which are tinier still relative to the weights they change: learning rate × gradient might be $10^{-6}$ of the weight.
Accumulation over thousands of steps, so a small bias in every step adds up.

Each needs its own fix.

Mixed precision and master weights

The standard recipe (Micikevicius et al., 2017) is mixed precision:

Keep an FP32 master copy of the weights.
For each step, cast a 16-bit copy and run the forward and backward passes with it: the expensive matrix multiplies run on fast 16-bit tensor cores.
Apply the update to the FP32 master weights, where small changes aren’t rounded away.

example · the bank account

Interest of 0.4p a day, credited to an account that only stores whole pounds, is rounded to £0 every day, forever. A bank keeps the true balance to fractions of a penny and only rounds what it prints. FP32 master weights are the bank’s ledger; the 16-bit copy used for the forward pass is the printed statement.

example · an update that vanishes

In BF16, the numbers next to 1.0 are $1 - 2^{-8}$ and $1 + 2^{-7} = 1.0078125$ (BF16 has 7 mantissa bits, so the spacing just above 1 is $2^{-7}$ ). Adding an update of 0.0001 to a BF16 weight of 1.0 gives 1.0001, which rounds straight back to 1.0. The same update applied to an FP32 master weight (spacing $2^{-23} \approx 1.2 \times 10^{-7}$ near 1) is kept.

See it happen, a thousand times in a row:

update size1.0e-4

fp32 master copy (exact)bf16, stochastic roundingbf16, round to nearest

fp32 after 1,0001.1000

bf16 nearest1.0000

bf16 stochastic1.0938

Each update (1.0e-4) is less than half the gap to the next bf16 number, so round-to-nearest throws it away every time: the weight never moves. Stochastic rounding sometimes rounds up, with just the right probability, so on average it follows the exact fp32 path. This is why training keeps an fp32 master copy of the weights, or uses stochastic rounding.

One weight starts at 1.0 and receives the same tiny update 1,000 times, as it would from a small learning rate. Near 1.0, neighbouring bf16 numbers are 0.0078 apart.

Loss scaling, and why BF16 replaced FP16

In FP16, many gradients are too small to represent and become zero. The fix is loss scaling: multiply the loss by a large number such as $2^{16}$ before the backward pass. By the chain rule every gradient is multiplied by the same factor, which lifts them into FP16’s range. Before the update, divide them back in FP32. If the scale is too large, some gradients overflow to infinity; dynamic loss scaling detects that, skips the step and halves the scale.

Try it

loss scale2⁰

stored normallysubnormal: fewer bits of precisionflushed to zero, or overflow

become zero67.2%

subnormal32.6%

overflow to infinity0%

In FP16, 67% of these gradients are below 2⁻²⁵, half of FP16’s smallest number, and silently round to zero: those weights stop learning. Multiply the loss by a power of two and every gradient shifts right by the same amount.

Histogram of gradient sizes on a log scale: each step to the right is twice as large. Shaded bands show what the format can store.

BF16 made all of this unnecessary. It has the same 8 exponent bits as FP32, and so the same range: gradients don’t underflow and loss scaling isn’t needed. It has less precision than FP16 (7 mantissa bits instead of 10), but for training, range turned out to matter more. BF16 with FP32 master weights is today’s default for training LLMs.

FP8 training

FP8 halves memory traffic again and doubles tensor-core throughput over BF16, but with only 3 or 2 mantissa bits and a narrow range, one scale per tensor isn’t enough.

Per-tensor scaling with history (NVIDIA’s Transformer Engine): each tensor’s scale is set from the largest values seen over recent steps. Weights and activations use E4M3, gradients E5M2.
Fine-grained scaling (DeepSeek-V3): one scale per 1 × 128 tile of activations and per 128 × 128 block of weights, E4M3 everywhere, and partial sums periodically moved to FP32 inside the matrix multiply. DeepSeek-V3 (671B parameters) was trained this way; in validation runs on smaller models, the FP8 loss stayed within 0.25% of a BF16 baseline.

Master weights, optimizer states and sensitive operations (normalisation, softmax, the output layer) stay in higher precision.

FP4 training and stochastic rounding

With only 15 values, FP4 rounding errors are large, and if they always push the same way they bias training. Stochastic rounding fixes the bias: round up with probability equal to how close the value is to the upper level. On average the rounded value equals the true value; each individual rounding is noisier but unbiased.

Recent recipes combine this with other tricks:

MXFP4 training (Tseng et al., 2025): stochastic rounding plus random Hadamard transforms on the backward pass to spread outliers.
NVFP4 pretraining (NVIDIA, 2025): a 12B model trained on 10 trillion tokens with NVFP4 matrix multiplies, using two-dimensional block scaling for weights, random Hadamard transforms on some gradients, stochastic rounding for gradients, and a few layers in BF16.

Smaller optimizer states

Adam keeps two extra numbers per weight. In FP32 that’s 8 bytes per parameter, more than the BF16 weights themselves. 8-bit optimizers (bitsandbytes) store them in 8 bits with block-wise scales, cutting optimizer memory by about 4× with the same results. This is what makes full fine-tuning of mid-sized models possible on a single GPU.

Low precision in RL rollouts

RL for LLMs generates huge numbers of rollouts with a fast inference engine, then trains on them. Running the rollouts in FP8 (or even lower) is tempting because generation dominates the cost. But the trainer computes log-probabilities in BF16, and the two no longer agree exactly: the rollouts become slightly off-policy. Importance-sampling corrections handle the mismatch, or FP16 is used on both sides: see training–inference mismatch.

Where this shows up

The formats themselves: float formats, FP8, MXFP4 and NVFP4.
Training a model that will be quantized afterwards: PTQ vs QAT.
Ternary weights trained from scratch: 2 bits and below.

going deeper

Loss scaling

With loss $\mathcal{L}$ and scale $S$ , backpropagation computes $\nabla (S\mathcal{L}) = S \nabla \mathcal{L}$ in FP16. The update uses $\frac{1}{S} \cdot \operatorname{fp32}(S \nabla \mathcal{L})$ . Dynamic scaling: if any gradient is inf or NaN, skip the step and set $S \leftarrow S/2$ ; after $N$ (e.g. 2,000) clean steps, set $S \leftarrow 2S$ .

Stochastic rounding

For $x$ between adjacent representable values $a < x < b$ :

\operatorname{SR}(x) = \begin{cases} b & \text{with probability } \frac{x - a}{b - a} \\ a & \text{otherwise} \end{cases} \qquad \mathbb{E}[\operatorname{SR}(x)] = x

Round-to-nearest has lower variance but is biased whenever the same $x$ is rounded many times, which is exactly what happens to small, repeated updates.

In papers

Mixed Precision Training (Micikevicius et al., 2017).
Deep Learning with Limited Numerical Precision (Gupta et al., 2015): stochastic rounding for low-precision training.
FP8 Formats for Deep Learning (Micikevicius et al., 2022).
DeepSeek-V3 Technical Report (2024): fine-grained FP8 training.
8-bit Optimizers via Block-wise Quantization (Dettmers et al., 2021).
Training LLMs with MXFP4 (Tseng et al., 2025).
Pretraining Large Language Models with NVFP4 (NVIDIA, 2025).