NF4, double quantization and QLoRA

Weights are bell-shaped, so put the 16 levels of a 4-bit code where the weights actually are. NF4 does that; double quantization shrinks the scales too; QLoRA trains small adapters on top of a frozen NF4 model.

Key idea: Non-uniform levels at the quantiles of a normal distribution (NF4), quantizing the scales themselves (double quantization), and fine-tuning adapters on a 4-bit base (QLoRA)
Real world: Shoe sizes come in half sizes from 6 to 11, where most people are, and in whole sizes or not at all at the extremes. The steps are fine where the crowd is and coarse where nobody is.
In an LLM: QLoRA fine-tunes a 65B model on one 48 GB GPU. The base weights sit frozen in NF4, about 4.13 bits each with double quantization; only small LoRA adapters in 16-bit are trained, and the NF4 weights are turned back into bf16 on the fly for each matrix multiply.

Where should 16 levels go?

A 4-bit code has 16 patterns. Integer quantization spaces their values evenly between −max and +max. That’s the right choice if values are spread evenly. Neural network weights aren’t: they follow a bell curve, roughly normal, with most values near zero and a thin tail on each side.

Even spacing then wastes codes: the outer levels are almost never used, and the crowded middle, where most weights live, gets coarse steps. Better: put the levels closer together where weights are common and further apart where they’re rare.

NF4 (4-bit NormalFloat) does exactly that. Its 16 levels are placed at quantiles of a normal distribution: points that cut the bell curve into slices of roughly equal probability, rescaled to the range −1 to 1, with one level exactly at 0 so that zero stays exact.

Try it

share of weights that land on each level

Dashed line: 6.25%, the share each level would get if all 16 were used equally. Entropy 3.88 of a possible 4.00 bits.

INT4 error1.68

FP4 error1.67

NF4 error ← shown1.26

NF4 places each level at a quantile of the normal distribution, so the levels are used much more evenly than INT4’s and the error is 25% lower than INT4 with the same 4 bits. The levels at exactly ±1 are rare because only the largest weight in each block sits there; the QLoRA paper calls NF4 information-theoretically optimal for normally distributed weights. (MSE shown × 1,000.)

double quantization (quantize the block scales too)

bits per weight, scales included4.127

65B model33.5 GB

saved by double quantization3.0 GB

Weights in blocks of 64, each divided by its block’s largest magnitude so they fall in [−1, 1]. The bell is how often each value occurs; vertical lines are the levels a 4-bit code can choose.

example · shoe sizes

A shoe shop that stocked every size from 1 to 20 in equal numbers would be full of unsold extremes and always out of size 9. Sizes are instead offered in half steps where most feet are and sparsely at the ends, and the stock of each is matched to demand. NF4’s levels are half sizes for weights: dense in the middle of the bell, sparse in the tails, so each is used much more evenly than INT4’s evenly spaced levels.

example · one block of 64 weights

bitsandbytes cuts each weight matrix into blocks of 64. For one block whose largest magnitude is 0.042:

Divide every weight by 0.042, so they lie in −1 to 1.
Replace each with the index of the nearest NF4 level. A weight of 0.0143 becomes $0.0143 / 0.042 = 0.340$ ; the nearest level is 0.3379, the 12th of the 16, so the stored code is 1011 (11 counting from 0).
Store the scale 0.042 once for the block.

To use the weight, look up level 11 (0.3379) and multiply by 0.042: 0.01419. The error is 0.0001, under 1%. A weight of similar size that falls halfway between two levels does worse, about 13%; the average over a whole block is what the widget’s error measures.

Double quantization

Each block of 64 weights needs a scale. Stored in fp32 that’s 32 bits per 64 weights: 0.5 extra bits per weight, an eighth of the 4-bit budget. QLoRA’s fix is to quantize the scales as well:

Group the scales of 256 blocks together, subtract their mean, and store each in 8 bits (FP8), with one fp32 scale for the group.
Cost per weight: $8/64 + 32/(64 \times 256) \approx 0.127$ bits instead of 0.5.

That saves about 0.37 bits per weight, roughly 3 GB on a 65B model. The same “scales of scales” idea appears in GGUF’s k-quants and in NVFP4: see GGUF and FP8, MXFP4 and NVFP4.

QLoRA: training on top of a 4-bit model

Fine-tuning a model normally needs its weights in 16-bit plus optimizer states, several times the model’s size. QLoRA (Dettmers et al., 2023) avoids almost all of that:

Load the base model with every linear layer in NF4 and double quantization. It is frozen.
Add LoRA adapters: two small low-rank matrices per layer, in bf16. Only these are trained.
In every forward and backward pass, each NF4 block is dequantized to bf16 on the fly, multiplied, and thrown away. The storage type is NF4; the compute type is bf16.
Gradients flow through the frozen 4-bit weights to the adapters.

The paper fine-tuned a 65B model on a single 48 GB GPU and matched 16-bit fine-tuning quality on their benchmarks. QLoRA made fine-tuning large models possible on one consumer or workstation GPU, and it’s still one of the most common ways to fine-tune.

full fine-tuning (bf16 + Adam)1040 GB

→ off the scale

LoRA on a bf16 base140 GB

QLoRA: NF4 base + adapters43 GB

24 GB48 GB80 GB

bf16 weightsNF4 weightsgradientsoptimizer statesLoRA adapters, all included

Full fine-tuning needs about 1 TB: a rack of GPUs. LoRA freezes the base model, so its gradients and optimizer states shrink to the adapters, but the bf16 weights alone are 130 GB. QLoRA stores the frozen base in NF4: about 34 GB, which with the adapters fits on a single 48 GB GPU. That is the trick that made fine-tuning a 65B model possible on one card.

Memory to fine-tune a 65-billion-parameter model, before activations (which gradient checkpointing keeps to a few GB). Adapters are rank 64 on every linear layer, about 0.8 billion trainable parameters.

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch
 
bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",           # NF4 levels instead of evenly spaced INT4
    bnb_4bit_use_double_quant=True,      # quantize the block scales too
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B", quantization_config=bnb)
# then attach LoRA adapters with peft and train as usual

Where this shows up

The bell curve and quantiles: common distributions.
Fine-tuning while quantized, more generally: PTQ vs QAT.
Two-level scales in other formats: GGUF, FP8, MXFP4 and NVFP4.
llama.cpp’s IQ4_NL also uses a fixed table of non-uniform levels: GGUF.

going deeper

Constructing NF4

For a $k$ -bit NormalFloat, the levels are built from the quantile function $Q_X$ of a standard normal. The paper averages neighbouring quantiles:

q_i = \frac12 \left( Q_X\!\Big(\frac{i}{2^k + 1}\Big) + Q_X\!\Big(\frac{i + 1}{2^k + 1}\Big) \right)

then normalises to $[-1, 1]$ . To get an exact zero, NF4 builds the levels asymmetrically: $2^{k-1}$ levels for the positive half and $2^{k-1} - 1$ for the negative half, sharing one zero. The 16 values used by bitsandbytes are

\begin{aligned} &-1,\ -0.6962,\ -0.5251,\ -0.3949,\ -0.2844,\ -0.1848,\ -0.0911,\ 0, \\ &0.0796,\ 0.1609,\ 0.2461,\ 0.3379,\ 0.4407,\ 0.5626,\ 0.7230,\ 1 \end{aligned}

Because weights are normalised by their block’s absmax first, the levels are optimal for “normal data divided by its maximum”, which is close to but not exactly a normal distribution: the levels at ±1 are rarely used.

Memory with double quantization

With block size $B_1 = 64$ for weights and $B_2 = 256$ for the scales:

\text{bits per weight} = 4 + \frac{8}{B_1} + \frac{32}{B_1 B_2} = 4 + 0.125 + 0.00195 \approx 4.127

In papers

QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023): NF4, double quantization and paged optimizers.
LoftQ (Li et al., 2023) initialises the LoRA adapters so that base plus adapters starts close to the original 16-bit weights, reducing the gap QLoRA leaves at low bit widths.