2 bits and below: codebooks and ternary weights

Below about 3 bits, rounding each weight on its own falls apart. Two ways down: store groups of weights as entries in a learned codebook (AQLM, QuIP#, llama.cpp’s i-quants), or train the model from the start to use only −1, 0 and +1 (BitNet b1.58).

Key idea: Vector quantization with codebooks (additive and lattice codebooks), and ternary weights trained from scratch (BitNet b1.58)
Real world: A GIF image can only use 256 colours. A fixed palette (every combination of a few reds, greens and blues) wastes most of its slots on colours the picture never uses; a palette chosen for that picture, a codebook, looks far better with the same 8 bits per pixel.
In an LLM: AQLM stores Llama 2 70B at about 2 bits per weight by encoding each group of 8 weights as one index into a learned codebook of 65,536 vectors. BitNet b1.58 takes another route: a 2B-parameter model trained from scratch with every weight equal to −1, 0 or +1, whose weights take about 0.4 GB.

Why rounding fails at 2 bits

At 2 bits a weight has 4 possible values. However well you choose the scale, rounding each weight to one of 4 levels throws away most of what it held, and the errors pile up through dozens of layers. The GGUF scoreboard shows the result: perplexity is nearly unchanged down to about 4 bits per weight, then climbs steeply below 3.

Two observations open a way down:

Weights aren’t independent. Neighbouring weights in a group have joint structure: some combinations are common, others never happen. Rounding each one separately can’t exploit that; quantizing them together can.
Training can adapt. A model that is trained knowing its weights will be ternary learns to work with them. A model quantized after training never got that chance.

Vector quantization: codebooks

Vector quantization treats a group of $g$ weights as a single point in $g$ -dimensional space, and stores it as the index of the nearest entry in a codebook: a list of representative points. With a codebook of $2^b$ entries, each group costs $b$ bits, $b/g$ bits per weight.

The point of a codebook is that it can put its entries where the data is. A grid of independently rounded weights spends many of its points on combinations that never occur.

Try it

grid points: every combination of per-weight levelsunused code

codes per pair4

unused / rarely used codes0 / 0

error, each weight alone0.355

error, codebook0.207

Rounding each weight separately gives a rectangular grid of 4 points. All are used, but the off-diagonal ones (top left, bottom right) sit away from where the pairs are, so their pairs are rounded far. At 2 bits, whole codes go unused.

Each dot is a pair of neighbouring weights (w₁, w₂). Both methods spend the same bits on each pair; dots take the colour of the code they are stored as.

example · a GIF palette

A fixed “3-3-2” palette gives 8 shades of red, 8 of green and 4 of blue: 256 colours spread evenly over the whole colour cube. A photo of a forest uses mostly greens and browns, so most of those 256 colours are wasted and the greens are banded. An adaptive palette (computed with the same k-means algorithm the widget uses) puts most of its 256 colours among the greens and browns. Same 8 bits per pixel, much better picture.

example · AQLM at 2 bits

AQLM splits each weight row into groups of 8 weights. A group is stored as the sum of codewords from one or more codebooks (additive quantization). With one codebook of $2^{16} = 65{,}536$ vectors of length 8, each group costs a 16-bit index: $16 / 8 = 2$ bits per weight, plus a small cost for the codebook and scales. Decoding a group means one table lookup. The codebooks are learned per layer to minimise the layer’s output error on calibration data, and then the whole model is fine-tuned briefly to recover more quality.

Lattice codebooks: QuIP# and llama.cpp’s i-quants

A learned codebook of 65,536 entries per layer is big and slow to search. QuIP# instead uses a codebook built from the E8 lattice, a famously efficient packing of points in 8 dimensions. Its structure means it doesn’t have to be stored in full and can be decoded quickly. QuIP# first applies randomized Hadamard rotations so the weights look like round, featureless Gaussian clouds, which is exactly what a lattice codebook fits best.

llama.cpp’s IQ2 and IQ3 formats borrow the idea. In IQ2_XXS, each group of 8 weights is stored as an 8-bit index into a fixed grid of 256 magnitude patterns drawn from the E8 lattice, plus 7 bits for the signs (the eighth sign is implied by keeping an even number of negatives), and a small scale per 32 weights: 2.06 bits per weight in total. See GGUF.

Ternary weights: BitNet b1.58

The most extreme route is to train with tiny weights from the start. BitNet b1.58 restricts every weight in its linear layers to $\{-1, 0, +1\}$ . Three values carry $\log_2 3 \approx 1.58$ bits of information, hence the name.

During training, a full-precision copy of the weights is kept and updated by the optimiser, but every forward pass uses the ternary version:

Compute $\gamma$ , the mean absolute value of the weight matrix.
Divide every weight by $\gamma$ , round to the nearest integer, and clip to $[-1, 1]$ .
Gradients pass through the rounding with the straight-through estimator.

Activations are quantized to 8 bits per token. The payoff is that a matrix multiply with ternary weights needs no multiplications: each output is a sum of some inputs minus a sum of others, times one scale.

Try it on one row:

weights, with the ±γ/2 rounding thresholds (γ = 0.026)

−1+1+1−1000+1−1−10−1

the dot product with the inputs: only additions and subtractions

0.026 × ( − 0.58 + 0.91 + 0.16 − 1.64 + 1.21 − 0.82 − 0.54 − 0.71 ) = -0.103

Zeros are simply skipped: 4 of the 12 inputs are never touched.

exact (16-bit weights)-0.174

ternary-0.103

multiplications1 (by γ), not 12

Every weight above the upper dashed line becomes +1, below the lower one −1, and in between 0. The answer is close to the exact dot product but not equal: that is why BitNet models are trained with ternary weights from the start, so the network learns to work with them, rather than rounded afterwards.

One row of 12 weights (bars) and the 12 inputs it multiplies. Each weight becomes −1, 0 or +1 after dividing by γ, the row’s average absolute weight.

example · ternary by hand

A row of four weights: $[0.31, -0.02, -0.47, 0.12]$ . Their mean absolute value is $\gamma = 0.23$ . Dividing gives $[1.35, -0.09, -2.04, 0.52]$ ; rounding and clipping gives $[1, 0, -1, 1]$ . The output for input $x$ is

y = 0.23 \times (x_1 - x_3 + x_4)

One addition, one subtraction and one multiply by the shared scale, instead of four multiplies. Weight 2 is simply skipped.

example · a panel of three-way switches

A ternary weight is like a three-way switch: off, forward, reverse. One switch can’t express much. But a room with thousands of switches, each set deliberately during design, can produce any lighting you want. That’s the BitNet bet: give a model enough ternary weights, train it knowing that’s all it has, and it can match a model with full-precision weights. Flipping a finished full-precision model’s weights to ternary, by contrast, destroys it.

Results so far: the BitNet b1.58 paper reports matching full-precision LLaMA-style models at 3B parameters and above on perplexity and zero-shot tasks, and the BitNet b1.58 2B4T release (2 billion parameters, 4 trillion training tokens) is competitive with full-precision open models of similar size, with weights taking about 0.4 GB. The catch: it has to be trained this way from the start, and fast inference needs special kernels (Microsoft’s bitnet.cpp), since GPUs have no ternary arithmetic.

Where this shows up

Hadamard incoherence before lattice codebooks: rotations.
The IQ formats in GGUF files: GGUF and llama.cpp.
Training through rounding: PTQ vs QAT.

going deeper

Vector quantization

For groups $w \in \mathbb{R}^g$ and a codebook $\mathcal{C} = \{c_1, \dots, c_K\}$ , quantization is

\hat w = c_{k^*}, \qquad k^* = \arg\min_k \Vert w - c_k \Vert^2, \qquad \text{bits per weight} = \frac{\log_2 K}{g}

A codebook is learned by k-means (Lloyd’s algorithm): assign each group to its nearest codeword, move each codeword to the mean of its group, repeat. Scalar quantization with $L$ levels per weight is the special case where $\mathcal{C}$ is the grid $\{\ell_1, \dots, \ell_L\}^g$ .

Additive quantization (AQLM)

AQLM represents each group as a sum of $M$ codewords, one from each of $M$ codebooks of size $2^B$ :

\hat w = \sum_{m=1}^{M} C_m[i_m], \qquad \text{bits per weight} \approx \frac{M B}{g}

The codebooks and codes are optimised to minimise the layer’s output error $\Vert WX - \hat W X \Vert^2$ on calibration data, alternating beam search over codes with gradient updates of codebooks, followed by fine-tuning of the whole model.

BitNet b1.58 quantization

\tilde W = \operatorname{RoundClip}\!\Big(\frac{W}{\gamma + \epsilon}, -1, 1\Big), \qquad \gamma = \frac{1}{nm} \sum_{ij} |W_{ij}|

with $\operatorname{RoundClip}(x, a, b) = \max(a, \min(b, \operatorname{round}(x)))$ . Activations use per-token absmax INT8.

In papers

AQLM: Extreme Compression of Large Language Models via Additive Quantization (Egiazarian et al., 2024).
QuIP# (Tseng et al., 2024): Hadamard incoherence plus the E8P lattice codebook.
BitNet (Wang et al., 2023): 1-bit (binary) weights trained from scratch.
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (Ma et al., 2024): ternary BitNet b1.58.
BitNet b1.58 2B4T Technical Report (Ma et al., 2025): an openly released 2B ternary model trained on 4T tokens.