The frontier

Where quantization research is heading as of late 2026. Trellis codes that beat rounding at the same bits, calibration-free and learned methods, mixture-of-experts models, quantized attention, fast kernels, precision as a scaling law, and native FP4 training.

Key idea: Trellis-coded quantization (QTIP, EXL3), HQQ and OmniQuant, PV-Tuning, quantizing experts, 8- and 4-bit attention, Marlin kernels, precision scaling laws, FP4 training
Real world: Directions given as turns. “Left, right, left” means nothing on its own, but from a known starting point each turn picks one road out of the junction you are at, and a few words reach an exact address. The newest quantizers work the same way.
In an LLM: gpt-oss-120b ships with its experts already in 4-bit MXFP4, so a 117-billion-parameter model fits on one 80 GB GPU. Trellis-quantized models at about 2 bits per weight now hold up far better than 2-bit rounding ever could.

A field that moves fast

The previous pages cover the methods behind most models you will download today. This page is a tour of what has appeared on top of them, and what is likely to matter next. Each section links back to the basics it builds on. Expect some of it to be standard practice, and some of it to be forgotten, within a year or two.

Trellis codes: more from the same bits

Every method so far gives each weight (or each small group of weights) its own code, chosen independently. Trellis-coded quantization breaks that rule. The bits for a whole row are written as one stream, and each weight is read through a sliding window of the stream: with a window of $L$ bits and 2 new bits per weight, each weight can take one of $2^L$ values, but neighbouring weights share most of their bits. The quantizer searches for the single bit stream whose windows best reproduce the whole row, using the Viterbi algorithm (the same dynamic programming used to decode phone signals).

Try it

window (state) bits L8

original weightrounded to the best 4-level gridtrellis

Weight 6 = codebook entry for the 8 bits in the purple window (01110011) = -0.17; the darker 2 are the bits it adds.

bits per weight, this row2 / 2.38

error, rounding (long rows)0.115

error, trellis (long rows)0.081

theoretical best at 2.02 bits0.061

Almost the same bits per weight (the window’s 6 start-up bits are paid once per row: 2.38 bits on this short row, 2.02 on rows of 256), but the trellis error is 30% lower (this row: 0.098 vs 0.052). Each weight can choose from 256 codebook values, constrained by its neighbours; the Viterbi algorithm finds the best bit stream for the whole row at once. QTIP uses a 16-bit window and gets close to the theoretical best.

Sixteen weights from one row. Rounding gives each weight 2 bits and 4 choices. The trellis also adds 2 bits per weight (shown under each column), but reads each weight through a window of the last 8 bits, so neighbouring weights share bits. Hover or tap a weight to see its window.

The result: at 2 bits per weight (plus a few start-up bits per row), the error falls well below what any per-weight rounding can achieve, approaching the theoretical limit for Gaussian data as the window grows. QTIP (Tseng et al., 2024) combines this with the Hadamard incoherence processing of QuIP#, and computes the codebook from the window bits with a few cheap instructions instead of storing a table, so it runs fast on GPUs. The EXL3 format in ExLlamaV3 is based on QTIP, and is one of the strongest options for running models at 2–4 bits on a single GPU.

example · directions

“Second left, then first right” is a few words, yet it picks out one street among thousands, because each turn is read from where the previous turns left you. A trellis code does the same: 2 new bits per weight, but read together with the bits before them, so the same 2 bits can mean different values at different places in the row.

example · a 70B model in 20 GB

At about 2.25 bits per weight, a 70B model takes roughly 20 GB, enough for one 24 GB GPU with room for a modest KV cache. With scalar 2-bit GGUF types, that model is badly damaged; with trellis or lattice codebooks and a bit of fine-tuning, it stays usable. This is the gap between 2023 and 2025 methods.

Smarter post-training methods

Two directions refine the PTQ recipe:

No calibration data at all. HQQ (Half-Quadratic Quantization, Dropbox) chooses each group’s scale and zero-point by directly optimising how well the dequantized weights match the originals, with an error measure that tolerates a few outliers. It needs no example inputs and quantizes a 70B model in minutes, at quality close to calibrated methods at 4 bits.
Learn the transformations. OmniQuant (Shao et al., 2023) learns the clipping range of each weight matrix and SmoothQuant-style scaling and shifting factors by gradient descent, block by block, on calibration data. It covers both weight-only and weight-and-activation settings.

Training through extreme compression

Below about 3 bits, even the best rounding needs repair. The usual tool is quantization-aware fine-tuning with the straight-through estimator, which works poorly when codes are discrete indices into a codebook. PV-Tuning (Malinovskii et al., 2024) alternates between optimising the continuous parameters (scales, codebooks) and making carefully chosen discrete updates to the codes, and gives the best results so far for 1–2-bit models built with AQLM and QuIP#.

Mixture-of-experts models

Most frontier models are now mixture-of-experts (MoE): each layer has many expert networks, and each token uses only a few. Experts hold the vast majority of parameters but each is used rarely, so they are where memory goes and where quantization pays off most:

Quantize experts harder than the rest. gpt-oss stores its expert weights in MXFP4 and keeps attention and other shared weights in bf16. DeepSeek-V3 stores weights in FP8 with 128 × 128 block scales.
Experts tolerate it well. Each expert sees a narrower slice of data and errors in one expert affect only the tokens routed there; MoE models often lose less quality at the same bits than dense models (Kim et al., 2023). QMoE pushed a 1.6-trillion-parameter MoE below 1 bit per parameter (Frantar & Alistarh, 2023).
Watch the router. The small router that picks experts is sensitive; it is usually left in high precision, since a small change can send tokens to the wrong experts.

example · a library’s stacks

A library keeps the most borrowed books on open shelves near the door and the rarely requested ones packed tightly in compact storage. Fetching from storage is slower, but most of the collection is there, and most visitors never notice. Experts are the rarely borrowed books.

example · gpt-oss-120b on one GPU

gpt-oss-120b has 117 billion parameters, almost all of them in its experts, but only about 5 billion are active per token. With experts in MXFP4 (4.25 bits) and the rest in bf16, the checkpoint is roughly 60 to 65 GB, fitting a single 80 GB H100. In bf16 it would need about 234 GB.

Quantized attention

Attention itself, the $QK^\top$ and $PV$ matrix multiplies, used to stay in 16-bit even in quantized models. For long contexts it dominates the work, so it is now being quantized too:

FlashAttention-3 (Shah et al., 2024) runs attention in FP8 on H100s, with block-wise scaling and a Hadamard-style incoherence trick to tame outliers.
SageAttention (Zhang et al., 2024) computes $QK^\top$ in INT8 after subtracting each key channel’s mean (keys have outlier channels, and the mean shift removes most of them without changing the attention weights); SageAttention2 (Zhang et al., 2024) goes to INT4 for $QK^\top$ and FP8 for $PV$ . Both are drop-in replacements that speed up long-context inference by 2× or more.

Kernels: making low bits fast

A 4-bit weight is only faster if the GPU code that multiplies it can unpack and use it at full memory speed. Marlin (Frantar et al., 2024) is a W4A16 matrix-multiply kernel that gets close to the ideal 4× speed-up over fp16 at batch sizes up to about 16–32, by carefully overlapping memory loads, dequantization and tensor-core maths. Marlin-style kernels are behind GPTQ and AWQ support in vLLM. When choosing a format for a GPU server, “is there a fast kernel for it on my hardware?” matters as much as the format’s accuracy.

Precision as a scaling law

Two 2024 results changed how people think about quantization:

Heavily trained models quantize worse. Kumar et al., 2024 fit scaling laws that include precision, and find that post-training quantization hurts more the more data a model was trained on; past a point, extra pre-training data can even make the quantized model worse. Ouyang et al., 2024 report the same pattern: undertrained models quantize easily, fully trained ones don’t. This matches practice: Llama 3, trained on 15 trillion tokens, loses noticeably more at low bits than Llama 2 did (Huang et al., 2024).
Training precision is a choice with a cost curve. The same work models training in low precision as reducing the model’s “effective” parameter count, and finds that training in about 7–8 bits is close to compute-optimal.

example · a packed suitcase

A suitcase thrown together loosely can be squashed a lot smaller. One packed by an expert, with every gap filled, barely compresses at all. A model trained on more data has packed more information into each weight, so there is less slack for rounding to remove.

example · why QAT releases are back

Because modern models are trained far past the point where PTQ is free, several labs now release official quantization-aware-trained versions of their models (Google’s Gemma 3 QAT checkpoints, for example), trained for a while with 4-bit rounding in the loop. They keep much more quality than PTQ files of the same size.

Training natively in 8 and 4 bits

The same formats are moving into pre-training. DeepSeek-V3 was trained largely in FP8 with fine-grained block scales (DeepSeek-AI, 2024). In 2025, NVIDIA pre-trained a 12-billion-parameter model on 10 trillion tokens in NVFP4 with quality close to FP8 (NVIDIA, 2025), using random Hadamard transforms and stochastic rounding, and Quartet (Castro et al., 2025) showed end-to-end FP4 training that is compute-optimal on Blackwell GPUs. A model trained in FP4 needs no quantization step to run in FP4: the gap between training and serving disappears. See training in low precision.

Quantization inside RL

Reinforcement learning for LLMs spends most of its time generating rollouts, so frameworks increasingly generate them with a quantized copy of the model (FP8, sometimes lower) while training in bf16. The catch is that the rollout model’s probabilities no longer exactly match the trainer’s, which makes the update slightly off-policy; recent work corrects for it with importance weights or by matching precisions (Qi et al., 2025). See training–inference mismatch.

Where this shows up

Codebooks and lattices, the precursors of trellis codes: 2 bits and below.
Rotations that make all of this work better: rotations.
The hardware formats: FP8, MXFP4 and NVFP4.
Choosing among what exists today: choosing a quantization.

going deeper

Trellis-coded quantization

A bit-shift trellis with $L$ -bit state and $k$ bits per weight has $2^L$ states; from state $s$ , the next state is $s' = (2^k s + b) \bmod 2^L$ for $b \in \{0, \dots, 2^k - 1\}$ . Each state $s$ has a reconstruction value $c(s)$ . Quantizing a sequence $w_1, \dots, w_T$ means finding the state path minimising

\sum_{t=1}^{T} \big(w_t - c(s_t)\big)^2 \quad \text{subject to } s_{t} = (2^k s_{t-1} + b_t) \bmod 2^L

which the Viterbi algorithm solves exactly in $O(T \cdot 2^{L} \cdot 2^k)$ time. The rate is $k$ bits per weight (plus $L - k$ bits once per sequence). For i.i.d. Gaussian data the distortion approaches the rate–distortion bound $D(R) = \sigma^2\, 2^{-2R}$ as $L$ grows: $2^{-4} = 0.0625\,\sigma^2$ at 2 bits, against $0.1175\,\sigma^2$ for the best scalar 4-level quantizer. QTIP uses $L = 16$ with codes $c(s)$ generated by hash-like “compute-based” functions, so no $2^{16}$ -entry table needs to be stored or read.

HQQ’s objective

For a group of weights $W$ , HQQ finds the zero-point $z$ (with fixed scale $s$ ) minimising a sparsity-promoting $\ell_p$ norm with $p < 1$ :

\min_{z} \big\Vert W - s\,(Q_z(W) - z) \big\Vert_p

The $p < 1$ norm lets a few large residuals stay large instead of distorting the fit for every weight, which is the same instinct as clipping. It is solved with half-quadratic splitting, alternating a closed-form shrinkage step and a closed-form update of $z$ .

Precision-aware scaling

Kumar et al. model post-training quantization as adding a loss penalty that grows with the ratio of training tokens to parameters, $D/N$ , and shrinks exponentially with the bits of precision. Holding precision fixed, there is a data budget beyond which more training makes the quantized model worse. For training in low precision, they model precision as scaling the effective parameter count, $N_{\text{eff}} = N\,(1 - e^{-P/\gamma})$ , so lower training precision behaves like a smaller model.