Weights, activations and the KV cache

You can quantize what is stored (weights, KV cache) or also what is computed (activations). Weight-only quantization speeds up chat with few users; quantizing activations too speeds up big batches. W4A16 and W8A8 name the combination.

Key idea: Weight-only (W4A16) vs weight-and-activation (W8A8, W4A8) quantization, memory-bound vs compute-bound decoding, and the KV cache as a third target
Real world: A restaurant kitchen. With few orders, the bottleneck is fetching ingredients from the storeroom; smaller packaging makes the trips faster. In the dinner rush, the bottleneck is the chefs; only faster cooking helps.
In an LLM: On a laptop with one user, a 4-bit model generates about 3× faster than bf16 because it reads 3× fewer bytes. On a server with hundreds of users at once, 4-bit weights barely help; fp8 weights and activations make the maths itself up to twice as fast.

Three things you can quantize

What	Stored or computed	Effect
Weights	stored, fixed after training	smaller model, less to read per token
Activations	computed fresh for every token	lets the matrix multiply itself run in int8, fp8 or fp4
KV cache	stored, grows with context	more users or longer contexts per GPU

Schemes are named by bits: W4A16 means 4-bit weights and 16-bit activations; W8A8 means 8-bit weights and activations; W4A8KV4 adds a 4-bit KV cache.

Weight-only: shrink what you read

In weight-only quantization, weights are stored in 4 bits (or 3, or 8) and expanded back to bf16 inside the matrix-multiply kernel, just before use. The arithmetic still happens in bf16. That sounds pointless, until you remember that generating one token for one user is limited by reading the weights, not by the arithmetic (model memory). Reading 4.5 bits instead of 16 is up to 3.5× faster.

This is what GGUF, GPTQ, AWQ and most “4-bit” downloads are: weight-only formats for low-batch inference.

Weights and activations: faster maths

When a server handles many requests at once, each weight read from memory is used for many tokens, and the bottleneck moves to the arithmetic units. Now storing weights in 4 bits doesn’t help much, because they are expanded to bf16 for the maths anyway.

To speed up the maths, both operands must be in a low-precision format the hardware can multiply directly: int8 or fp8 on recent GPUs (about 2× the bf16 rate), fp4 on the newest (about 4×). That requires quantizing activations, on the fly, every step, which is harder because of activation outliers.

Try it

batch (users at once)1context per user256

W16A16 (bf16)W8A8 (fp8)W4A16 (4-bit weights)

W16A16 · memory-bound209 tok/s

W8A8 · memory-bound417 tok/s (2.0×)

W4A16 · memory-bound739 tok/s (3.5×)

KV cache in memory34 MB

With one user, the GPU spends nearly all its time reading weights and the arithmetic units sit idle. Fewer bytes means more speed: 4-bit weights are 3.5× faster than bf16 even though the maths still runs in bf16.

Each decoding step reads all the weights and every sequence’s KV cache, then does one multiply-add (2 operations) per parameter per sequence; whichever takes longer sets the speed. Both axes are logarithmic. Dashed lines are the compute roofs no batch size can beat. Ideal ceilings, not benchmarks.

example · a restaurant kitchen

At 3 p.m. there are two orders, and the chef spends most of the time walking to the storeroom. Pre-chopped, compactly packed ingredients make each trip faster. At 8 p.m. there are fifty orders, ingredients are fetched once for many dishes, and the bottleneck is chopping and cooking. Packaging no longer matters; more chefs or faster knives do.

example · one GPU, two workloads

Llama 3.1 8B on one H100. A single chat user: bf16 tops out around 200 tokens/s from bandwidth; 4-bit weights raise that ceiling to about 700. A batch of 256 short requests: the GPU’s arithmetic becomes the limit, 4-bit weights stop helping, and fp8 weights and activations (W8A8) are the fastest, up to about twice bf16 when the KV cache is small. With long contexts the KV cache dominates the reads again, which is why it gets quantized too. That is why chat apps on laptops use GGUF or AWQ, while large serving clusters use FP8 (and now NVFP4).

The KV cache

The KV cache is stored like the weights, read every step like the weights, and with long contexts and many users can be larger than the weights. Quantizing it to fp8 or 4 bits halves or quarters that memory, which means more simultaneous users, longer contexts and faster attention. Keys and values have their own outlier patterns, so they need their own recipes: see quantizing the KV cache.

Where this shows up

Bandwidth and the speed ceiling: model memory.
Activation outliers, the obstacle to W8A8: LLM.int8() and SmoothQuant.
Hardware formats for fast low-precision maths: FP8, MXFP4 and NVFP4.

going deeper

Arithmetic intensity

A linear layer with weight $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ applied to a batch of $B$ tokens does $2 B d_{\text{in}} d_{\text{out}}$ FLOPs and reads $d_{\text{in}} d_{\text{out}} \cdot b_w / 8$ bytes of weights. Its arithmetic intensity is

I = \frac{2 B d_{\text{in}} d_{\text{out}}}{d_{\text{in}} d_{\text{out}} b_w / 8} = \frac{16 B}{b_w} \text{ FLOP per byte}

The layer is memory-bound when $I$ is below the GPU’s ratio of peak FLOP/s to bandwidth, about 300 FLOP per byte for bf16 on an H100. At $b_w = 16$ that means $B \lesssim 300$ ; at $b_w = 4$ , $B \lesssim 75$ . Below those batch sizes, fewer weight bits are a direct speed-up; above them, only faster arithmetic helps.

Notation

Papers write schemes as W $x$ A $y$ KV $z$ : $x$ -bit weights, $y$ -bit activations, $z$ -bit KV cache. QServe (Lin et al., 2024) uses W4A8KV4 to get both small weights and fast int8 arithmetic. Dynamic activation quantization computes scales at run time (per token); static uses scales fixed during calibration.