Quantizing the KV cache

Every token in the context leaves keys and values behind in every layer. With long contexts or many users, that cache outgrows the model itself. Storing it in 8, 4 or even 2 bits is how long-context serving fits in memory.

Key idea: KV cache size, FP8 and integer KV caches, per-channel keys and per-token values (KIVI), keeping recent tokens in full precision
Real world: Taking notes in a day-long meeting. You can’t write down every word, so older discussion gets summarised in shorthand, while the last few minutes stay in full sentences because you’re still working with them.
In an LLM: Llama 3 8B needs 128 KiB of KV cache per token in 16-bit: 16 GiB for one 128k-token conversation, over three times the size of its 4-bit weights. Store keys and values in FP8 and that halves; in 4 bits it quarters.

The other big memory user

During generation, each layer’s attention looks back at the keys and values of every earlier token. Recomputing them for every new token would be far too slow, so they are stored: the KV cache (introduced in how much memory a model needs). Its size is

\text{KV bytes} = 2 \times \text{layers} \times \text{KV heads} \times \text{head size} \times \text{tokens} \times \text{bytes per value}

The 2 is for keys and values. It grows linearly with context length and with the number of conversations served at once, while the weights stay fixed.

Try it

context32kusers at once1

weights + KV cache against a 24 GB GPU

24 GB

Fits, with 14.1 GiB to spare.

memory, GiB

weights, 4.5-bit4.2

KV cache4.0

weights, 16-bit (for reference)15.0

KV per token128 KiB

KV cache total4.0 GiB

weights + KV8.2 GiB

The cache is still smaller than the weights. Push the context or the number of users up and watch it overtake them.

Memory for the model’s weights (at about 4.5 bits per weight) plus its KV cache, which grows with every token of context and every user served at once. 4- and 2-bit caches include about half a bit per value for their scales and offsets.

Two levers shrink it: architecture (grouped-query attention, which shares each key/value head among several query heads, cut Llama 3’s cache 4× compared with an equivalent model without it) and quantization.

How the KV cache is quantized

FP8 KV cache. The simplest and now common in serving: keys and values stored as FP8 (E4M3 or E5M2), usually with per-tensor or per-head scales. Half the memory, usually no measurable loss. In vLLM: kv_cache_dtype="fp8".
INT8 / INT4 blocks. llama.cpp can store the cache in its own block formats: --cache-type-k q8_0 --cache-type-v q8_0 (quantizing V requires flash attention, -fa). Q8_0 is close to lossless; Q4_0 saves more but costs some quality, more for keys than for values.
2-bit research methods such as KIVI and KVQuant push further by exploiting how keys and values are shaped.

The two groupings on a matrix of tokens × channels: per token gives each row its own scale, per channel gives each column its own.

a KV cache: rows = tokens, columns = channels

★

one scale covers: one token’s activations (4096 values)
used by: Dynamic activation quantization in W8A8 and FP8 serving; values in a KV cache.

scales in a 4096 × 4096 layerone per token, computed on the fly

extra bits per weighttiny

numbers sharing the lit scale4,096

The ★ outlier sets the scale for its whole group: here 4,096 real numbers get a coarser grid because of it. Fewer, bigger groups store fewer scales but let one outlier spoil more numbers.

A toy 16 × 64 matrix, coloured by which numbers share a scale. The ★ is one large outlier weight; its group is lit up by default. Hover or tap any square to light up its group. Row-wise blocks are drawn at a quarter of their real length (one square ≈ 4 weights); the numbers below use real sizes for a 4096 × 4096 layer.

Keys and values are shaped differently

Look at a key cache as a table: one row per token, one column per channel. Keys have outlier channels: a few columns are large in every token (partly because of how rotary position embeddings act on certain dimensions). Values have no such column pattern, but some tokens are larger overall than others. The right grouping follows the shape:

Keys: per channel. Each column gets its own scale, so the outlier columns don’t ruin the others.
Values: per token. Each row gets its own scale. The attention output is a weighted mix of value rows, so errors should be controlled row by row.

keep the last 4 tokens in full precision

keys

error after 2-bit quantization

Outlined: one example group (a token row) sharing a single 2-bit scale and offset.

relative error48%

grouped per channel instead16%

Keys have outlier channels: two columns are large in every token. Grouping by token puts an outlier in every group, so each row’s 4 levels are spread wide and the ordinary channels lose their detail.

A slice of one attention head’s cache: 24 tokens (rows) × 16 channels (columns). Blue is positive, red negative. Each group of numbers gets its own 2-bit scale and offset.

That’s exactly what KIVI does at 2 bits. Per-channel grouping has a practical snag, though: a column’s scale needs many tokens’ worth of that column, but new tokens arrive one at a time. KIVI keeps the most recent tokens (a window of a few dozen to a hundred or so) in full precision, and quantizes them as a group once the window fills. Recent tokens, which usually matter most for the next prediction, are therefore never quantized.

example · meeting notes

At 9 am you write full sentences. By 4 pm the morning is a page of shorthand: “budget ↑10%, A. owns hiring”. That’s enough to follow the discussion, while the last ten minutes are still in full so you can respond precisely. Older context, compressed; recent context, exact: KIVI’s full-precision window.

example · long context on one GPU

Serving Llama 3 8B with a 4-bit weight file (about 4.6 GiB) to 8 users with 32k-token contexts on a 24 GB GPU: a 16-bit KV cache would need $128\text{ KiB} \times 32{,}768 \times 8 = 32$ GiB, and doesn’t fit. FP8 KV needs 16 GiB, which fits alongside the weights with a little room to spare. A 4-bit KV cache would need 8 GiB and leave room for longer contexts or more users.

Where this shows up

The size formula and why decoding is limited by memory reads: how much memory a model needs.
What else to quantize besides weights: weights, activations and the KV cache.
Hadamard rotations make 4-bit KV caches work too: rotations.

going deeper

Per-channel and per-token grouping

For the key cache $K \in \mathbb{R}^{T \times d}$ (tokens × channels) and value cache $V$ , KIVI uses asymmetric $b$ -bit quantization with groups along different axes:

\hat K_{:,j} = Q_b\big(K_{:,j}\big) \text{ per channel } j \qquad \hat V_{t,:} = Q_b\big(V_{t,:}\big) \text{ per token } t

with $Q_b(x) = \operatorname{round}\big((x - \min x)/s\big)\, s + \min x$ and $s = (\max x - \min x)/(2^b - 1)$ , applied to groups of a fixed number of tokens. The newest $R$ tokens are kept in full precision.

Why values per token

The attention output for a query is $o = \sum_t p_t\, V_{t,:}$ with attention weights $p_t$ . Quantization error in a value row is multiplied by that token’s weight $p_t$ , so the error in $o$ is a weighted sum of per-row errors. Controlling each row’s error relative to its own size keeps every term small. Keys enter through dot products $q \cdot K_{t,:}$ , where an outlier channel affects every token’s score, so it’s the channel that needs its own scale.

In papers

KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (Liu et al., 2024).
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (Hooper et al., 2024): quantizes keys before rotary embeddings are applied, uses non-uniform levels, and keeps a few outliers in a sparse side matrix.
QuaRot and QServe (W4A8KV4) quantize the cache to 4 bits as part of a full low-precision serving stack.