GGUF and llama.cpp

The file format behind local LLMs. A GGUF file holds the tokenizer, settings and every tensor, each stored in blocks with their own scales. Names like Q4_K_M, Q8_0 and IQ3_XXS describe those blocks, and this page decodes them.

Key idea: The GGUF file layout, legacy block quants (Q8_0, Q4_0), k-quants with super-blocks (Q4_K…), mixes (_S, _M, _L), i-quants (IQ…) and the importance matrix
Real world: A shipping container with the packing list taped inside the door: one box that holds everything, plus a description of what’s where, so whoever opens it can unpack without asking.
In an LLM: Meta-Llama-3-8B-Instruct-Q4_K_M.gguf is a single 4.6 GiB file: the tokenizer, the chat template, the architecture settings and all 291 weight tensors, most of them in 4-bit k-quant blocks and a few in 6-bit. llama.cpp, Ollama and LM Studio can memory-map it and start generating.

One file with everything

GGUF is the file format of llama.cpp and the tools built on it (Ollama, LM Studio, and many others). Its predecessors (GGML and GGJT) had fixed headers that couldn’t describe new architectures or settings without breaking old files. GGUF stores everything a runtime needs as open-ended key–value metadata, so one file is self-contained. It is laid out like this:

Part	What it holds
Header	the magic bytes `GGUF`, a version number, how many tensors and how many metadata entries follow
Metadata	typed key–value pairs: `general.architecture = llama`, context length, number of layers and heads, RoPE settings, the full tokenizer vocabulary and merges, the chat template…
Tensor info	for each tensor: its name (`blk.0.attn_q.weight`), shape, quantization type, and where its data starts
Padding	zeros up to the next 32-byte boundary (`general.alignment`)
Tensor data	the blocks themselves, back to back

Because the tensor data is aligned and stored exactly as the program will use it, llama.cpp can memory-map the file: the operating system pages weights in from disk as they’re needed, with no loading step or extra copy.

example · the packing list

A shipping container arrives with a list on the door: “pallet 1, rows 1–4: 200 boxes of screws, size M4 …”. The receiver can find any item without unpacking everything. The metadata and tensor info are GGUF’s packing list; the tensor data is the cargo, and each tensor’s type tells the reader how its boxes are packed.

example · reading a real file’s list

The gguf Python package (from the llama.cpp repository) prints it:

pip install gguf
gguf-dump Meta-Llama-3-8B-Instruct-Q4_K_M.gguf | head -40

You’ll see keys such as general.architecture, llama.block_count = 32, llama.attention.head_count_kv = 8, tokenizer.chat_template, and then one line per tensor with its shape and type: most are Q4_K, but some attn_v and ffn_down tensors and the output layer are Q6_K, and the norms are F32.

Reading a GGUF name

Open a model’s download page and you’ll see a list like Q8_0, Q6_K, Q5_K_M, Q4_K_M, IQ4_XS, Q3_K_L, IQ2_XXS. Each name is built from the same few pieces:

Piece	Means
`Q` or `IQ`	ordinary quantization with evenly spaced levels, or an i-quant with non-uniform levels or a codebook
the number	roughly how many bits most weights get: 8, 6, 5, 4, 3, 2, 1
`_0`, `_1`	legacy blocks of 32: a scale only, or a scale and a minimum
`_K`	k-quant: super-blocks of 256 with small sub-block scales
`_S`, `_M`, `_L`	the mix: how many sensitive tensors get extra bits (small, medium, large)
`_XXS` … `_M`, `_NL`	i-quant size variants; `NL` is “non-linear” levels

Click any name to take it apart and see its size and measured quality:

Q4_K_M

QQuantized with uniform levels: each weight is an integer code × a scale (± a minimum).

4About 4 bits per weight for most tensors, before scales and upgrades.

_Kk-quant: super-blocks of 256 weights with one fp16 scale, and small sub-blocks with their own 4–8-bit scales.

_MMedium mix: attn_v and ffn_down get extra bits in about half the layers, plus the output layer.

bits per weight, block≈4.9

file average4.90

8B file4.58 GiB

KL from fp160.031

The default most people should pick: about a third of fp16, small loss.

Click a name from a typical model’s download list. Sizes and KL divergence are measured on Llama 3 8B by the llama.cpp project (fp16: 14.97 GiB; i-quants made with an importance matrix); “–” means not in that table. “Block” bits per weight is the type’s own layout (≈ for mixes); “file average” divides the real file by the parameter count, so it includes the layers kept at higher precision.

A quick rule for choosing:

Q8_0 if it fits: indistinguishable from the original.
Q6_K or Q5_K_M with a little memory to spare.
Q4_K_M as the default: about a third of fp16, small loss.
IQ4_XS or IQ3_M when 4-bit doesn’t quite fit.
Q2_K and the IQ2 types only for large models (70B and up) on small hardware; an 8B model at 2 bits is badly damaged.
Prefer files made with an imatrix below about 4 bits. Many uploaders mark them (or put “imat” in the repo name).

The rest of this page explains what is inside those blocks.

Blocks: how each tensor is stored

Every quantized tensor is cut into fixed-size blocks, each with its own scale, exactly the per-group scaling idea. The type name says how a block is laid out.

Here is how the two families tile a weight matrix: legacy blocks of 32 with one scale each, and k-quant super-blocks of 256 split into sub-blocks of 32. Switch between them:

weights W: rows = output channels, columns = input channels

★

Thick lines: super-blocks (each row here is one super-block of 256), shaded lighter around the highlighted one. Thin lines: sub-blocks of 32 with their own 6-bit scale.

one scale covers: super-blocks of 256 (fp16 scale and min), each split into 8 sub-blocks of 32 (6-bit scale and min)
used by: Q2_K to Q6_K, and the _S/_M/_L mixes built from them.

scales in a 4096 × 4096 layer65,536 super + 524,288 sub

extra bits per weight0.5 bits in total

numbers sharing the lit scale32 (inside a super-block of 256)

The ★ outlier sets the scale for its whole group: here 32 (inside a super-block of 256) real numbers get a coarser grid because of it. Small blocks keep the damage local, at the cost of storing more scales.

A toy 16 × 64 matrix, coloured by which numbers share a scale. The ★ is one large outlier weight; its group is lit up by default. Hover or tap any square to light up its group. Row-wise blocks are drawn at a quarter of their real length (one square ≈ 4 weights); the numbers below use real sizes for a 4096 × 4096 layer.

Try it

256 weights in one block, as 8 sub-blocks of 32

Hover or tap a sub-block to see which bytes belong to it.

144 bytes per block

d, dmin (fp16): 4 B
8 × (6-bit scale + 6-bit min): 12 B
256 × 4-bit: 128 B

bits per weight4.5

8B model, all tensors4.21 GiB

w = d · scaleₛ · q − dmin · minₛ, per 32-weight sub-block s. A k-quant super-block: 256 weights share one 16-bit scale, and each small sub-block gets its own cheap 6-bit (or 4- or 8-bit) scale relative to it. Scales stay fine-grained without costing 16 bits each.

Every tensor in a GGUF file is cut into blocks. Pick a type to see one block: its weights (top) and its bytes (bottom), drawn to scale.

Legacy quants: Q8_0, Q4_0, Q4_1, Q5_0, Q5_1

The original formats use blocks of 32 weights and one fp16 scale (_0), or a scale plus a minimum (_1, the asymmetric version). Q4_0 is 16 bytes of 4-bit values plus a 2-byte scale: 18 bytes for 32 weights, 4.5 bits per weight. Q8_0, at 8.5 bits per weight, is still the go-to “practically lossless” format.

k-quants: Q2_K to Q6_K

The k-quants (llama.cpp PR #1684) group blocks into super-blocks of 256 weights. The super-block has one fp16 scale (and one fp16 minimum); each smaller sub-block has its own scale stored in only 6 bits (or 4, or 8), relative to the super-block’s. Sub-block scales stay fine-grained without costing 16 bits each.

Worked out for Q4_K: 2 + 2 bytes for the super-block scale and minimum, 12 bytes for eight 6-bit sub-block scales and eight 6-bit minimums, and 128 bytes for 256 four-bit values: $144 \times 8 / 256 = 4.5$ bits per weight.

Mixes: what _S, _M and _L mean

A file called Q4_K_M isn’t all Q4_K. Some tensors are much more sensitive than others, so the quantize tool gives them more bits:

Q4_K_M uses Q6_K for the attention value projection (attn_v) and MLP down projection (ffn_down) in about half of the layers (the first and last eighth of the layers plus every third one in between), and for the output layer; everything else is Q4_K.
_S (small) uses fewer upgrades, _L (large) more.

That’s why the 8B model’s Q4_K_M file is 4.58 GiB: about 4.9 bits per weight on average, not 4.5.

Here is the whole model laid out, one square per tensor per layer. Switch between the mixes and watch the averages; the computed sizes land within 0.01 GiB of the real files:

layers 0 → 31

attn_q

attn_k

attn_v

attn_output

ffn_gate

ffn_up

ffn_down

output

Q4_K (4.5 bits)Q5_K (5.5 bits)Q6_K (6.5625 bits)

average bits per weight4.89

computed size4.57 GiB

real file (llama.cpp)4.58 GiB

attn_v and ffn_down are the most sensitive tensors, so in Q4_K_M they get Q6_K in half the layers: the first and last four, and every third one in between. Those few upgrades, plus the output layer, lift the average from 4.5 to 4.89 bits per weight.

Llama 3 8B, one square per tensor per layer. Colour shows the k-quant type each tensor is stored in. The embeddings are assumed to use the base type; the output layer gets Q6_K.

i-quants and the importance matrix

The i-quants (IQ1_S to IQ4_XS) push below 4 bits using two ideas:

Codebooks. IQ2 and IQ3 store groups of 8 weights as indices into a fixed grid of points from the E8 lattice, borrowed from QuIP#: see 2 bits and below. IQ4_NL and IQ4_XS use a fixed table of 16 unevenly spaced levels, like a hand-tuned NF4.
The importance matrix (imatrix). Run some text (a few tens of thousands of tokens is typical) through the full-precision model and record, for every input column of every weight matrix, the average squared activation. When choosing scales and values, errors on columns with large activations are weighted more heavily. It’s the same insight as AWQ and the diagonal of GPTQ’s Hessian, applied cheaply.

# 1. collect activation statistics
llama-imatrix -m model-f16.gguf -f calibration.txt -o imatrix.dat
# 2. quantize with them
llama-quantize --imatrix imatrix.dat model-f16.gguf model-IQ3_XXS.gguf IQ3_XXS
# 3. measure the damage against the original
llama-perplexity -m model-f16.gguf -f test.txt --kl-divergence-base base.kld
llama-perplexity -m model-IQ3_XXS.gguf --kl-divergence-base base.kld --kl-divergence

An imatrix helps every format, but below about 3.5 bits it is essential.

How much each choice costs

The llama.cpp project measures every format on the same model and text. These are real numbers for Llama 3 8B: file size against the KL divergence between the quantized model’s next-token probabilities and the FP16 model’s (the best single measure of damage).

no importance matrixwith importance matrix

size4.58 GiB

vs f1631%

perplexity (f16: 6.23)6.41

KL divergence0.031

The usual sweet spot: about a third of the size of f16, with perplexity up by a few percent.

Measured on Llama 3 8B by the llama.cpp project: file size (bottom) and the matching average bits per weight (top) against KL divergence from the FP16 model’s next-token predictions (log scale). Click a point or a name.

The shape is typical. From 16 down to about 5 bits per weight almost nothing is lost. Around 4.5–5 bits (Q4_K_M, IQ4_XS) the loss is small and the file is a third of FP16. Below about 3 bits quality drops off a cliff, and only importance-matrix i-quants remain usable. As a rule, a bigger model at 4 bits beats a smaller model at 8 bits of the same file size.

Where this shows up

Groups, super-blocks and the cost of scales: per-tensor, per-channel and per-group scales.
The lattice codebook behind IQ2 and IQ3: 2 bits and below.
KL divergence and perplexity as quality measures: measuring the damage.
Choosing a file for your machine: choosing a quantization.

going deeper

Block formulas

For a legacy block with fp16 scale $d$ (and minimum $m$ ) and integer codes $q$ :

\text{Q4\_0: } w = d\,(q - 8) \qquad \text{Q4\_1: } w = d\,q + m \qquad \text{Q8\_0: } w = d\,q

For a k-quant super-block with fp16 $d$ and $d_{\min}$ , and sub-block $s$ with 6-bit scale $a_s$ and minimum $b_s$ :

\text{Q4\_K: } w = d \cdot a_s \cdot q - d_{\min} \cdot b_s

Bits per weight

Type	Block	Bytes	Bits per weight
Q8_0	32	34	8.5
Q5_1 / Q5_0	32	24 / 22	6.0 / 5.5
Q4_1 / Q4_0	32	20 / 18	5.0 / 4.5
Q6_K	256	210	6.5625
Q5_K	256	176	5.5
Q4_K	256	144	4.5
Q3_K	256	110	3.4375
Q2_K	256	84	2.625
IQ4_NL	32	18	4.5
IQ4_XS	256	136	4.25
IQ3_XXS	256	98	3.0625
IQ2_XS / IQ2_XXS	256	74 / 66	2.3125 / 2.0625
IQ1_S	256	50	1.5625

These are per block; a whole file’s average is higher because of mixes, and because some tensors (norms, sometimes embeddings and the output layer) stay in higher precision.

The importance-weighted objective

With $\bar a_j^2$ the mean squared activation of input column $j$ over the calibration text, the quantizer chooses each block’s scale (and, for i-quants, each grid point) to minimise

\sum_j \bar a_j^2 \, (w_j - \hat w_j)^2

instead of the plain squared error. This is the diagonal approximation of the GPTQ Hessian $H = 2XX^\top$ .

In papers and code

GGUF and its formats are documented in code and pull requests rather than papers:

GGUF specification (ggml repository).
k-quants: llama.cpp PR #1684; importance matrix: PR #4861; IQ2_XXS: PR #4773; IQ4_NL: PR #5590; IQ4_XS: PR #5747. All by Iwan Kawrakow (ikawrakow).
Scoreboard numbers: the llama.cpp perplexity README (Llama 3 8B, Wikitext-2).