How much memory a model needs

Parameters × bytes per parameter gives the size of the weights. Add the KV cache for the conversation, and remember that every generated token reads all of it, so fewer bytes also means more speed.

Key idea: Model size = parameters × bits ÷ 8, the KV cache, VRAM vs disk, and why generation speed is limited by memory bandwidth
Real world: Moving house: what matters isn’t just whether your furniture fits in the van, but how many trips it takes. Smaller boxes mean fewer trips.
In an LLM: Llama 3.1 8B in bf16 is 16 GB, too big for a 12 GB gaming GPU. In 4-bit it is about 4.6 GB: it fits, and it generates text roughly three times faster because there are fewer bytes to read per token.

The one formula

A model’s weights take

\text{bytes} = \text{parameters} \times \frac{\text{bits per parameter}}{8}

So for an 8-billion-parameter model:

Format	Bits	Size
fp32	32	32 GB
bf16 / fp16	16	16 GB
8-bit	about 8.5	about 8.5 GB
4-bit	about 4.5	about 4.5 GB
2-bit	about 2.6	about 2.6 GB

The “about” is because quantized formats also store scales, which add a fraction of a bit per weight: see granularity.

Disk size for any model

Pick a model size and read off how big its file is in each format. A 4B model, for example, is 16 GB in fp32, 8 GB in bf16, 4 GB in int8 and about 2.5 GB as a 4-bit GGUF.

parameters4B

fp323216.0 GB

bf16 / fp16168.0 GB

int8 / fp884.0 GB

GGUF Q8_08.524.3 GB

GGUF Q6_K6.573.3 GB

GGUF Q5_K_M5.72.9 GB

GGUF Q4_K_M4.92.5 GB

GPTQ/AWQ 4-bit4.252.1 GB

GGUF IQ3_XXS3.261.6 GB

GGUF Q2_K3.171.6 GB

GGUF IQ2_XXS2.41.2 GB

16–32 bits8 bits4–6 bitsunder 4 bits

A 4B model is 16.0 GB in fp32, 8.0 GB in bf16, 4.0 GB in int8 and about 2.5 GB as a Q4_K_M GGUF. The small number next to each format is its bits per weight, scales included: Q4_K_M keeps some layers at 6 bits, so it averages about 4.9, not 4.5. To run it you also need memory for the KV cache and the runtime, usually 1–2 GB more for short chats.

Size of the weights file: parameters × bits per weight ÷ 8. GB is 10⁹ bytes; GiB (what many tools show) is about 7% larger per unit, so the GiB number is about 7% smaller.

The KV cache

While generating, a transformer keeps the keys and values of every previous token so it doesn’t recompute them: the KV cache. Its size grows with the length of the conversation and the number of users:

\text{KV bytes per token} = 2 \times \text{layers} \times \text{KV heads} \times \text{head size} \times \text{bytes per value}

For Llama 3.1 8B in bf16 that is $2 \times 32 \times 8 \times 128 \times 2 = 131{,}072$ bytes, 128 KiB per token. A 32,000-token conversation needs 4 GB of KV cache, a quarter of the bf16 weights.

Try it

context (thousand tokens)8k

laptop GPU · 8 GB

RTX 4090 · 24 GB

weights 16.1 GBKV cache 1.1 GB

total17.1 GB

smallest setup that fitsRTX 4090

tokens/s ceiling there~59

Each generated token has to read every weight once (plus the KV cache), so memory bandwidth caps the speed: on a RTX 4090 that’s about 59 tokens per second at most for one user. Halve the bytes and you roughly double that ceiling, which is why quantization makes models faster, not just smaller.

Weights take parameters × bits ÷ 8 bytes. The KV cache (the model’s memory of the conversation so far, kept in bf16 here) grows with every token of context.

Speed: every token reads every weight

To generate one token, the GPU must read every weight from memory once. For a single user, the arithmetic is quick; the reading is slow. So a rough ceiling on speed is

\text{tokens per second} \lesssim \frac{\text{memory bandwidth}}{\text{bytes read per token}}

An RTX 4090 reads about 1 TB/s. With 16 GB of bf16 weights that is at most about 60 tokens per second; with 4.6 GB of 4-bit weights, over 200. This is why quantization is about speed as much as size: what to quantize shows when this stops being true.

example · moving house

You have 16 m³ of belongings and a van that holds 20 m³, but the van takes an hour per round trip. Pack the same things into 4.5 m³ (vacuum bags, flat-packing) and they still fit, but now everything goes in one trip with room to spare for a friend’s boxes. Compression helps whether the bottleneck is space or trips.

example · a 70B model at home

Llama 3.1 70B in bf16 is 141 GB: no single consumer GPU comes close. At about 4.5 bits per weight it is around 40 GB: two 24 GB GPUs, or one Mac with 64 GB of unified memory. At about 2.5 bits, 22 GB fits a single 24 GB card, at a real cost in quality. Choosing that trade-off is what the GGUF and choosing pages are about.

Disk, RAM and VRAM

Disk holds the file. A quantized GGUF or safetensors file is roughly the size of its weights.
VRAM (GPU memory) is where weights must live to run fast. It is the scarce resource.
RAM can hold weights that don’t fit in VRAM (llama.cpp can split layers between them), but CPU memory bandwidth is much lower, so those layers run much slower.

Training needs far more memory than inference: gradients, optimizer states (Adam keeps two extra numbers per weight, usually in fp32) and saved activations. A rough rule is 16 bytes per parameter for mixed-precision training with Adam, which is why QLoRA freezes a 4-bit base model and only trains small adapters.

Where this shows up

Bits and bytes: bits, bytes and integers.
Memory-bound vs compute-bound: what to quantize.
Shrinking the KV cache: quantizing the KV cache.

going deeper

The arithmetic

For $P$ parameters at $b$ bits per weight (including scale overhead), weights take $P b / 8$ bytes. The KV cache for batch $B$ and context $T$ is

M_{KV} = 2 \cdot L \cdot H_{kv} \cdot d_h \cdot T \cdot B \cdot \tfrac{b_{KV}}{8}

with $L$ layers, $H_{kv}$ key/value heads (grouped-query attention makes this smaller than the number of query heads) and head size $d_h$ .

A decoding step at batch size $B$ costs about $2PB$ floating-point operations and reads $Pb/8 + M_{KV}$ bytes. Its time is roughly

t \approx \max\Big(\frac{Pb/8 + M_{KV}}{\text{bandwidth}},\; \frac{2PB}{\text{FLOP/s}}\Big)

The ratio of FLOPs to bytes is the arithmetic intensity; at small $B$ it is far below what a GPU can sustain, so decoding is memory-bound.

Training memory

Mixed-precision training with Adam typically keeps bf16 weights (2 bytes), bf16 gradients (2), an fp32 master copy (4) and two fp32 Adam moments (8): 16 bytes per parameter before activations. A 7B model therefore needs over 110 GB to fully fine-tune, versus about 4 GB of frozen 4-bit weights plus small trainable adapters under QLoRA (Dettmers et al., 2023).