What a forward pass costs

A transformer does about 2 FLOPs per parameter for every token it reads, 6 when training, plus an attention term that grows with the context. But when generating for one user, arithmetic is not the bottleneck. Every new token has to read every weight from memory, so speed is set by memory bandwidth, and batching many users together is almost free.

Key idea: FLOPs ≈ 2N per token (6N to train), attention’s extra term, memory-bound decode, prefill vs decode, batching
Real world: A pizza oven that bakes eight pizzas in the time it takes to bake one. If only one order comes in, the oven is mostly empty. The bottleneck is fetching the dough, so a busy pizzeria bakes many orders together.
In an LLM: Llama 3 8B in 16-bit is 16 GB. An H100 reads 3.35 TB/s, so one user can get at most about 210 tokens per second. Batch 64 users with 1,024-token chats together and each still gets over 130 tokens per second, while the GPU produces over 8,000.

Counting the arithmetic

Almost all the work in a transformer is matrix multiplication with the weights: the query, key, value and output projections, the MLP and the unembedding. Pushing one token through a weight matrix with $m \times n$ entries takes $m \times n$ multiplications and about as many additions: 2 per weight. So for a model with $N$ parameters:

\text{FLOPs per token} \approx 2N \quad\text{(forward)}, \qquad \approx 6N \quad\text{(training: forward + backward)}

The backward pass costs about twice the forward pass, because for each weight matrix it computes two products: one for the gradient of the input and one for the gradient of the weights (backpropagation).

That leaves attention itself. A token at position $T$ compares its query with $T$ keys and mixes $T$ values, in every layer. Those are products between activations, not with weights, so they are not in $N$ . They cost about $4 \times \text{layers} \times T \times d_{\text{model}}$ FLOPs per token and grow with the context.

training (forward + backward)

context8k tokens

weights: 2 × 7.5B params = 15.0 GFLOP (78%)attention: 4 × 32 × 8k × 4096 = 4.3 GFLOP

per token19.3 GFLOP

attention share22%

attention = weights at28,610 tokens

At 8k tokens the attention part is 22% of the work. It grows with every token of context, while the weight part stays fixed.

Arithmetic for one token, split into the part that multiplies by the weights and the part that compares with earlier tokens. Pick a model and drag the context length.

For Llama 3 8B, attention only matches the weight part at about 29,000 tokens of context. For ordinary chats the rule “2 FLOPs per parameter per token” is close. For very long contexts, attention dominates: summed over a whole sequence, its cost grows with $T^2$ , which is what efficient attention tries to tame.

example · training Llama 3

Llama 3 8B was trained on about 15 trillion tokens. At 6 FLOPs per parameter per token that is $6 \times 8 \times 10^9 \times 15 \times 10^{12} \approx 7 \times 10^{23}$ FLOPs. An H100 does at most about $10^{15}$ FLOP/s in 16-bit, and well-run training reaches roughly 40% of that, so it is about 500,000 GPU-hours: three weeks on 1,000 GPUs. Meta reported 1.3 million H100 hours for the 8B model: the same order of magnitude, with real-world overheads on top. The browser model on the previous page needed about $3 \times 10^9$ FLOPs in total.

Memory is the real limit

A GPU has two speeds:

Compute: an H100 does up to about 989 trillion 16-bit FLOPs per second (dense).
Memory bandwidth: it reads its memory at 3.35 TB per second.

Divide one by the other: the GPU can do about 300 FLOPs in the time it takes to read one byte. A computation that does fewer FLOPs per byte it reads (its ) leaves the arithmetic units waiting: it is .

Now look at generating one token for one user. Every weight is read from memory once, 2 bytes in 16-bit, and used for 2 FLOPs. That is 1 FLOP per byte, 300 times below what the GPU could do. So the arithmetic is irrelevant and the speed limit is just

\text{tokens per second} \lesssim \frac{\text{memory bandwidth}}{\text{bytes of weights}} = \frac{3.35 \times 10^{12}}{16.1 \times 10^{9}} \approx 210

for Llama 3 8B in 16-bit on an H100. In 4-bit the weights are a quarter of the size and the limit is about 830 tokens per second, which is why quantization speeds up generation as well as saving memory (what to quantize). Real systems get somewhat below these ceilings.

batch1

tokens/s, all conversations togetherarithmetic limit

read per step: weights 16.1 GB + KV cache 0.13 GBmemory-bound

time reading memory 4.83 mstime on arithmetic 0.02 ms

per conversation207 tok/s

all together207 tok/s

FLOPs per byte read1.0

One conversation: every token means reading all 16.1 GB of weights, but only about 15 billion FLOPs. The GPU spends 100% of the step waiting on memory, so the ceiling is about 207 tokens per second. The prompt is different: prefill handles all its tokens in one pass, like a big batch, so it can run near the arithmetic limit, up to about 66k tokens per second.

How fast Llama 3 8B can generate on one GPU, from two limits: reading memory and doing arithmetic. Drag the batch size, the number of conversations decoded together.

Batching: many users for the price of one

The weights read for one user can be used for many. If the server decodes 64 conversations in the same step, it reads the weights once and does 64 times the arithmetic. While the GPU is memory-bound, that extra arithmetic is free: each user barely slows down and the total output grows almost 64-fold. This is why every serving system batches requests, and why a busy server is much cheaper per token than a quiet one.

Two things stop the free lunch:

Arithmetic catches up. Somewhere around a batch of a few hundred (about 300 at 16-bit on an H100, if the KV cache cost nothing), the arithmetic takes as long as the reads, and the GPU becomes compute-bound. More users then only slow everyone down.
The KV cache. Each conversation also reads its own KV cache every step, and that is not shared. With 1,024 tokens of context, one conversation’s cache for Llama 3 8B is 134 MB, read every step. With 64 conversations that is 8.6 GB per step, more than half the size of the weights. With long contexts the cache reads take over, and they also fill the memory, which caps the batch.

example · the pizza oven

A pizzeria’s oven bakes up to eight pizzas at once in four minutes. The slow part is not the oven but the cook rolling dough and fetching toppings for each bake. With one order, the oven bakes one pizza in four minutes. With eight orders, the same four minutes produce eight pizzas. Up to the oven’s limit, extra orders are nearly free; past it, they queue. An LLM server is the same: the weights are fetched once per step whether one user or a hundred are waiting.

Prefill is different

The prompt does not go through one token at a time. In , all its tokens are multiplied by each weight matrix together, so every weight read is used thousands of times. Prefill is compute-bound: an H100 could in principle process about $989 \times 10^{12} / (2 \times 7.5 \times 10^{9}) \approx 66{,}000$ prompt tokens per second for Llama 3 8B, against about 210 decode tokens per second for a single user.

That is why chat interfaces show two speeds: a short wait before the first word (time to first token, set by prefill and the queue), then a steady stream (tokens per second, set by memory bandwidth). It is also why reading a long document is quick but writing a long answer is slow, and why API prices charge less for input tokens than for output tokens.

going deeper

Where 2N comes from

A linear layer $y = xW$ with $W$ of shape $d_{\text{in}} \times d_{\text{out}}$ does $d_{\text{in}} d_{\text{out}}$ multiply-adds per token, so $2\,d_{\text{in}} d_{\text{out}}$ FLOPs. Summing over all weight matrices gives $2N$ , where $N$ counts the parameters that are multiplied: all of them except the input embedding table, which is only looked up (for Llama 3 8B, $N \approx 7.5\text{B}$ of 8.03B). Biases, normalization and activation functions add well under 1%.

Training: the backward pass computes $\partial L/\partial x = (\partial L/\partial y)\,W^\top$ and $\partial L/\partial W = x^\top (\partial L/\partial y)$ , two products the size of the forward one, so $6N$ in all (Kaplan et al., 2020, table 1).

The attention term

Per layer and per token at position $T$ : $q \cdot k_j$ for $T$ keys ( $2 T d$ FLOPs over all heads) and $\sum_j a_j v_j$ ( $2 T d$ ), so $4\,T\,d_{\text{model}}$ per layer. Attention equals the weight part when $4 L T d = 2N$ , at $T = N / (2 L d)$ : about 29,000 tokens for Llama 3 8B, 53,000 for Llama 3 70B, 6,700 for GPT-2 small (which can only read 1,024).

Time per decode step $\approx \max\big(\text{bytes read} / \text{bandwidth},\ \text{FLOPs} / \text{peak FLOP/s}\big)$ , with bytes read = weights + batch × context × KV bytes per token, and FLOPs = batch × $(2N + 4 L T d)$ . Assumptions: H100 SXM 3.35 TB/s and 989 TFLOP/s dense BF16; A100 80 GB SXM 2.04 TB/s and 312 TFLOP/s; 80 GB of memory; 16-bit KV cache; perfect overlap of reading and arithmetic and no other overheads, so real numbers are lower. Strictly, the input embedding table is read one row at a time, which raises the single-user ceiling for Llama 3 8B from about 209 to about 223 tokens per second. This picture is the roofline model (Williams et al., 2009).

Llama 3 training compute

Meta’s model card reports 1.3 million H100-80GB GPU hours for Llama 3 8B and 6.4 million for 70B (model card); the paper reports up to 3.8 × 10²⁵ FLOPs for the 405B model (Llama Team, 2024).