How big should the vocabulary be?

Key idea

Vocabulary V costs V × d parameters for embeddings (twice if untied) and V × d multiply-adds per token for the output; it buys fewer tokens per text, with diminishing returns

Real world

A phrasebook. A thin one has only common words, so you spell out everything else letter by letter. A thick one has whole phrases, so you say more with each lookup, but it is heavier to carry and many pages are never used.

In an LLM

Llama 2 had 32,000 tokens, Llama 3 has 128,256. The same English text takes about 12% fewer tokens and Hindi less than half as many, but Llama 3 8B spends 1.05 billion parameters, 13% of the model, on its two vocabulary-sized matrices.

Two ends of a dial

The first tokenization page set out the extremes: bytes give a tiny vocabulary and very long sequences, whole words give short sequences and a vocabulary that never ends. BPE lets you pick any size in between by choosing when to stop merging. This page is about where to stop.

The Vocabulary size $V$ appears in exactly two places in a transformer:

• The embedding table at the input: one row of $d$ numbers per token, so $V \times d$ parameters (the embeddings page shows how it is used).
• The output layer (the unembedding): one score per token at every position, computed by multiplying the final vector by another $V \times d$ matrix. Some models reuse the embedding table here (tied embeddings), most large ones don’t.

Everything else in the model (attention, MLPs, norms) does not depend on $V$ at all.

What a bigger vocabulary costs

Four costs grow with $V$:

1. Parameters. $V \times d$ for the input table, the same again for an untied output layer. For a small model this dominates: in Gemma 3 1B the 262,144 × 1,152 table is 302 million parameters, 30% of the model. In Llama 3 70B the two 128,256 × 8,192 matrices are 2.1 billion, only 3%.
2. Compute at the output. Every position computes $V$ scores, each a dot product of length $d$: about $V \times d$ multiply-adds per token, then a softmax over $V$ entries. During training the full $T \times V$ matrix of logits also has to be stored, which for long sequences is gigabytes; training code splits it into chunks.
3. Rare tokens get little training. Each token’s embedding is only updated when that token appears. Split the same text among more tokens and each one is seen less. In the chart, our 5.6 MB tokenizer experiment shows the typical token appearing about 300 times at a 2,000-token vocabulary but only once at 50,000. Real models train on trillions of tokens, which is why they can afford big vocabularies, but the rarest entries can still end up as glitch tokens.
4. The tokenizer itself. A bigger vocabulary needs more merges learned from more text, and its rare tokens reflect whatever happened to be in that text.

What it buys

The benefit is compression: more bytes of text per token. Shorter sequences mean:

• more text fits in the context window,
• attention, whose cost grows with the square of the length, gets cheaper,
• generation takes fewer steps for the same answer,
• and each prediction covers more meaning, so the model has fewer, more informative steps to learn.

The blue curve shows the catch: compression improves quickly at first, then flattens. In our experiment going from 1,000 to 4,000 tokens raises bytes per token from 2.7 to 3.5, while going from 16,000 to 50,000 only adds 0.1. English is mostly covered by the first few tens of thousands of tokens. The extra tokens in modern vocabularies are mostly for other languages and for code.

Vocabulary sizes over time

Two things drove the jump around 2024:

• Models got bigger. In a 70B model the vocabulary matrices are a few per cent of the parameters, so a bigger vocabulary is cheap. Scaling studies found that the best vocabulary grows with model size, and that most models had been using vocabularies smaller than optimal (Tao et al., 2024).
• Many languages. A model trained on dozens of languages and lots of code needs tokens for all of them, or those users pay several times more for the same text.

The cost of a language

Tokens are what users pay for, what fills the context window, and what each generation step produces. If a tokenizer was built mostly from English, other languages fall apart into small pieces. Here is the same sentence, the first article of the Universal Declaration of Human Rights, in twelve languages:

With GPT-2’s tokenizer the Hindi sentence takes 294 tokens against 33 for English, almost 9 times as many. Hindi speakers would get a ninth of the text into the same context, pay nine times as much per sentence, and wait nine times as long for an answer. GPT-4o’s and Gemma 3’s much larger, multilingual vocabularies bring the same sentence down to 54 and 45 tokens. The gap has shrunk a lot but not closed: in the newest tokenizers Greek still costs about twice as much as English, and Hindi and Japanese between 1.3 and 2.8 times, depending on the tokenizer.

The number of tokens per word is sometimes called a tokenizer’s fertility. Petrov et al. (2023) measured it across languages and found differences of up to 15 times for the same content in early tokenizers.

Choosing a size today

There is no single right answer, but the pattern in 2025–2026 models is clear:

Small models in families with huge vocabularies, such as Gemma 3 1B, pay a large share of their parameters for it, and tie their input and output embeddings to halve the cost.