Why tokens?

A model needs a fixed list of units to read and write. Characters make the text very long; whole words need a huge list and still meet words they have never seen. Subword tokens sit in between, and every modern LLM uses them.

Key idea: Characters: short list, long text. Words: short text, huge list, unknown words. Subwords: common words whole, rare words in pieces
Real world: A box of Lego. Single studs can build anything but take forever. Pre-made walls are quick but you can only build the houses they were made for. A real set has big common bricks and small ones for the odd corner.
In an LLM: GPT-2 has 50,257 tokens. “ the” and “ cat” are one token each; “ Gryphon” is three (“ Gry”, “ph”, “on”); and there is no text it cannot represent.

A model reads a list of numbers

A neural network does arithmetic on numbers, not on letters. So before a model can read text, the text has to be cut into units, and each unit replaced by its number in a fixed list. That list is the vocabulary, and each unit in it is a .

The vocabulary is not a detail. Every entry gets its own row in the embedding table at the bottom of the model and its own score in the unembedding at the top, so its size sets the cost of both ends of the map. And the way the text is cut decides how many tokens the model has to read, which sets the cost of everything in between.

There are three obvious ways to cut text.

Characters

Use every character as a token. The vocabulary is small and nothing is ever unknown: any word, name or typo can be spelled out.

The problem is length. English averages about five characters per word, plus a space, so a page of text becomes thousands of tokens. That hurts twice:

Cost. Attention compares every token with every other one, so its work grows with the square of the length: five times as many tokens is about 25 times the attention cost.
Context. A model’s context window is measured in tokens. At one token per character, a 4,096-token window holds less than two pages.

Each character also means very little by itself, so the model must spend layers assembling letters into words before it can think about meaning.

Words

Use every word as a token. Text gets short: one token per word.

But English alone has hundreds of thousands of words, before counting names, numbers, typos, code and every other language. A vocabulary has to stop somewhere, and every word that did not make the list becomes a single unknown token, . The model cannot tell one unknown word from another, and it can never write one.

Words also hide structure. “run”, “runs”, “running” and “rerun” are four unrelated entries, each learned from scratch.

Subwords

The compromise: keep common words whole, and cut rare words into smaller pieces that are themselves in the vocabulary. A vocabulary of 30,000 to 250,000 pieces covers any text. Common words cost one token, rare words a few, and nothing is ever unknown, because in the worst case a word can be spelled out character by character (or byte by byte).

Type anything below. The word row uses a fixed list of the 20,000 most common words from ten novels; the bottom row is GPT-2’s real tokenizer.

show token IDs

characters · 59

The·Gryphon·and·the·Mock·Turtle·danced·a·Lobster·Quadrille.

words · 11

TheGryphonandtheMockTurtledancedaLobsterQuadrille.

GPT-2 subword tokens

Loading GPT-2 data…

characters59

words11

GPT-2 subwords0

sequence length: how many tokens the model has to read

Loading GPT-2’s merges and the word list…

The same text as characters, as words from a fixed 20,000-word list, and as GPT-2’s real tokens. Edit the text or pick an example. A · marks a space. Tick “show token IDs” to see the numbers the model actually reads.

Try the rare words example. Words win on length, but “Gryphon”, “Mock”, “Turtle”, “Lobster” and “Quadrille” are not in the word list, so a word-level model would read “The [UNK] and the [UNK] [UNK] danced a [UNK] [UNK]”. GPT-2 spends a few extra tokens and keeps every letter. With typos, the word list fails on four of the six words, while GPT-2 just uses more, smaller pieces: “Tokenizaton” becomes “Token”, “iz”, “aton”.

The trade-off, measured

Here is the trade-off on a whole book. The slider sets the vocabulary size. For words, a bigger vocabulary means fewer unknowns. For subwords, a bigger vocabulary means fewer tokens per word. Bytes are the extreme: 256 entries, nothing unknown, five and a half tokens per word.

Loading GPT-2 data…

The trade-off, measured.

The two charts tell the story. The word curve in the bottom chart never reaches zero: even 38,181 words learned from 2 million words of other novels leave 2.5% of Alice unknown, because Alice is full of Gryphons, Dormice and treacle. The BPE line in the top chart drops fast and then flattens: the first few thousand merges capture common words and endings; the next 40,000 buy only a little more. That flattening is why vocabularies stopped at tens of thousands of tokens for years, and why choosing the size is a real decision.

example · Lego

You could build a house from single 1×1 bricks: anything is possible, but it takes thousands of pieces. A kit of pre-built walls is fast, but only builds the houses the kit was designed for. Real Lego sets mix large common bricks with small ones for the odd corner. Subword tokenizers are the same: common words are big bricks, and anything unusual is built from small ones.

example · a new word

The word “ChatGPT” did not exist when GPT-2’s tokenizer was trained in 2019. A word-level model from 2019 would read it as [UNK]. GPT-2 reads it as “Chat”, “G”, “PT”: it has never seen the word, but it can still read and write it from pieces, and learn what it means from context.

Kinds of subword tokenizer

Every subword tokenizer answers two questions: which pieces go in the vocabulary, and how a new text is cut into those pieces. The main answers, each on its own page:

BPE (byte-pair encoding): build the vocabulary by repeatedly merging the most frequent pair of adjacent pieces. Used by GPT-2, GPT-4, Llama and most modern LLMs. See BPE.
Byte-level tokenizers start from the 256 possible bytes rather than characters, so that no text in any language is unknown. See bytes and Unicode.
WordPiece (BERT) and Unigram (T5, ALBERT) choose pieces by likelihood rather than raw counts; Unigram even keeps a probability for every way of splitting a word. See WordPiece and Unigram.

Page 06 has a table that sets BPE, WordPiece and Unigram side by side, and a widget that cuts one word all three ways.

After that, the section covers the strange side effects of tokenization, how big the vocabulary should be, special tokens and models that skip tokens altogether.

going deeper

Measuring a tokenizer

The usual measure is compression: characters (or bytes) per token on some reference text. Higher means shorter sequences. On the Alice text above, GPT-2 averages $151{,}099 / 44{,}528 \approx 3.4$ bytes per token; on ordinary web English it is closer to 4. The Llama 3 report gives 3.17 characters per token for Llama 2’s tokenizer and 3.94 for Llama 3’s on the same English data.

Sequence length matters because of the cost of a forward pass: for $T$ tokens and width $d$ , the matrix multiplications cost about $2 \cdot N_{\text{params}} \cdot T$ operations and attention about $4 \cdot T^2 \cdot d$ per layer. Fewer tokens per text means less of both.

How the numbers above were made

Test text: the full Project Gutenberg text of Alice’s Adventures in Wonderland (27,441 words, 151,099 bytes).
Words: split with the regular expression \w+|[^\w\s] (words and single punctuation marks), case-sensitive. The vocabulary is the $V$ most frequent words in ten other Gutenberg novels (2,002,106 words, 38,181 distinct).
BPE: GPT-2’s merges, truncated to the first $V - 256$ , applied with GPT-2’s own pre-tokenizer. With all 50,000 merges the result is identical to GPT-2’s tokenizer.

In papers

Subword units for neural networks: Sennrich, Haddow and Birch, 2016, which introduced BPE for translation to remove the unknown-word problem.
A survey of tokenization from words to bytes: Mielke et al., 2021.
Llama 3’s tokenizer compression figures: Grattafiori et al., 2024, section 3.2.