Bytes and Unicode

Key idea

Every character is a Unicode number stored as 1–4 UTF-8 bytes. Byte-level tokenizers start from the 256 bytes: nothing is unknown, but rare scripts are spelled out byte by byte

Real world

Phone numbers. A local call needs a few digits; an international call to a distant country needs a long prefix first. Every phone in the world can be reached, but some cost more digits to dial.

In an LLM

“😀” is code point U+1F600, stored as 4 bytes: F0 9F 98 80. GPT-2 reads “ 😀” as 2 tokens, Llama 2 as 5, Gemma as 1. And the same text costs GPT-2 about 8 times as many tokens in Hindi as in English.

Characters are numbers

A computer stores text as numbers. Unicode is the agreed list: it gives every character in every writing system a number, its Code point, written U+ and four to six hex digits. “A” is U+0041, “é” is U+00E9, “東” is U+6771, “😀” is U+1F600. Unicode 17.0 (2025) assigns 159,801 characters, out of room for 1,114,112.

Code points still have to be stored as bytes, and a byte only holds a number from 0 to 255. The standard encoding is UTF-8. It uses 1 byte for the 128 ASCII characters (English letters, digits, basic punctuation), 2 bytes for most other European, Greek, Cyrillic, Hebrew and Arabic letters, 3 bytes for most Asian scripts including Chinese, Japanese, Korean and Hindi, and 4 bytes for emoji and rarer characters.

Try the emoji family. It looks like one character, but it is five code points: woman, an invisible joiner, woman, joiner, girl. That is 18 bytes. A flag is two code points, “regional indicator” letters G and B, that your screen draws as one picture. Tokenizers never see the picture, only the bytes.

UTF-8’s design has two useful properties. ASCII text is unchanged, byte for byte, so old English files are already valid UTF-8. And the first bits of every byte say whether it starts a character or continues one, so a program can always find where characters begin.

Why not start from characters?

BPE, as on the previous page, starts from single characters. But which characters? There are almost 160,000 of them, most very rare. A vocabulary that included them all would spend most of its rows on symbols that hardly ever appear, and one that left them out would turn every missing character into an unknown token.

The fix in GPT-2 was to run BPE on bytes instead.

Byte-level BPE

Byte-level BPE starts from the 256 possible byte values, not from characters. Text is converted to UTF-8 first, and merges are learned over byte sequences. Since every possible text is some sequence of bytes, and all 256 bytes are in the vocabulary, nothing can ever be unknown. Common byte sequences, like “ the” or the three bytes of a frequent Chinese character, are merged into single tokens. Rare ones stay as separate bytes.

There is one practical wrinkle. BPE implementations, and the merges file they save, work with text strings, and many bytes are spaces, control codes or not valid characters on their own. So GPT-2 maps each of the 256 bytes to a printable character first. Bytes that are already printable keep their own character; the other 68 are moved to letters from U+0100 upwards. The space, byte 20 in hex, becomes Ġ, and the newline becomes Ċ. That is why GPT-2’s vocabulary file is full of tokens like “Ġthe”.

Now watch real text go through all three stages. Notice that a token can end half-way through a character.

“ 😀” is the space plus 4 bytes. GPT-2 has a merge for the space plus the first three bytes (shared by many emoji that start F0 9F 98), but none for the last one, so the emoji is split across two tokens, neither of which is a character on its own. “ 東京” fares worse: five tokens (space included) for two characters. The model has to learn to put such pieces back together.

Byte fallback

SentencePiece tokenizers, used by Llama 2, Mistral and Gemma, take a middle road. They build the vocabulary from characters, covering at least 99.95% of the characters in the training data, plus 256 extra tokens <0x00> to <0xFF>, one per byte. A character missing from the vocabulary is written as its UTF-8 bytes using those tokens. This is Byte fallback: nothing is unknown, but byte tokens are only used when needed. (More on SentencePiece in WordPiece and Unigram.)

Here is how six real tokenizers handle “ 😀”:

Why some languages cost more

Put the two effects together. A Hindi character takes 3 bytes against 1 for an English letter, and a tokenizer trained mostly on English text learned few merges for Hindi byte sequences. So the same meaning takes many more tokens.

With GPT-2, Article 1 is 33 tokens in English and 296 in Hindi, for the same sentence. Tamil and Burmese come out at about one token per byte: GPT-2 learned almost nothing about them. Newer tokenizers with larger, more multilingual vocabularies close much of the gap: GPT-4o brings Hindi down to 56 tokens. But no tokenizer here is fair to every language.

This matters in practice. Paid APIs charge per token. A context window holds fewer pages. Generation is slower, since it produces one token at a time. And a model has fewer chances to learn from text that is cut into tiny, meaningless pieces. Choosing a bigger vocabulary is the main remedy.

def bytes_to_unicode():
    bs = list(range(ord("!"), ord("~") + 1)) + list(range(ord("¡"), ord("¬") + 1)) + list(range(ord("®"), ord("ÿ") + 1))
    cs = bs[:]
    n = 0
    for b in range(256):
        if b not in bs:          # space, control codes, etc.
            bs.append(b)
            cs.append(256 + n)   # move to Ā, ā, Ă, ...
            n += 1
    return dict(zip(bs, map(chr, cs)))