Tokenizer quirks

Key idea

The model sees token IDs, not characters: numbers split irregularly, letters are hidden inside tokens, “cat” ≠ “ cat” ≠ “Cat”, and rare tokens can be left untrained

Real world

Reading a sentence printed as a row of shorthand symbols. You can learn that one symbol means “strawberry”, but if someone asks how many r’s it has, you have to remember the spelling, because you can’t see it.

In an LLM

In GPT-4o’s tokenizer “ strawberry” is a single token, ID 101830 when it follows a space. The model gets that one number. Counting the r’s means recalling, from training, which letters that number stands for.

The model sees IDs, not letters

The BPE page showed how a tokenizer cuts text into pieces from a learned list, and bytes showed how every text can be cut that way. What reaches the model is a list of ID numbers, one per piece. Each ID is turned into a vector by looking up a row of a table (the embeddings page shows how). Nothing in that vector says how the piece is spelled, how many characters it has or which digits it holds. The model can only learn such facts indirectly, from how tokens are used.

Most of the time that does not matter: the model learns that “ strawberry” is a red fruit, which is what you usually need. But whenever a task depends on the characters themselves, the tokenizer gets in the way. Below are seven real tokenizers, from GPT-2 (2019) to Qwen3.5 (2026), on the same texts.

Numbers

Digits are where tokenizers differ most, and there are three families:

• Learned merges (GPT-2, GPT-3). Number tokens are whatever BPE found frequent: “ 2024” is one token because years are common, while 1234567 becomes 123|45|67. The same digits can split differently depending on their neighbours.
• Groups of up to three, from the left (GPT-4, GPT-4o, Llama 3, DeepSeek-V3). Before BPE runs, the Pre-tokenization step cuts every run of digits into chunks of at most 3. All 1,000 three-digit strings are in the vocabulary, so 1234567 is always 123|456|7.
• One digit per token (Llama 2, Gemma 3, Qwen3.5). Simple and regular, but a 12-digit number costs 12 tokens.

To add two numbers on paper you line up the digits by place value, starting from the right. Chunks counted from the left don’t line up when the numbers have different lengths. Try it:

Spelling: the strawberry problem

“How many r’s are in strawberry?” became famous in 2024 because many chatbots answered 2. The tokens explain why the question is hard. In GPT-4o, “ strawberry” after a space is one token. Without the space, it is st|raw|berry: three pieces, none of which is “just the r’s”. Either way the model has to recall the spelling of an ID rather than look at letters.

Models do learn a lot of spelling, because text contains spelled-out words, rhymes, acrostics and typos. But it is knowledge about the tokens, not perception of letters, and it is patchy for rare tokens. Today’s reasoning models often get the answer right by writing the word out letter by letter first: then each letter is its own token, and counting becomes easy. Changing the input so that the information is visible in the tokens is the same trick as the commas above.

Spaces and capitals

The leading space is part of the token. In GPT-2, “cat” at the start of a text is ID 9246, but “ cat” in the middle of a sentence is ID 3797, and “ Cat” is 5181. These are three unrelated rows in the embedding table; the model has to learn from data that they mean the same thing.

That has practical effects:

• Prompts that end in a space. If your prompt ends with “The answer is ” (with a trailing space), the next token must be one that starts without a space, like “4” rather than “ 4”. Those tokens are rarer in that position, so the model can behave oddly. This is why prompt-writing guides warn against trailing spaces.
• Capitals and shouting. “hello” and “ Hello” are single tokens, but “ HELLO” splits into “ HEL|LO” in GPT-4’s tokenizer. Text in capital letters takes more tokens and uses rarer ones.
• Llama 2 adds a space at the start. Its SentencePiece tokenizer puts a space in front of the whole text, so “cat” at the very start becomes “ cat”, the same token as in mid-sentence. A text that really starts with a space then gets an extra space token: the lone · chip in the gallery.

Glitch tokens

In early 2023 Jessica Rumbelow and Matthew Watkins found a set of strange tokens in the vocabulary shared by GPT-2 and GPT-3, including “ SolidGoldMagikarp”, “ petertodd”, “ TheNitromeFan” and “ davidjl”. When asked to repeat “ SolidGoldMagikarp”, ChatGPT answered “distribute”. Other such tokens produced evasions, insults or nonsense, and some broke determinism even at temperature 0.

The explanation most people accept is a mismatch between two datasets:

1. The tokenizer was trained on one big scrape of web text. Some Reddit usernames (several from r/counting, a forum where people count to infinity one comment at a time) appeared there thousands of times, so BPE gave each of them a single token.
2. The model was trained on a different, filtered dataset where those strings almost never appeared.

A token that never appears in training never gets a useful embedding. Its row in the table stays close to where initialisation and weight decay left it, and the model has no idea what to do when it shows up. These are now called glitch tokens or under-trained tokens. Land and Bartolo (2024) built methods to find them automatically and found them in many open models.

You can see traces of this in GPT-2’s own embedding table. Among its 50,257 tokens, the ones whose embeddings sit closest to the average embedding include unused control characters, and strings like “ externalToEVA”, “quickShip”, “embedreportprint” and “ TheNitrome”: fragments of web-page boilerplate and usernames that were common when the tokenizer was built and rare in the training text.

Newer tokenizers split these strings into ordinary pieces (the gallery shows “ Solid|Gold|Mag|ik|arp”), and model builders now check for under-trained tokens. But any token that is rare in training is a weak spot; vocabulary size is partly a decision about how many rare tokens you can afford.

Code and whitespace

GPT-2 had one token per space, so a line indented by 12 spaces cost 11 tokens of indentation. Tokenizers built after code became a major use (GPT-4 onwards) have tokens for runs of spaces, for common symbols like “):” followed by a newline, and for combinations such as “(a”. The gallery’s Python example drops from 16 tokens in GPT-2 to 12 in GPT-4. Shorter sequences mean more code fits in the context and each step of generation does more.