PTLearning

Special tokens and chat templates

Some tokens are not text at all: they mark the start and end of a text, fill out batches, and separate the turns of a conversation. A chat is turned into one flat stream of tokens by a chat template, and the model only behaves like an assistant if that template matches the one it was trained on.

Basics used here
Key idea
Reserved tokens for structure (BOS, EOS, PAD, UNK, role markers); a chat template flattens messages into one token stream; the model only follows the format it was trained on
Real world
A screenplay. The actors’ names in capitals and the stage directions in brackets are not lines anyone speaks, but without them you couldn’t tell who says what or when a scene ends.
In an LLM
When you chat with a Llama 3 model, your message is wrapped as <|start_header_id|>user<|end_header_id|> … <|eot_id|>, and the reply starts after an assistant header. When the model generates <|eot_id|> (ID 128009), the app stops.

Tokens that aren’t text

Besides the tens of thousands of text pieces, every vocabulary reserves a few IDs for . They never come out of tokenizing ordinary text: they are inserted by code. The classic set:

  • BOS (beginning of sequence) is put at the start of every text: <|begin_of_text|> (ID 128000) in Llama 3, <bos> (2) in Gemma.
  • EOS (end of sequence) marks the end, and the model generates it to stop: <|endoftext|> (50256) in GPT-2.
  • PAD (padding) fills short rows in a batch: <pad> (0) in Gemma.
  • UNK (unknown) stands for text the vocabulary can’t represent: [UNK] in BERT. Byte-level tokenizers never need it.

The matters most. The model has no other way to say “I’m done”: generation is a loop that would otherwise run until it hits a length limit. During training, EOS is put between documents, so the model learns to predict it where texts end. GPT-2 even uses the same token, <|endoftext|>, as BOS and EOS.

The unknown token belongs to older tokenizers. Word-level and WordPiece tokenizers map anything outside their vocabulary to UNK, losing it (see WordPiece and Unigram). Byte-level tokenizers can always fall back to raw bytes, so modern models effectively never produce UNK.

Padding: making batches rectangular

GPUs process many sequences at once as one big matrix, which must be a rectangle. Texts have different lengths, so short ones are filled with a and a mask tells the model to ignore it:

BOSThe·cat·sat.EOSPADPAD
11111100
BOSHello!EOSPADPADPADPAD
11110000
BOSTokens·in,·tokens·out.EOS
11111111
Right padding is the usual choice for training: every text starts at position 0, and the padded positions are masked out of attention and out of the loss. The <EOS> token is learned: it is how the model says “I’m done”. <BOS> gives every text the same first token, a fixed place attention can always fall back on.
Three texts of different lengths in one batch. A GPU wants a rectangle, so short rows are filled with <PAD>. The mask underneath (1 = real, 0 = padding) tells attention and the loss to ignore the padding.

The padding token’s ID barely matters, because the attention mask hides it from every other token and the loss ignores its positions. Many models simply reuse EOS as padding. What matters is that the mask is right: a padding token that leaks into attention changes the outputs of real tokens.

Chats are one stream of tokens

A chatbot looks like a conversation of messages, but the model only reads one sequence of tokens and predicts the next one. A is the recipe that flattens the messages into that sequence. Each model family has its own, with role names and special tokens that mark where each message starts and ends:

Loading GPT-2 data…

Three things stand out:

  • The structure is made of special tokens. In Qwen’s format (called ChatML), <|im_start|> and <|im_end|> are single tokens with their own IDs (151644, 151645). The role name after them (system, user, assistant) is ordinary text.
  • Templates add text you didn’t write. The Llama 3.1 and later templates write a knowledge cutoff date and today’s date into the system message. gpt-oss gets a whole system message with its identity and reasoning settings, and your system prompt becomes a “developer” message. Gemma 3 has no system role at all: the system prompt is pasted into the start of the first user message.
  • Generation starts at an open assistant header. The “Waiting for the reply” view is what the model actually receives: everything up to and including the start of the assistant turn. The model writes the reply and then the end-of-turn token, and the app stops.

Tool calls are text in a format

When a model “calls a tool”, it generates tokens in a format the app recognises. Qwen writes JSON between <tool_call> and </tool_call> tokens; gpt-oss addresses a message to functions.get_weather in a separate channel and ends it with <|call|>. The app stops generation, parses the call, runs the function, and appends the result as a new message (role tool, ipython or functions.get_weather, depending on the template). Then the model continues. The tool descriptions themselves are injected as text, usually into the system message, which is why long tool lists use up context.

example · radio procedure

Pilots and air traffic control use fixed words: the call sign first, then the message, then “over” when they expect a reply. Anyone trained in the procedure knows exactly when to speak. A pilot trained on one procedure and spoken to in another hesitates, interrupts, or waits for an “over” that never comes.

example · the wrong template

Send Llama 3 a conversation in Qwen’s ChatML format. Llama 3’s vocabulary has no <|im_start|> token, so the text “<|im_start|>” is cut into 6 ordinary tokens: “<”, “|”, “im”, “_start”, “|”, “>”. The model has never seen a conversation look like this. It may continue your message instead of answering it, invent the next user turn, or never produce <|eot_id|> and keep going until the length limit.

Why the template must match training

An instruction-tuned model learned its assistant behaviour from many conversations in exactly one format. Its knowledge of when to answer, when to stop and which voice to use is tied to those particular tokens: what normally follows <|start_header_id|>assistant<|end_header_id|> and two newlines is a helpful answer, ending in <|eot_id|>. Change one detail and the model is slightly out of distribution:

  • A missing newline after the header changes the tokens the reply starts from.
  • A missing BOS token removes the position many attention heads use as a resting place (an attention sink).
  • A base (not instruction-tuned) model may have the chat tokens in its vocabulary but never trained. In Llama 3 base models the chat tokens such as <|start_header_id|> and <|eot_id|> are in the vocabulary, but their embeddings were reportedly never trained, the same as the reserved tokens nobody uses. Fine-tuning a base model on chats therefore has to teach them from scratch.

This is why the template is shipped with the model, as a small Jinja program in its tokenizer configuration, and why inference libraries call it for you (tokenizer.apply_chat_template in Hugging Face transformers). Small mismatches cause real bugs: wrong stop tokens that make a model ramble, or a lost system prompt.

Special tokens and safety

Because special tokens carry structure, ordinary text must never be able to produce them. If a user types the characters “<|im_end|>” into a chat box, a correct app tokenizes them as plain text (several ordinary tokens), not as the special token. Otherwise the user could close their own message and write a fake system or assistant turn. Tokenizer libraries guard against this: tiktoken, for example, refuses to encode text containing special-token strings unless you explicitly allow it.

going deeper

The chat template as code

The Llama 3 template, simplified:

{{- bos_token }}
{%- for message in messages %}
  {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n' }}
  {{- message['content'] | trim + '<|eot_id|>' }}
{%- endfor %}
{%- if add_generation_prompt %}
  {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
{%- endif %}

In Python:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
msgs = [{"role": "user", "content": "What is the capital of France?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True)
print(tok.decode(ids))

Training only on the reply

When fine-tuning on chats, the loss is usually computed only on the assistant’s tokens (including the end-of-turn token), not on the system and user messages. The model learns to write replies and to stop, not to imitate users. The mask that does this is the same idea as the padding mask above.

In papers and docs

Related

  • tokenizer-quirks - A model never sees letters or digits, only token IDs. That one fact explains a surprising list of failures: shaky arithmetic, miscounted letters, words that change meaning with a space or a capital letter, and “glitch tokens” that made GPT-3 behave strangely. Here are real examples from seven tokenizers.
  • bytes - Computers store text as Unicode code points packed into UTF-8 bytes, 1 to 4 per character. Byte-level BPE starts from the 256 possible bytes, so no text is ever unknown. The price is that scripts with more bytes per character, and less training data, cost many more tokens.
  • what-an-llm-does - A language model does one thing. It reads a list of tokens and gives a probability for every possible next token. Generation is that one step in a loop. Pick a token, append it, run again.
  • causal-mask - A language model must predict each token without seeing it. The causal mask is a triangle of −∞ added to the attention scores so that every token can only look at itself and earlier tokens. It also lets training check a prediction at every position of a text in one pass.