PTLearning

Beyond tokens

Tokenizers cause most of the quirks on the previous pages, so researchers keep trying to remove them. Byte-level models read raw bytes; the Byte Latent Transformer groups bytes into patches where the text gets hard to predict; H-Net learns its own chunks. They now match tokenized models in experiments, but every major production model still uses a tokenizer.

Basics used here
Key idea
Feed bytes instead of tokens; to keep sequences short, group bytes into patches (fixed, by entropy, or learned) so the big model runs once per patch, not once per byte
Real world
Reading aloud. You glide through a predictable phrase like “once upon a time” in one breath, and slow down at a strange name where you can’t guess what comes next.
In an LLM
In “George R.R. Martin”, a small byte model is unsure at the “G”, the “R” and the “M”, and sure about the letters that follow. The Byte Latent Transformer starts a new patch at each uncertain byte, so the big model spends one step per chunk of predictable text.

Why get rid of the tokenizer?

The tokenizer is the one part of a language model that is not learned end to end. It is built once, before training, by a separate algorithm, and then frozen. The quirks page showed what that costs: digits split irregularly, letters hidden inside tokens, rare tokens left untrained, and some languages paying several times more per sentence. A model that read raw bytes would have none of these problems: 256 possible inputs, every text representable, every letter visible.

The obstacle is length. English averages about 4.5 bytes per token in modern tokenizers, so a byte-level model has 4.5 times as many steps, and attention, which compares every position with every other, costs about 20 times as much. The history of tokenizer-free models is a series of ways around that.

Every byte: ByT5

ByT5 (Xue et al., 2021) took Google’s T5 and fed it UTF-8 bytes directly. It was more robust to typos and noise and better at spelling-sensitive tasks, and it was competitive with the token version at small and medium sizes. But it was slower to train and run, and it moved most of its parameters into a deep encoder to cope with long inputs. It showed that bytes work, not that they are cheap.

Fixed patches: MEGABYTE

MEGABYTE (Yu et al., 2023) cut the byte stream into fixed patches, for example 4 or 8 bytes. A large global model runs once per patch; a small local model predicts the bytes inside each patch. The big model now does a quarter or an eighth of the steps. The weakness is that fixed patches ignore the text: a word is cut wherever the count falls, and a predictable run of spaces gets as much compute as a hard-to-guess name.

Patches by surprise: the Byte Latent Transformer

The Byte Latent Transformer (BLT, Pagnoni et al., 2024) makes the patches depend on the text. A small byte-level language model reads the input and, at every byte, measures its uncertainty about the next byte: the entropy of its prediction, in bits. Where the entropy is high, a new patch starts. This is called .

Loading GPT-2 data…

The pattern is the same in every sentence: entropy spikes at the first letter of a word, at names, digits and codes, and falls within words, where the rest of the letters follow. So patches tend to start at word boundaries and stretch over predictable text, much like tokens, but they are decided per text rather than looked up in a fixed list, and the threshold is a dial for how much compute to spend.

BLT has three parts:

  1. A small local encoder turns the bytes of each patch into one vector.
  2. A large global transformer, the bulk of the parameters, runs once per patch.
  3. A small local decoder turns each patch vector back into bytes, predicting them one at a time.

In a compute-controlled comparison, BLT matched Llama 3’s tokenizer-based training at up to 8 billion parameters and 4 trillion bytes, while being more robust to noisy input and better at character-level tasks. Its key argument is about scaling: because average patch size is adjustable, at a fixed inference budget you can make patches longer and spend the saved steps on a larger global model.

bytes (ByT5)4000 steps · attention ×20.3
BPE tokens (4.5 bytes)889 steps · attention ×1.0
patches (6 bytes)667 steps · attention ×0.6
attention × is relative to BPE tokens
Patches of 6.0 bytes mean 25% fewer big-model steps than BPE tokens. BLT’s point is that you can spend the saving on a bigger model at the same cost, since patch size, unlike a fixed vocabulary, is a dial. The small byte models still run on every byte, but they are tiny.
A page of English, about 4,000 bytes. Each bar is the number of steps the big model must take: one per unit. Attention compares every unit with every other, so its cost grows with the square.

Learning the chunks: H-Net and since

BLT still uses a hand-chosen rule (an entropy threshold) to place boundaries. H-Net (Hwang, Wang and Gu, 2025) learns them. A routing module inside the network decides, from the content, where one chunk ends and the next begins, and the whole model, chunking included, is trained end to end on bytes. The authors report that a byte-level H-Net beat a BPE-tokenized transformer of the same compute, that a two-stage version matched a tokenized transformer twice its size, and that the gains were largest where tokenizers are weakest: Chinese, code and DNA.

Other recent work, all 2025:

  • AU-Net (Videau et al., NeurIPS 2025) builds a U-Net over bytes: bytes are pooled into word-level and then multi-word vectors and expanded back, and deeper stages predict further ahead. It matched BPE baselines at equal compute.
  • SuperBPE (Liu et al., COLM 2025) goes the other way: keep the tokenizer, but let BPE learn tokens that cross spaces, like “by the way”. With a 200k vocabulary it used up to 33% fewer tokens than plain BPE, and an 8B model trained with it averaged 4 points better on 30 tasks with 27% less inference compute.
  • Bolmo (Minixhofer et al., Ai2, December 2025) “byteifies” an existing tokenized model: it keeps the trained transformer as the global model, adds a byte-level encoder, decoder and boundary predictor, and retrains briefly. Its 7B version, built from Olmo 3, beat BLT’s 7B model, by a wide margin on science and maths questions, and the conversion cost under 1% of a normal pretraining run.
example · speed reading

A fast reader doesn’t look at every letter. Their eyes jump in chunks across common phrases and slow down on unfamiliar names, numbers or technical words. A child sounding out letters reads every byte; a speed reader with a fixed rhythm is MEGABYTE; a good reader who slows down only where the text surprises them is BLT.

example · counting the r’s again

A tokenized model sees “ strawberry” as one ID and has to remember its spelling. A byte-level model sees s, t, r, a, w, b, e, r, r, y: ten bytes, each visible to its local encoder. That is why byte-level models consistently do better on spelling, character counting and typo-heavy text, and why BLT, H-Net and Bolmo all report big gains on character-level benchmarks.

What is mainstream

Be careful not to over-read the research. As of late 2026, every widely used frontier and open model, from GPT-5 and Claude to Llama 4, Qwen3.5, DeepSeek-V3 and Gemma 4, still uses a subword tokenizer, typically byte-level BPE with 128k to 262k tokens. Byte-level and learned-chunking models have matched tokenized ones in controlled experiments up to around 8B parameters, and they are clearly better at spelling, noisy text and some languages. But they have not yet been shown to be better at the largest scales, and inference tools, caching and decoding are all built around tokens. The likeliest path is the one Bolmo took: converting strong tokenized models rather than replacing them.

going deeper

Entropy patching precisely

Let a small byte model give p(xt∣x<t)p(x_t \mid x_{\lt t}). Its entropy at byte tt is

H(xt)=−∑v=0255p(v∣x<t)log⁡2p(v∣x<t).H(x_t) = -\sum_{v=0}^{255} p(v \mid x_{\lt t}) \log_2 p(v \mid x_{\lt t}).

BLT tried two rules for starting a new patch at byte tt:

  • Global threshold: H(xt)>θgH(x_t) > \theta_g.
  • Approximate monotonic constraint: H(xt)−H(xt−1)>θrH(x_t) - H(x_{t-1}) > \theta_r, a jump in entropy.

The widget’s byte model is a 6-layer, 11M-parameter transformer we trained on eight public-domain novels (5.6 MB); it reaches about 1.65 bits per byte on held-out text. BLT’s entropy model was a 100M-parameter transformer trained on its pretraining data, so its curves are sharper, but they have the same shape.

In papers

Related

  • bytes - Computers store text as Unicode code points packed into UTF-8 bytes, 1 to 4 per character. Byte-level BPE starts from the 256 possible bytes, so no text is ever unknown. The price is that scripts with more bytes per character, and less training data, cost many more tokens.
  • vocab-size - A bigger vocabulary packs more text into each token, so sequences get shorter and every language gets cheaper. But each extra token adds a row to the embedding table and a score the model must compute at every step, and rare tokens get too little training. Models moved from 32k–50k tokens to 128k–262k as they grew.
  • tokenizer-quirks - A model never sees letters or digits, only token IDs. That one fact explains a surprising list of failures: shaky arithmetic, miscounted letters, words that change meaning with a space or a capital letter, and “glitch tokens” that made GPT-3 behave strangely. Here are real examples from seven tokenizers.
  • efficient-attention - Attention’s two costs are the KV cache, which grows with every token, and the score matrix, which grows with the square of the length. Modern models share keys and values between heads (GQA, MLA), let most layers look only at recent tokens, compute attention in tiles that never store the full matrix (FlashAttention), and pick a few keys to read per query (sparse attention).