Transformers
How a transformer turns a list of tokens into the next one, taken apart piece by piece with every matrix shown. The matrix maths has its own course, Linear algebra for LLMs; Maths for ML covers softmax and cross-entropy.
The big picture
Tokenization
why-tokens- Characters, words or subwords?bpe- Merge the most frequent pair, repeat.bytes- UTF-8, byte-level BPE and byte fallback.wordpiece-unigram- Greedy longest-match, and best paths.tokenizer-quirks- Why models miscount letters and digits.vocab-size- Bigger vocabularies: shorter text, more parameters.special-tokens- BOS, EOS, padding and chat formats.beyond-tokens- Bytes, patches and learned chunks.