Probability for LLMs

A language model is a probability machine: it gives every next token a probability, and its training papers are written in the language of probability. This course teaches that language from fractions up. Entropy and KL are in Maths for ML; the notation page is a cheat sheet for reading papers.

Foundations

Describing randomness

Learning from data

Estimating with samples

Sampling from models

Inference and structure

Uncertainty and evaluation

Reference