Reading the notation

A cheat sheet for the probability notation in LLM papers. Every symbol, how to read it aloud, and the formulas you'll meet most often, each linked to the page that explains it.

Papers compress a lot into a little notation. Most of it is the same few dozen symbols. Keep this page open next to a paper.

Symbols

Notation	Read as	Means
$P(A)$ , $p(x)$	"probability of A", "p of x"	Probability of an event, or of a value $x$ . Capital $P$ for events, small $p$ for distributions
$p(y \mid x)$	"p of y given x"	Probability of $y$ when $x$ is known: conditional
$p(x, y)$	"p of x and y"	Joint probability of both
$x \sim p$	"x drawn from p"	$x$ is a random sample from distribution $p$
$x_1, \dots, x_n \overset{\text{iid}}{\sim} p$	"independent and identically distributed"	$n$ independent samples from the same $p$ : independence
$\mathbb{E}_{x \sim p}[f(x)]$	"expectation of f under p"	Probability-weighted average of $f$ : expected value
$\operatorname{Var}[X]$ , $\sigma^2$	"variance"	Average squared distance from the mean: variance
$\propto$	"proportional to"	Equal up to a constant that makes things add to 1
$Z$	"partition function", "normaliser"	That constant: $Z = \sum_y \tilde p(y)$
$\pi_\theta$	"pi theta"	The policy (the model) with weights $\theta$
$\pi_\text{ref}$ , $\pi_{\theta_\text{old}}$	"reference", "old policy"	The model before RL; the model that generated the batch
$\mu$	"behaviour policy"	Whatever actually generated the samples (often the inference engine)
$q$ , $o$ , $y$	"query", "output"	A prompt and a generated answer; $o_{i,t}$ is token $t$ of answer $i$
$y_{\lt t}$ , $o_{i,\lt t}$	"y before t"	All tokens before position $t$
$\lvert y \rvert$	"length of y"	Number of tokens in the answer
$\mathbb{1}[\cdot]$	"indicator"	1 if the condition is true, else 0
$\hat\theta$ , $\hat A$	"theta hat"	An estimate of $\theta$ or of the advantage, computed from samples
$\nabla_\theta$	"gradient with respect to theta"	Direction in weight space that increases the quantity fastest
$\mathrm{sg}[\cdot]$ , $\operatorname{stopgrad}$	"stop gradient"	Treat this as a constant when differentiating
$\mathrm{KL}(p \Vert q)$ , $\mathbb{D}_\text{KL}$	"KL from q to p"	KL divergence; order matters: divergences
$\mathcal H(p)$ , $H(p)$	"entropy of p"	Average surprise: entropy
$\sigma(\cdot)$	"sigmoid"	$1/(1 + e^{-x})$ : sigmoid. Also used for standard deviation; context decides
$\mathcal N(\mu, \sigma^2)$	"normal"	Gaussian with mean $\mu$ , variance $\sigma^2$ : common distributions
$\operatorname{Bern}(p)$ , $\operatorname{Beta}(a, b)$	"Bernoulli", "Beta"	A coin flip; a distribution over probabilities: common distributions
$\arg\max_x f(x)$	"arg max"	The $x$ that makes $f$ largest (not the largest value itself)
$\mathcal L$ , $\mathcal J$	"loss", "objective"	What's minimised; what's maximised
$\operatorname{clip}(r, 1-\epsilon, 1+\epsilon)$	"clip"	Keep $r$ inside that range: PPO
$\mathcal O(\cdot)$	"big O"	Grows at most like; ignore constants

Formulas you'll meet

Chain rule for a sequence, and the log-likelihood it gives (chain rule):

p_\theta(y \mid x) = \prod_{t=1}^{\lvert y \rvert} p_\theta(y_t \mid x, y_{\lt t}), \qquad \log p_\theta(y \mid x) = \sum_t \log p_\theta(y_t \mid x, y_{\lt t})

Bayes' rule (Bayes):

p(z \mid y) = \frac{p(y \mid z)\, p(z)}{p(y)} \propto p(y \mid z)\, p(z)

Marginalising a hidden variable (reasoning as a hidden variable):

p(y \mid x) = \sum_z p(z \mid x)\, p(y \mid x, z)

Maximum likelihood = minimising cross-entropy = minimising forward KL to the data (likelihood):

\theta^* = \arg\max_\theta \sum_i \log p_\theta(y_i \mid x_i)

Monte Carlo estimate of an expectation (Monte Carlo):

\mathbb{E}_{x \sim p}[f(x)] \approx \frac1n \sum_{i=1}^n f(x_i), \quad x_i \sim p, \qquad \text{SE} = \frac{\sigma}{\sqrt n}

Importance sampling (importance sampling in depth):

\mathbb{E}_{x \sim p}[f(x)] = \mathbb{E}_{x \sim q}\Big[\frac{p(x)}{q(x)} f(x)\Big]

Policy gradient with a baseline (score function):

\nabla_\theta \mathbb{E}_{y \sim \pi_\theta}[r(y)] = \mathbb{E}_{y \sim \pi_\theta}\big[(r(y) - b)\, \nabla_\theta \log \pi_\theta(y)\big]

GRPO objective, as written in papers (GRPO):

\mathcal J(\theta) = \mathbb{E}\Bigg[\frac{1}{G} \sum_{i=1}^{G} \frac{1}{\lvert o_i \rvert} \sum_{t=1}^{\lvert o_i \rvert} \min\Big(r_{i,t}(\theta) \hat A_{i,t},\ \operatorname{clip}\big(r_{i,t}(\theta), 1 - \epsilon, 1 + \epsilon\big) \hat A_{i,t}\Big) - \beta\, \mathbb{D}_\text{KL}\big[\pi_\theta \Vert \pi_\text{ref}\big]\Bigg]

with

r_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,\lt t})}{\pi_{\theta_\text{old}}(o_{i,t} \mid q, o_{i,\lt t})}, \qquad \hat A_{i,t} = \frac{r_i - \operatorname{mean}(\mathbf r)}{\operatorname{std}(\mathbf r)}

where $r_i$ (no $t$ ) is answer $i$ 's reward. Yes, $r$ means two things in one formula; papers do this.

KL-regularised optimum and DPO (tilted distributions):

\pi^*(y \mid x) = \frac{1}{Z(x)}\, \pi_\text{ref}(y \mid x)\, e^{r(x, y)/\beta}, \qquad \mathcal L_\text{DPO} = -\log \sigma\Big(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_\text{ref}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_\text{ref}(y_l \mid x)}\Big)

Unbiased pass@k (Monte Carlo):

\text{pass@}k = \mathbb{E}_\text{problems}\Big[1 - \binom{n - c}{k} \Big/ \binom{n}{k}\Big]

Speculative decoding acceptance (rejection sampling):

P(\text{accept } x) = \min\Big(1, \frac{p(x)}{q(x)}\Big), \qquad \alpha = \sum_x \min(p(x), q(x)) = 1 - \mathrm{TV}(p, q)

Evidence lower bound (reasoning as a hidden variable, diffusion LMs):

\log p(y) \ge \mathbb{E}_{z \sim q}\big[\log p(y, z) - \log q(z)\big]

Habits of notation

Subscripts on $\mathbb{E}$ tell you what's random. $\mathbb{E}_{y \sim \pi_\theta}$ means the answers come from the current model; $\mathbb{E}_{y \sim \pi_{\theta_\text{old}}}$ means they came from an older one, and an importance ratio must appear somewhere.
Hats mean estimates. $A$ is the true advantage; $\hat A$ is what was computed from a group of samples.
Dropping arguments: $\pi_\theta(o_t)$ usually means $\pi_\theta(o_t \mid q, o_{\lt t})$ . Papers drop the conditioning once it's been stated.
Log base: in ML papers, $\log$ is the natural log unless the units are bits.
Where the sum's normaliser goes ( $1/\lvert o_i \rvert$ per answer, or one $1/\sum_i \lvert o_i \rvert$ for the batch) is a real design choice, not a typo: GRPO fixes.
"Unbiased" always means unbiased for something. Check whether that's a value or a gradient: Monte Carlo.

Symbols

Formulas you'll meet

Habits of notation

Related