Glossary

Every term used in this course, what it means, and what it looks like when the agent is a language model.

Hover over a term, or tap it, for a real-world example and an LLM example.

The basics

Term	Symbol	Meaning	For a language model
		The learner that makes decisions	The model
		Everything outside the agent	Prompt, tools, sandbox and checker
	$s$	What the agent observes right now	Prompt plus the tokens generated so far
	$a$	One choice the agent makes	The next token. Zoomed out, the whole answer.
	$\pi(a \mid s)$	Probability of each action in each state	The model's softmax over its vocabulary
	$r$	Number the environment returns after an action	A score, usually once at the end
	$\tau$	One run from start to finish	One full response
		An episode generated to train on	One sampled answer and its score
	$G$	Total (discounted) reward of an episode	The final score
	$\gamma$	How much less a reward counts per step of delay	Usually 1
		RL with one decision and no next state	Scoring one answer to one prompt
	$\varepsilon$	Trying actions that don't currently look best	Sampling temperature, several samples per prompt

Values and gradients

Term	Symbol	Meaning	For a language model
	$V(s)$	Expected return from state $s$	Expected final score of a half-written answer
	$Q(s, a)$	Expected return after action $a$ in state $s$	Value of one particular next token
	$V_\phi$	A learned network that estimates $V$	Used by PPO, not by GRPO
	$A$	How much better than expected an action turned out	How much better than average an answer scored
	$b$	The expected reward subtracted to get the advantage	The group's mean score in GRPO
		Data from the policy being updated	Answers from the current weights
		Data from a different policy	Answers from older weights, or another model
	$\pi / \mu$	Corrects for data from another policy	Per-token probability now ÷ when sampled
	$\rho$	PPO's importance weight	Computed per token
	$\varepsilon_\text{high}$	Upper clip larger than lower clip	DAPO's 0.28 vs 0.2

Rewards and preferences

Term	Symbol	Meaning	For a language model
	$r_\phi$	Learned scorer trained on comparisons	Scores chat answers
		SFT → reward model → RL	The original ChatGPT recipe
	$\pi_\text{ref}$	A frozen copy of the starting policy	The model before RL training
		Preference training without a reward model	Chosen vs rejected answer pairs
		Rewards computed by a program	Answer checkers and unit tests
		Scores each step, not just the outcome	Step-by-step solution grading
		Per-prompt checklist scored by a judge	Open-ended tasks with no answer key
		Scoring well without doing the intended task	Padding answers, editing tests

Training dynamics and systems

Term	Symbol	Meaning	For a language model
		Several samples for the same input	Several answers to one prompt in GRPO
	$H$	How spread out the policy's choices are	Falls during RL; when it's gone, learning stalls
		Solved by at least one of $k$ attempts	Measures what the model can do at all
		Updates between generating and training	Versions behind in asynchronous RL
		Sampler and trainer disagree on probabilities	vLLM vs the training code
		Which tokens count in the loss	Tool outputs are masked
		Student imitates teacher	Fine-tuning on a bigger model's answers
		Teacher scores the student's own tokens	Merging specialist models

Related

rl-loop - An agent acts, the world responds with a new state and a reward, and the agent learns to collect more reward. Every later page builds on these five parts.
papers - Every paper and method cited in this course, with one line on what it changed and where it's explained. Use it while reading new papers.