The RL loop

An agent acts, the world responds with a new state and a reward, and the agent learns to collect more reward. Every later page builds on these five parts.

Key idea: Agent, state, action, reward, policy
Real world: A robot vacuum sees its map and battery level (state), picks a direction (action), and gets points for dirt collected and a penalty for bumping into furniture (reward).
In an LLM: The model is the agent, the prompt plus the text so far is the state, each token is an action, and a checker or reward model scores the finished answer.

The five parts

Policy: the agent's decision rule. Given a state, it says which action to take, or how likely each one is.
State: what the agent sees right now.
Action: one choice the agent makes.
Reward: a number the environment sends back after an action. It can be positive, negative or zero, and it is often zero until the episode ends.
Return: the total reward over one , from start to finish. RL tries to make the average return as high as possible.

Flip the switch. The loop is the same; only the names change.

What is an episode?

An episode is one complete attempt at the task, from the start to a natural end. When it ends, the world resets and the next episode starts fresh. RL learns by comparing many episodes: which ones scored well, and what the agent did differently in them.

example · episodes in the real world

Chess: one game, from the first move to checkmate or a draw. A program that plays 10,000 games has 10,000 episodes.
Robot vacuum: one cleaning run, from leaving the dock to returning to it.
Video game: one life, or one level, until the character dies or reaches the exit.

example · episodes in an LLM

One episode is one complete answer to one prompt: the model writes tokens until it produces the end token, then a checker scores the answer.

Prompt "What is 7 × 8?" → the model writes 7 × 8 = 56. and stops. That is one episode, with 7 actions and 1 reward.
In GRPO the model writes 8 answers to the same prompt. That is 8 separate episodes from the same starting state.
A training run over 1,000 prompts with 8 answers each is 8,000 episodes.

Some tasks have no natural end, like a thermostat that runs all year. These are called continuing tasks, and the discount γ keeps their return finite. Every LLM task on this site is episodic.

The same loop everywhere

Agent	State	Action	Reward
Chess program	Board position	A legal move	+1 win, −1 loss, 0 draw, only at the end
Thermostat	Room and outside temperature, time of day	Heat up, cool down, off	Minus energy cost, minus discomfort, every few minutes
Robot vacuum	Map, position, battery	Move in a direction	+ dirt collected, − bumps, + docking before the battery dies
Video recommender	What the user just watched	Which video to show next	Watch time
Language model	Prompt plus tokens so far	The next token	A score for the finished answer

The rows look different, but an RL algorithm only needs to know the loop: observe, act, receive a reward, repeat.

Rewards now vs rewards later

Much of the difficulty is that rewards are delayed. Often they arrive only at the end. The chess program makes 40 moves and learns whether it won only when the game is over. A language model writes a whole answer before a checker scores it. Rewards and returns covers when rewards arrive, how they add up, and the discount γ that makes later rewards count less.

Watch one episode

Here is the loop for a language model answering a maths question. Each token is one action, and the score only arrives when the answer is finished.

Q: What is 7 × 8? A:

Policy at step 1: top next tokens

·758%

·The24%

·Seven7%

plus ~150,000 other tokens with tiny probability

Trajectory

t	action	reward
No actions yet

Press Generate next token.

going deeper

Notation

The policy. Click or hover any coloured part.

\hl{0}{\pi}(\hl{1}{a} \hl{2}{\mid} \hl{3}{s})

Read it as: the probability that the policy picks action a when it is in state s.

The return.

\hl{0}{G} = \hl{1}{r_1} + \hl{2}{\gamma}^{\hl{3}{1}}\, \hl{1}{r_2} + \hl{2}{\gamma}^{\hl{3}{2}}\, \hl{1}{r_3} + \hl{4}{\cdots} = \hl{5}{\sum_{t \ge 1}} \hl{2}{\gamma}^{\hl{3}{t-1}}\, \hl{1}{r_t}

example:

\begin{aligned} \hl{0}{G} &= \hl{1}{0} + \hl{2}{0.9}^{\hl{3}{1}} \cdot \hl{1}{0} + \hl{2}{0.9}^{\hl{3}{2}} \cdot (\hl{1}{-1}) + \hl{2}{0.9}^{\hl{3}{3}} \cdot \hl{1}{0} + \hl{2}{0.9}^{\hl{3}{4}} \cdot \hl{1}{10} \\ &= 0 + 0 - 0.81 + 0 + 6.56 = 5.75 \end{aligned}

\begin{aligned} \hl{0}{G} &= \hl{1}{0} + \hl{2}{0.9}^{\hl{3}{1}} \cdot \hl{1}{0} + \hl{2}{0.9}^{\hl{3}{2}} \cdot (\hl{1}{-1}) \\ &\quad + \hl{2}{0.9}^{\hl{3}{3}} \cdot \hl{1}{0} + \hl{2}{0.9}^{\hl{3}{4}} \cdot \hl{1}{10} \\ &= 0 + 0 - 0.81 + 0 + 6.56 = 5.75 \end{aligned}

γ = 0.9. Two empty moves (reward 0), a bump (−1), another empty move (0), then docking (+10). The empty moves add 0. The bump costs 0.81 and the dock is worth 6.56, because each comes a few steps after the start.

The goal.

\max_{\hl{0}{\pi}}\; \hl{1}{\mathbb{E}}_{\hl{2}{\tau} \sim \hl{0}{\pi}}\big[\, \hl{3}{G} \,\big]

Read it as: find the policy whose episodes score the highest return on average. With a single reward at the end and γ = 1, that is the policy that gets the answer right most often.