- Key idea
- Multi-turn episodes, loss masking, environments, and credit across turns
- Real world
- A mechanic fixing a car runs a diagnostic, reads the result, replaces a part, and tests again. They're judged on whether the car runs at the end, not on any single step.
- In an LLM
- A coding agent in a sandbox runs the tests, reads the failure, edits a file, and runs the tests again. The reward is 1 if the hidden tests pass when it's done.
What changes with agents
In single-turn RL, the episode is one answer. In agentic RL, an episode alternates between the model and an environment: the model writes a tool call, the environment runs it and returns output, the model reads that and continues. Episodes can run for dozens or hundreds of turns.
Try it
Train only on what the model wrote
The environment's output is part of the state, not an action. The model chose to run pytest; it didn't choose what pytest printed. So tool outputs are kept in the context but masked from the loss (ignored when working out what to learn). Training on them would teach the model to predict tool output, and to imagine results instead of calling tools.
When a mechanic's repair works, you credit their decisions: which test to run, which part to replace. You don't credit them for what the diagnostic machine printed. Training on tool output is like praising the mechanic for the machine's readout.
Credit across turns
A single reward at the end of a 50-turn episode says little about which turn mattered. Current approaches:
- Outcome only: give every model token the episode's advantage, exactly like GRPO over long answers. Still the default.
- Grouping by state (GiGPO): when different rollouts reach the same intermediate state, compare what they did next there.
- Turn-level rewards: process rewards or judges that score individual turns.
- A critic again: some 2026 recipes bring back a value model for long agent episodes.
Environments
An environment packages a task, a sandbox to act in, and a verifier. Examples:
| Task | Sandbox | Reward |
|---|---|---|
| Question answering with search | Search API | Exact match of the final answer |
| Fixing a GitHub issue | Container with the repository | Hidden tests pass |
| Terminal tasks | Shell in a container | Final state checked by a script |
| Data analysis | Python interpreter | Answer matches a reference |
Each rollout runs in its own isolated container, and these are created and thrown away in huge numbers. Building enough realistic, hard-to-hack environments is now widely seen as the main bottleneck in agentic RL, and several labs and projects run shared environment hubs.
A coding environment rewards "all tests pass". The agent discovers it can edit the test file. The fix belongs in the environment, not the model: make the tests read-only, run hidden tests the agent never sees, and check the diff. See reward hacking.
Masked loss
With a mask for tokens the model generated and for prompt and environment tokens:
The importance ratios and clipping from earlier pages apply to the masked tokens unchanged.