Rewards and returns

When an agent gets its rewards, how they add up to one score, and why late rewards can count for less. Language models get a single reward at the end, and don’t shrink it.

Key idea: Sparse vs dense rewards, the return, the discount γ, return to go, reward shaping
Real world: A footballer gets a bonus only when the team wins the match (sparse, at the end), while a courier gets paid for each parcel delivered (dense, every step).
In an LLM: The model writes a whole answer, then gets one score: +1 if the maths is right. Every token before the last gets 0, and γ = 1 so the score counts in full for all of them.

When does the reward arrive?

The RL loop sends a reward after every action, but most of those rewards can be zero. Tasks differ in when the real signal comes.

Dense rewards arrive at almost every step. A courier is paid per parcel. A thermostat pays an energy cost every few minutes.
arrive rarely, often only once, at the end of the episode. A chess program gets +1, −1 or 0 when the game ends and nothing for each move.
Mixed: many tasks have both. A robot vacuum's episode is one cleaning run, from leaving the dock to returning to it. During the run it gets small rewards as things happen: + for dirt picked up, −1 the moment it bumps into a chair. At the end it gets a large +10 for docking before the battery dies.

Task	Rewards during the episode	Reward at the end
Courier	+ per parcel	none
Robot vacuum	+ dirt, − bumps	+10 for docking
Chess	none	+1, −1 or 0
LLM answer	none (0 for every token)	+1 if correct

Dense rewards are easier to learn from: the agent finds out quickly whether a move helped. Sparse rewards are honest but slow: after a 40-move win, which move deserves the credit?

example · rewards at the end in an LLM

In LLM training the reward almost always arrives at the end. A half-written proof or half a line of code can't be marked right or wrong, so the model writes its whole answer first and then gets one score.

Maths: the final answer is compared with the key: +1 or 0, once.
Code: the finished program runs against unit tests: the fraction that pass, once.
Chat (RLHF): a reward model reads the whole reply and gives one score.

Every token before the last gets a reward of 0. The exception is a process reward model, which scores each reasoning step; most current recipes still use one outcome reward per answer.

The return: adding rewards up

The G is the total reward of one episode. With no discount it is a plain sum: a vacuum that bumps once (−1) and then docks (+10) has a return of 9. RL tries to make the average return as high as possible.

A plain sum has one blind spot: it ignores when a reward comes. Docking after 5 steps or after 500 scores the same 9.

The discount γ

The γ (gamma) is a number between 0 and 1. A reward that comes $k$ steps later is multiplied by $\gamma^k$ before it is added in:

G = r_1 + \gamma\, r_2 + \gamma^2 r_3 + \cdots

Three ways to read it:

Sooner is better. £100 today is worth more than £100 in a year: you could invest today's £100 and have more in a year. At 10% interest, £100 a year from now is worth about £91 today, so a year's γ is about 0.91.
The future is uncertain. If there is a 1% chance each step that the episode simply stops (the battery dies, the user leaves), a reward $k$ steps away only arrives with probability $0.99^k$ . That is γ = 0.99.
It keeps sums finite. A thermostat runs forever. If γ were not below 1, its return would add up forever.

example · γ in the real world

A robot vacuum with γ = 0.9. Unlike an LLM, it gets a reward in the middle of the episode as well as at the end: two empty moves, a bump (−1, given at once), another empty move, then it docks (+10, the end of the episode).

Step	1	2	3	4	5
Reward	0	0	−1	0	+10
Weight $\gamma^{t-1}$	1	0.9	0.81	0.729	0.656
Weighted	0	0	−0.81	0	+6.56

Return $G = -0.81 + 6.56 = 5.75$ . Docking one step earlier would be worth $10 \times 0.729 = 7.29$ , so γ pushes the vacuum to finish quickly. A plain sum (γ = 1) would give 9 either way.

example · γ in an LLM

The model answers "What is 7 × 8?" with 7 × 8 = 56. and an end token: 7 actions, reward +1 on the last.

With γ = 1 the first token's return is 1. Every token gets full credit for the right answer.
With γ = 0.9 the first token sees $0.9^6 = 0.53$ . A short answer still works.
With γ = 0.99 and a 1,000-token reasoning answer, the first token sees $0.99^{999} \approx 0.00004$ . The early reasoning, often the part that mattered most, would get almost no signal.

So LLM training sets γ = 1. Length is controlled in other ways, such as a length penalty or a token limit.

discount γ

reward rweight from step 1

0.9

0.81

0.73

0.66

return to go G

return from step 1 (“move”)

G = 0 + 0 × 0.9 − 1 × 0.81 + 0 × 0.73 + 10 × 0.66 = +5.75

return of the episode+5.75

plain sum (γ = 1)+9

weight on the last step0.66

The +10 for docking sits 4 steps after the start, so it counts 0.66 × 10. The bump at step 3 is nearer, so it costs almost its full −1. Click step 4: from there the dock is one step away and worth much more.

One episode, one column per step. Top: the reward at each step. Middle: the weight γ to the power of the delay. Bottom: the return still to come from each step. Click a column to see its sum.

What if the only reward is at the end?

The formula adds one term per step, but a step with reward 0 adds nothing, whatever its weight. If the only reward comes at the last step $T$ , every term but one vanishes:

\hl{0}{G} = \hl{1}{0} + \hl{2}{\gamma} \cdot \hl{1}{0} + \hl{2}{\gamma}^2 \cdot \hl{1}{0} + \cdots + \hl{2}{\gamma}^{\hl{3}{T-1}}\, \hl{4}{r_T} = \hl{2}{\gamma}^{\hl{3}{T-1}}\, \hl{4}{r_T}

So with a reward only at the end, discounting means one thing: the same result counts for less if it took more steps.

What that does depends on γ:

Task	Where rewards come	γ used	What the discount does
Robot vacuum	During the run and at the end	Usually below 1, such as 0.9 or 0.99	Weighs each reward by when it came: an early bump counts more than a late one, and docking sooner is worth more
Chess	Only at the end	Below 1 in many textbook setups; AlphaZero used 1	A win in 20 moves is worth $0.99^{19} = 0.83$ , a win in 60 is worth $0.99^{59} = 0.55$ , so the agent prefers quick wins. With γ = 1 a win is a win
LLM	Only at the end	1	Nothing: $\gamma^{T-1} = 1$ , so $G = r_T$ . A right answer scores 1 whether it took 10 tokens or 1,000

So for an LLM the discount is switched off. It appears in the formula because the formula is written for every RL task, and robots and games often need it.

How far can γ see?

A rough rule: γ looks about $1 / (1 - \gamma)$ steps ahead. γ = 0.9 sees about 10 steps, γ = 0.99 about 100, γ = 0.999 about 1,000. Beyond that, rewards fade towards zero.

γ0.99answer length500 tok

final reward, seen from token 10.66%

horizon ≈ 1 ÷ (1 − γ)100 steps

With γ = 0.99 the agent effectively looks about 100 steps ahead. A 500-token answer is far beyond that: the first token feels only 0.66% of the final reward, so it gets almost no learning signal.

The curve is how much a reward k steps away counts, γ to the power k. Pick γ, then stretch the answer: the dot is the end of the answer, where an LLM’s reward arrives.

Games and robots often use γ = 0.99, which suits episodes of a few hundred steps. LLM answers run to thousands of tokens, past the horizon of any γ that is useful elsewhere.

Return to go: credit for what comes after

An action can't change rewards that came before it. So instead of the whole episode's return, each step $t$ is judged by the : the discounted rewards from step $t$ onwards.

G_t = r_t + \gamma\, r_{t+1} + \gamma^2 r_{t+2} + \cdots

example · return to go

Real world: the vacuum bumped at step 3. From step 4 on, the return to go is $0 + 0.9 \times 10 = 9$ : the bump is in the past and doesn't count against step 4's move.
In an LLM: with one reward at the end and γ = 1, every token's return to go equals the final score. A right answer gives all 7 tokens +1; a wrong one gives all of them 0. Telling the useful tokens from the filler is left to the baseline and the value function, or, in GRPO, to comparing many answers to the same prompt.

Reward shaping

When rewards are sparse, it is tempting to add small rewards along the way to guide the agent. This is . It helps learning, but the agent optimises exactly what you reward, not what you meant.

example · shaping that went wrong

Real world: in the boat-racing game CoastRunners, OpenAI's agent got points for hitting targets along the course. It found a lagoon where three targets respawned, and circled there forever, crashing and on fire, never finishing the race. It still scored about 20% more than human players.
In an LLM: a reward of +0.1 per reasoning step, meant to encourage careful thought, teaches the model to pad its answers with steps. A reward for "sounds confident" teaches it to sound confident when wrong. This is reward hacking.

Some shaping does work in LLMs. DeepSeek-R1-Zero added a small format reward for putting its reasoning between think tags, alongside the main accuracy reward. And PPO-based RLHF adds a small KL penalty at every token: switch the widget above to "LLM + KL" to see a reward that is mostly sparse with a little dense part.

going deeper

Notation

G_t = \sum_{k \ge 0} \gamma^k\, r_{t+k} \qquad G_t = r_t + \gamma\, G_{t+1}

The second form, a return defined in terms of the next one, is the basis of the Bellman equations on the value page. For a reward that is constant at 1 forever, $G = 1 / (1 - \gamma)$ , which is where the horizon rule comes from.

Potential-based shaping

Ng, Harada and Russell (1999) showed that a shaping reward of the form $F(s, s') = \gamma\,\Phi(s') - \Phi(s)$ , for any function $\Phi$ of the state, never changes which policy is optimal: the extra terms telescope away over an episode. Shaping rewards not of this form can change the optimal policy, as CoastRunners did.

Sources

Policy invariance under reward transformations (Ng, Harada, Russell, ICML 1999)
Faulty reward functions in the wild (OpenAI, 2016, CoastRunners)
DeepSeek-R1 (accuracy and format rewards, 2025)
Training language models to follow instructions with human feedback (InstructGPT: RLHF reward with per-token KL, 2022)