Policy gradient

Skip values. Make actions that scored above average more likely, and the rest less likely.

Key idea: Policy gradient with a baseline, which gives the advantage
Real world: Learning to throw darts. You don't calculate the value of each stance. You throw, see where it lands, and repeat whatever you did on the throws that beat your usual score.
In an LLM: The core loss of every RL trainer: loss = -(advantage * logprob).mean(), applied to text the model generated itself.

The update in three steps

Sample an action from the policy.
Score it and subtract the baseline, the score you expected. The result is the advantage.
Update: raise the action's in proportion to the advantage. A negative advantage lowers it.

Values tell you which states are good. Policy gradient changes the policy directly, which works even when actions come on a smooth scale (how hard a robot turns a joint) or there are too many to list (all possible sentences).

example · one update by hand, in an LLM

A language model completes "The capital of Australia is". To keep it small, say it only considers three tokens, Canberra, Sydney and Melbourne, all with logit 0 (the raw score before the softmax turns scores into probabilities), so each has probability ⅓.

Sample: the model writes Canberra.
Score: a checker gives reward 1. The baseline (average reward so far) is 0.4, so the advantage is $A = 1 - 0.4 = 0.6$ .
Update: the direction that raises $\log \pi(\text{Canberra})$ fastest (its gradient with respect to the three logits) is $(1 - \tfrac13,\ -\tfrac13,\ -\tfrac13)$ . With learning rate 1, the logits move by $0.6 \times$ that:

(0,\ 0,\ 0) \;\to\; (0.4,\ -0.2,\ -0.2)

After the softmax, Canberra goes from 33% to 48%, and the other two drop to 26% each.

If the model had written Sydney and scored 0, the advantage would be −0.4 and Sydney would become less likely. A darts player does the same thing without the maths: repeat the throw that beat the usual score, avoid the one that didn't.

Try it

Four candidate answers to "Why is the sky blue?", scored 0.05 to 0.90 by a grader. Watch the policy shift toward the best one.

Prompt: “Why is the sky blue? One line.”

“Rayleigh scattering: air scatters short blue wavelengths far more than red.”reward ≈ 0.9015.1%

“Because it reflects the ocean.”reward ≈ 0.2037.1%

“Sunlight scatters off air; blue scatters most, so it comes from every direction.”reward ≈ 0.7520.4%

“idk lol”reward ≈ 0.0527.5%

lr0.50Subtract baseline

step0

baseline0.00

expected reward0.376

Why the baseline matters

Without a baseline, the update size depends on the raw reward. If every reward is positive, every action gets pushed up, and only the amount differs. The policy still drifts the right way, but slowly and noisily. Subtracting the expected reward means only surprises cause updates: better than expected goes up, worse goes down.

Where it's used

Robotics and control: walking, grasping and other tasks with continuous actions, usually through PPO.
AlphaGo: its policy network was improved by policy gradient through self-play, after first learning from human games.
Language models: the gradient of a log-probability is exactly what ordinary fine-tuning computes. Policy gradient applies it to text the model wrote itself, weighted by the advantage.

going deeper

REINFORCE

\theta \leftarrow \theta + \text{lr} \cdot A \cdot \nabla_\theta \log \pi_\theta(a \mid s) \qquad A = r - b

For a softmax policy with logits $\theta$ , the gradient of $\log \pi(a)$ with respect to $\theta_j$ is $\mathbb{1}[j = a] - \pi_j$ . That is the whole update this widget runs. The baseline $b$ here is a running mean of rewards.