Training an LLM with RL

Fine-tuning and RL use the same model, loss and optimiser. Two things change, where the training text comes from and how much each token counts.

Key idea: Every earlier page, in one training run
Real world: A tennis coach can show you the perfect serve (imitation), or let you serve a hundred times and tell you which ones went in (RL). The second way can take you past the coach's own example.
In an LLM: GSM8K maths problems, Qwen 2.5 1.5B Instruct, two rule-based rewards and GRPO, small enough for one GPU with LoRA.

Fine-tuning vs RL, side by side

	Supervised fine-tuning (SFT)	RL (policy gradient / GRPO)
Training text	From a dataset of ideal answers	Sampled from the model itself
Weight per token	Always 1	The advantage, positive or negative
Dataset needs	Prompts and ideal answers	Prompts and a way to score answers
Model learns to	Imitate	Do more of what scored above average
Cost per step	One forward and backward pass	Generate G answers first, then train

RL fine-tuning is SFT on the model's own samples, weighted by how much better than average each one scored.

Three ways RL is used on LLMs

Method	Where the reward comes from	Typical algorithm	Example
RLHF	A reward model trained on human preference comparisons	PPO with a KL penalty	InstructGPT, early ChatGPT: helpful, polite answers
RL with verifiable rewards	A program: answer checker, unit tests, format rules	GRPO and its variants	DeepSeek-R1: long step-by-step reasoning in maths and code
DPO	Preference pairs, used directly	No sampling, so not strictly RL (DPO)	Many open chat models
On-policy distillation	A stronger teacher model scores every token	Policy gradient with per-token advantages (distillation)	Merging specialist models in DeepSeek-V4, Kimi K3

The script below is the second kind: the reward is a Python function, so it is cheap, exact and hard to hack.

What to watch when you run it

Read the answers, not just the reward curve. Reward rising while answer length explodes is reward hacking happening for real.
Groups that all score the same. They teach nothing (GRPO). Keep prompts the model gets right sometimes (data and length).
Watch entropy. If the model’s choices narrow too early (entropy collapse), the reward will plateau soon after (entropy).
Start from a model that can already do the task occasionally. RL sharpens behaviour the model sometimes shows; it rarely invents it from nothing. A round of SFT first is common.
Prefer checkable rewards. Exact answers, unit tests and format checks are harder to hack than a learned reward model.
Watch drift. A sudden KL spike (the model jumping away from its starting point) usually means an exploit or a learning rate that's too high.

going deeper

The two losses

\text{SFT:}\quad L = -\sum_t \log \pi(y_t \mid x, y_{<t}) \qquad y \text{ from the dataset}

\text{PG:}\quad L = -\sum_t A \cdot \log \pi(y_t \mid x, y_{<t}) \qquad y \text{ sampled from } \pi

The script

This uses Hugging Face TRL. The peft_config line trains a small LoRA adapter instead of every weight, which cuts memory a lot. Remove it for full fine-tuning.

from datasets import load_dataset
from peft import LoraConfig
from trl import GRPOConfig, GRPOTrainer
import re
 
ds = load_dataset("openai/gsm8k", "main", split="train")
ds = ds.map(lambda x: {
    "prompt": x["question"] + "\nPut the final number in <answer></answer>.",
    "target": x["answer"].split("####")[-1].strip(),
})
 
# Reward functions get the completions plus any extra dataset columns ("target") as kwargs
def correctness(completions, target, **kwargs):
    out = []
    for c, t in zip(completions, target):
        m = re.search(r"<answer>(.*?)</answer>", c)
        out.append(1.0 if m and m.group(1).strip() == t else 0.0)
    return out
 
def format_ok(completions, **kwargs):
    return [0.2 if re.search(r"<answer>.*?</answer>", c) else 0.0 for c in completions]
 
args = GRPOConfig(
    output_dir="qwen-grpo",
    learning_rate=1e-5,            # sweep this; LoRA often tolerates higher
    num_generations=8,             # G, the group size (GRPO page)
    max_completion_length=256,
    epsilon=0.2,                   # lower clip (PPO page)
    epsilon_high=0.28,             # clip-higher (Fixing GRPO page)
    loss_type="dapo",              # token-level loss averaging (Fixing GRPO page)
    scale_rewards="none",          # don't divide by the group's std (Dr. GRPO)
    mask_truncated_completions=True,  # cut-off answers don't count (Fixing GRPO page)
    beta=0.0,                      # no KL penalty, common for verifiable rewards (KL page)
    per_device_train_batch_size=8,
    logging_steps=10,
)
 
trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    reward_funcs=[correctness, format_ok],
    args=args,
    train_dataset=ds,
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear"),
)
trainer.train()

Written against the TRL GRPOTrainer API. Option names and defaults change between TRL releases (older versions take scale_rewards=False, for example), so check the docs for the version you install.

Logged metrics to check

reward and each reward function's mean: is it rising?
completions/mean_length: is it exploding?
kl: is it spiking? Only logged when beta is above 0.
frac_reward_zero_std: share of groups with no signal

Going further

Sutton and Barto, Reinforcement Learning: An Introduction (free textbook)
OpenAI Spinning Up: key concepts and policy gradients
Proximal Policy Optimization (Schulman et al., 2017)
Training language models to follow instructions with human feedback (InstructGPT, 2022)
DeepSeekMath, where GRPO was introduced, and DeepSeek-R1
TRL GRPOTrainer documentation
The paper cheat sheet for everything cited in this course