PTLearning

Paper cheat sheet

Every paper and method cited in this course, with one line on what it changed and where it's explained. Use it while reading new papers.

Reading a new RL paper

Most new methods change one of a few things. Ask which:

  1. The advantage: what baseline, what normalisation? (GRPO, fixes)
  2. The ratio and clip: per token or per answer, clipped how? (PPO, importance sampling)
  3. The loss average: over answers, over tokens, or a constant? (fixes)
  4. The reward: verifier, reward model, judge, rubric, teacher? (rewards)
  5. The data: which prompts, how filtered, what curriculum? (data)
  6. The system: how stale, how corrected? (async, mismatch)

Then check the evaluation: which base model, which benchmarks, and is the gain at pass@1 only (pass@k)?

Algorithms and objectives

PaperDateWhat it changedChapter
PPO2017Limits how far the probability ratio can move (clipping)PPO
InstructGPTMar 2022SFT → reward model → PPO (RLHF)Reward models
DPOMay 2023Preference loss without a reward model or RL loopDPO
IPO, KTO, SimPO, ORPO2023–24DPO variantsDPO
RLOOFeb 2024Leave-one-out baselineFixes
DeepSeekMathFeb 2024GRPO: group mean as baseline, no criticGRPO
REINFORCE++Jan 2025Batch-level advantage normalisationFixes
Dr. GRPOMar 2025Removes length and std biasesFixes
DAPOMar 2025Clip-higher, dynamic sampling, token-level loss, overlong shapingFixes
VAPOApr 2025Critic-based PPO made to work for long reasoningFixes
MiniMax-M1Jun 2025CISPO: clip the weight, keep every token's gradientFixes
GSPOJul 2025Sequence-level ratioFixes
Rethinking KL RegularizationOct 2025Which KL estimator, in the reward or the lossKL
SAPONov 2025Smooth gate instead of a hard clipFixes
GDPOJan 2026Normalise each of several rewards separatelyFixes

Rewards

PaperDateWhat it changedChapter
Reward model overoptimisationOct 2022Proxy reward up, true reward down: scaling lawsReward models
Let's Verify Step by StepMay 2023Process reward models from human step labelsRewards
Implicit PRM, PRIME2024–25Step rewards learned from outcome labelsRewards
DeepSeek-R1Jan 2025RL with verifiable rewards at scaleRLVR
Monitoring reasoning modelsMar 2025Chain-of-thought monitors; training against them hides hackingRewards
Rubrics as RewardsJul 2025Checklist rewards for open-ended tasksRewards
CoT MonitorabilityJul 2025Why readable reasoning is worth protectingRewards
Emergent misalignment from reward hackingNov 2025Learning to hack in training spreads to wider misbehaviourRewards

Training dynamics and evaluation

PaperDateWhat it changedChapter
Self-ConsistencyMar 2022Majority vote over samplespass@k
Test-time computeAug 2024Spending compute well when the model answerspass@k
OverthinkingDec 2024Reasoning models waste tokens on easy problemsLength
L1Mar 2025Length budget in the prompt, trained by RLLength
Does RL incentivise reasoning?Apr 2025RL sharpens; base models catch up at large kpass@k
TTRL, INTUITOR2025Rewards without answer keyspass@k
One training exampleApr 2025One prompt is enough for big gains on some modelspass@k
Entropy mechanismMay 2025Performance is bought with entropyEntropy
ProRLMay 2025Long RL can expand what's solvablepass@k
80/20 forking tokensJun 2025Train on the high-entropy 20% of tokensEntropy
Spurious rewards, contaminationJun–Jul 2025Random rewards "work" on Qwen math, likely via leakagepass@k
Pass@k trainingAug 2025Reward diversity directlypass@k
ScaleRLOct 2025RL reward follows an S-curve as compute growsRecipes
PASS@(k,T)Apr 2026RL expands capability for agentspass@k

Systems, agents and distillation

PaperDateWhat it changedChapter
STaRMar 2022Fine-tune on your own correct answersDistillation
MiniLLM, GKDJun 2023On-policy distillationDistillation
Kimi k1.5Jan 2025Partial rollouts, long-to-shortAsync
SWE-RLFeb 2025RL on software engineeringAgents
Search-R1Mar 2025Tool-use RL, masking retrieved tokensAgents
GiGPOMay 2025Credit assignment by grouping repeated statesAgents
AReaLMay 2025Fully asynchronous RLAsync
RL's RazorSep 2025RL forgets less than fine-tuningDistillation
PipelineRLSep 2025In-flight weight updatesAsync
R3 routing replayOct 2025Same MoE experts in sampler and trainerMismatch
IcePopOct 2025Mask tokens with large sampler–trainer gapsMismatch
FP16 vs BF16Oct 2025Precision fixes most of the mismatchMismatch
Self-Distilled ReasonerJan 2026Model as its own teacherDistillation
OPD then RLVRSep 2026Distil first, then RLDistillation

Technical reports from labs are listed on frontier recipes.

Related

  • glossary - Every term used in this course, what it means, and what it looks like when the agent is a language model.
  • frontier-recipes - Modern post-training (everything after the first big round of learning from text) chains supervised fine-tuning, several kinds of RL, and distillation. This page maps the published recipes onto the ideas in this course.