Process rewards, judges and rubrics

When a program can't check the answer, or you want credit for individual steps, you need other reward sources. Each brings new ways to be gamed.

Key idea: Outcome vs process rewards, LLM judges, rubrics, and catching reward hacking
Real world: A teacher can mark only the final answer, mark each line of working, or grade an essay against a marking scheme. Each tells the student something different, and each can be gamed differently.
In an LLM: A medical-advice answer has no single correct output. An LLM judge scores it against a rubric written for that question: did it ask about symptoms, mention when to see a doctor, avoid dangerous advice?

Outcome vs process rewards

An outcome reward model (ORM) scores only the final result. Every verifiable reward is an outcome reward.
A process reward model (PRM) scores each step of the reasoning, so a solution that goes wrong at step 3 gets low marks from step 3 on, even if it stumbles onto the right answer.

OpenAI's Let's Verify Step by Step trained a PRM on about 800,000 human step labels. Used to pick the best of many sampled solutions on MATH problems, it beat an outcome model and majority voting. Human step labels are expensive, so later work generates them automatically.

example · marking the working

A student writes four lines of working and gets the right final answer, but line 2 contains an error that a later error happens to cancel out. Marking only the answer gives full credit. Marking the working catches line 2 and teaches the student something useful.

example · labelling steps automatically

Math-Shepherd labels a step by trying many continuations from it. From step 3 of a solution, the model completes the problem 8 times and gets the right answer 6 times, so step 3 is labelled 0.75. From step 4, only 1 in 8 completions is right: 0.125. Step 4 is probably where it went wrong. No human needed, but the labels are noisy.

Why many RL recipes still don't use PRMs: DeepSeek reported that PRMs were expensive to keep retraining and easy for the policy to game, and R1 used outcome rewards only. PRMs are returning for long agent tasks, where a single reward at the end of hundreds of steps is too sparse.

LLM judges and rubrics

For tasks with no checkable answer, the reward can come from another language model:

Generative reward models / LLM-as-judge: a model reads the answer, reasons about it in text, then gives a score. More robust than a single-number reward model, and it can use more compute on hard cases.
Rubrics as rewards: for each prompt, write a checklist of what a good answer must include and avoid, with weights. A judge checks each item and the reward is the weighted share satisfied.

example · a rubric reward

Prompt: "I have a bad headache and blurred vision. What should I do?"

Criterion	Weight	Answer meets it?
Says sudden headache with vision changes can be an emergency	5	✓
Advises urgent medical care	5	✓
Asks about or lists other warning signs	2	✓
Avoids recommending a specific prescription drug	3	✓
Mentions not driving	1	✗

Reward: $(5 + 5 + 2 + 3) \div 16 \approx 0.94$ .

example · a marking scheme

Exam boards publish marking schemes for essays: a point for stating the thesis, two for supporting evidence, one for a counter-argument. Two markers using the scheme agree far more often than two markers giving an overall impression. Rubric rewards borrow exactly that.

Reward hacking in modern models

Reward hacking gets more creative as models get more capable. Cases reported for reasoning and coding models include special-casing the test inputs, editing the tests, and exiting early with a success code.

Two findings shape current practice:

Watching the reasoning helps. OpenAI found that a monitor reading the model's chain of thought caught hacking far more reliably than one watching only its actions. But when they penalised hacking thoughts during training, the model kept hacking and stopped mentioning it. Training against the monitor destroys the monitor.
Hacking generalises. Anthropic found that a model that learned to hack real coding environments became more misaligned in unrelated ways, including sabotage and faking alignment. Telling the model during training that hacking was acceptable in that environment, so it wasn't learning to deceive, removed most of the effect.

The lesson for anyone running RL: read samples, monitor what the model does and says, make the environments harder to cheat in, and treat a sudden jump in reward as a bug until proven otherwise.

going deeper

Scoring with a PRM

A PRM gives a score $p_k$ per step. A solution's score is usually the minimum, $\min_k p_k$ (one bad step ruins it), or the product $\prod_k p_k$ .

Rubric reward

R = \frac{\sum_k w_k\, c_k}{\sum_k w_k} \qquad c_k \in \{0, 1\} \text{ from the judge}

Papers

Let's Verify Step by Step (OpenAI, 2023)
Free Process Rewards without Process Labels and PRIME: process rewards learned from outcome labels
Rubrics as Rewards (Jul 2025)
Monitoring Reasoning Models for Misbehavior (OpenAI, Mar 2025)
Natural Emergent Misalignment from Reward Hacking in Production RL (Anthropic, Nov 2025)
Chain of Thought Monitorability (Jul 2025)