- Key idea
- Exploration vs exploitation
- Real world
- You have a favourite restaurant. Going back is a safe good meal (exploit). Trying the new place down the road might be better or worse, but it's the only way to find out (explore).
- In an LLM
- Sampling temperature and the number of answers sampled per prompt. At temperature 0 the model always gives the same answer, so it never discovers a better one.
The dilemma
Every choice either exploits what you know or explores to learn more. Only exploiting locks you into the first thing that worked. Only exploring means you never use what you learned. Every RL agent needs some balance.
The same trade-off shows up outside RL:
- A/B testing and ads: keep showing the best-performing ad, or give the new one some traffic?
- Clinical trials: give patients the treatment that looks best so far, or keep testing the alternatives?
- Games: AlphaGo Zero adds noise to its search during self-play so it keeps trying unusual moves.
- Language models: a model that always solves maths problems the same way never finds out whether another approach, such as checking its work, scores higher.
The setup: four slot machines
Each arm pays 1 with a hidden probability, otherwise 0. One arm is best. You have to find it by pulling. This is a bandit: RL with one decision and no next state, the simplest place to study exploration.
Pull the arms yourself
Let a strategy pull 400 times
ε-greedy: pick the arm with the best average so far. A fraction ε of the time, pick an arm at random instead. Move the slider and the 300 simulated runs re-run for that ε.
After some pulls, your average payouts are , and ε = 0.1.
- With probability 0.9 you exploit: pull arm 2, the best so far.
- With probability 0.1 you pick one of the four at random, which can also land on arm 2.
So arm 2 gets and each other arm .
You pull arm 2 and it pays 1. It had paid an average of 0.6 over 5 pulls, so the new average over 6 pulls is
That "old estimate plus a step toward the new sample" pattern comes back in value learning and every later algorithm.
For a maths problem, a model has three ways to start, with logits 2, 1 and 0: (A) answer straight away, (B) work step by step, (C) work step by step and then check. Suppose C scores best but the model rarely picks it. Temperature divides the logits before the softmax (which turns scores into probabilities):
| Temperature | A | B | C | Chance 8 samples never try C |
|---|---|---|---|---|
| 0 | 100% | 0% | 0% | 100% |
| 0.5 | 87% | 12% | 1.6% | 88% |
| 1 | 67% | 24% | 9% | 47% |
At temperature 0 the best approach is never sampled, so RL can never reward it. At temperature 1 with a group of 8, more than half of the prompts include at least one try of C.
In language models
plays the role of ε. A model sampled at temperature 0 always gives its current favourite answer, so there is nothing to compare it with. RL methods for language models sample several answers per prompt at a temperature around 1, partly for this reason (GRPO).
Smarter exploration
ε-greedy explores blindly. Two common alternatives explore where it is most useful:
Upper confidence bound (UCB) picks the arm with the highest optimistic estimate. Arms that have been tried less often get a larger bonus:
Thompson sampling keeps a belief distribution over each arm's payout, draws one sample from each, and pulls the arm with the highest draw. For 0/1 rewards the belief is a distribution.
The incremental mean from the example, written generally: