Outcomes, events and probability

The ground rules. A probability is a number between 0 and 1 attached to an event, and a handful of rules say how those numbers combine.

Key idea: Sample space, events, the three rules of probability, “or”, “not” and “at least one”
Real world: Roll two dice. There are 36 equally likely outcomes, and “the sum is 7” is an event made of 6 of them, so its probability is 6/36.
In an LLM: Generate one answer to a maths question. Every possible string of tokens is an outcome; “the final answer is 42” is an event made of millions of different strings, and its probability is the sum over all of them.

Outcomes and events

Probability always starts by listing what can happen.

The sample space is the set of every possible outcome. For one die it is 1 to 6. For two dice it is the 36 pairs (1,1), (1,2), …, (6,6).
An event is a set of outcomes you care about: “the sum is 7”, “a double”, “at least one 6”.
The probability of an event is the total probability of the outcomes inside it.

When every outcome is equally likely, this is just counting: the probability of an event is the number of outcomes in it divided by the number of outcomes in total.

example · two dice

“The sum is 7” contains (1,6), (2,5), (3,4), (4,3), (5,2), (6,1): six outcomes out of 36, so its probability is $6/36 = 1/6$ . “A double” also contains six outcomes, so it is also $1/6$ .

example · one sampled answer

Ask a model “What is 6 × 7?” and sample one reply. The sample space is every possible sequence of tokens, an astronomically large set. The event “the reply is correct” contains “42”, “The answer is 42.”, “6 × 7 = 42”, and countless other phrasings. Its probability is the sum of the probabilities of all of them, which is why two prompts that look similar can have very different success rates: the event is a huge, oddly-shaped set.

Outcomes are not always equally likely, and for a language model they never are. The idea stays the same: add up the probability of every outcome in the event.

Try it

Pick two events and watch which outcomes they contain. Then switch to one model reply: the same rules, but each outcome now brings its own probability instead of an equal share.

first die →second die →

1,1

2,1

3,1

4,1

5,1

6,1

1,2

2,2

3,2

4,2

5,2

6,2

1,3

2,3

3,3

4,3

5,3

6,3

1,4

2,4

3,4

4,4

5,4

6,4

1,5

2,5

3,5

4,5

5,5

6,5

1,6

2,6

3,6

4,6

5,6

6,6

A onlyB onlyA and B

P(A)11/36

P(B)6/36

P(A and B)1/36

P(A or B)16/36

P(not A)25/36

P(A or B) = 11 + 6 − 1 = 16 cells out of 36. The overlap (purple) is subtracted once because adding A and B counts it twice.

Each cell is one outcome: first die across, second die down. All 36 are equally likely, so a probability is just a count divided by 36.

The rules

Everything in probability follows from three rules, called the axioms:

Every probability is between 0 and 1.
The probability that something in the sample space happens is 1.
If two events can’t happen together, the probability that one or the other happens is the sum of their probabilities.

Three useful consequences follow straight away.

Not. The probability an event does not happen is 1 minus the probability it does: $P(\text{not } A) = 1 - P(A)$ .

Or. When two events can overlap, adding their probabilities counts the overlap twice, so you subtract it once:

P(A \text{ or } B) = P(A) + P(B) - P(A \text{ and } B)

At least one. “At least one success in several tries” is easiest to compute through its opposite, “no successes at all”:

P(\text{at least one}) = 1 - P(\text{none})

example · at least one six

Roll four dice. The chance that none shows a 6 is $(5/6)^4 \approx 0.48$ , so the chance of at least one 6 is about $0.52$ . Counting the ways to get one, two, three or four sixes directly works too, but takes much longer.

example · at least one right answer

A model solves a hard problem 20% of the time. Sample 10 answers. If the attempts are independent, the chance that all 10 fail is $0.8^{10} \approx 0.11$ , so the chance at least one is right is about $0.89$ . This is pass@k, the most common way to report how good a model is at problems where you can check the answer. The independence page shows why real models fall short of this number.

Two ways to read a probability

“The probability of rain tomorrow is 70%” can mean two things:

Frequency: on days like this one, it rains 70% of the time.
Belief: given what I know, I’m 70% confident it will rain.

The rules are identical either way. Language-model papers use both, often in the same paragraph. The model’s next-token probability is a frequency-style object (sample many times and count). A reward model’s “probability that this answer is preferred” or a model saying “I’m 80% sure” is a belief, and whether that belief matches reality is the subject of calibration.

Where this shows up

pass@k is an “at least one” probability: Does RL teach new skills?.
Sampling many answers and picking one is a bet on these rules: exploration.
The next page turns outcomes into numbers: random variables.

going deeper

Notation

A sample space is usually written $\Omega$ and an event $A \subseteq \Omega$ . “And” and “or” are written as set intersection and union:

P(A \cap B) \;\; \text{(A and B)} \qquad P(A \cup B) = P(A) + P(B) - P(A \cap B) \;\; \text{(A or B)} \qquad P(A^c) = 1 - P(A) \;\; \text{(not A)}

The axioms (Kolmogorov, 1933), for a probability measure $P$ :

P(A) \ge 0, \qquad P(\Omega) = 1, \qquad P\Big(\bigcup_i A_i\Big) = \sum_i P(A_i) \text{ for disjoint } A_i

For $k$ independent attempts, each succeeding with probability $p$ :

P(\text{at least one success}) = 1 - (1 - p)^k

In papers

For a language model, outcomes are sequences $y = (y_1, \dots, y_T)$ from a vocabulary $\mathcal{V}$ , so the sample space is $\mathcal{V}^*$ , every finite string. An event such as “the answer is correct” is written with an indicator: $\mathbb{1}[\text{correct}(y)]$ is 1 if the answer is right and 0 otherwise. Its probability is then an expectation, a form you’ll see everywhere:

P(\text{correct}) = \sum_{y} \pi(y \mid x)\, \mathbb{1}[\text{correct}(y)] = \mathbb{E}_{y \sim \pi(\cdot \mid x)}\big[\mathbb{1}[\text{correct}(y)]\big]

The pass@k formula above is the one used for code models in Chen et al., 2021; the Monte Carlo page shows the unbiased estimator they actually use.