Reasoning as a hidden variable

Treat the chain of thought as an unobserved variable between question and answer. Majority voting, rejection-sampling fine-tuning and several RL methods turn out to be standard tools for latent-variable models.

Key idea: Latent variables, marginalisation, majority voting as the mode of a marginal, EM and the evidence lower bound
Real world: A doctor never observes a disease directly, only symptoms. Reasoning about the hidden cause, and averaging over the possible causes, gives a better prediction of what happens next than any single guess about the cause.
In an LLM: A reasoning model writes a long chain of thought $z$ , then a final answer $y$ . Only $y$ is graded. The probability of an answer is a sum over every chain of thought that could lead to it, and different chains reaching the same answer reinforce each other.

Latent variables

A latent variable is something that shapes what you see but isn't itself observed. Models with latent variables describe the observed thing $y$ through a hidden one $z$ :

p(y \mid x) = \sum_z p(z \mid x)\, p(y \mid x, z)

Summing out the hidden variable is marginalisation. It's the law of total probability, with $z$ ranging over every possibility.

For reasoning models, $x$ is the question, $z$ the chain of thought, and $y$ the final answer. The sum over $z$ ranges over every possible reasoning text, so it can never be computed. But it can be sampled.

example · a diagnosis

A cough could be a cold (70%), flu (25%) or pneumonia (5%). The chance of needing a hospital is 1%, 5% and 60% respectively. Marginalising:

0.7 \times 0.01 + 0.25 \times 0.05 + 0.05 \times 0.6 = 0.0495 \approx 0.05

The rare cause contributes most of the risk.

example · three ways to the same answer

Asked for $17 \times 24$ , a model might multiply directly, split it as $17 \times 20 + 17 \times 4$ , or compute $20 \times 24 - 3 \times 24$ . Each path has its own probability and each can slip. The probability of the final answer "408" is the sum over all the paths that end there.

Majority voting estimates the marginal

Self-consistency samples many chains of thought, throws away the reasoning, and returns the most common final answer. In probability terms: it estimates the marginal $p(y \mid x)$ by counting, and returns its mode. Many different, individually unlikely chains that agree on one answer add up.

That's also its limit. If the most likely answer is wrong, voting converges to the wrong answer with more and more confidence. By contrast, pass@n ("did any sample get it right?") keeps rising with more samples, but needs a verifier to pick out the right one.

Try it

P(correct answer)0.35P(most common wrong answer)0.25

15 sampled chains of thought, reduced to their final answers

✓✗4✓✗7✗4✗1✗1✓✓✗7✓✓✗1✗2✓

vote: right (7 of 15) · a verifier would find the right one

pass@n: any sample right (needs a verifier)majority vote: most common answer rightone sample

majority of 12894%

pass@128100%

The right answer is the most common one, so voting converges to it, even though any single sample is usually wrong. Majority voting estimates the mode of the answer distribution, with the reasoning path marginalised out. Make a wrong answer more common than the right one.

One maths question, sampled many times. Each sample ends in a final answer; reasoning paths differ, answers repeat.

example · polling a crowd

Ask 100 people to guess the number of sweets in a jar and take the most common guess. Usually better than a typical individual, as long as the crowd isn't biased the same way. If most people anchor on the same wrong number, asking more people only makes the vote more confident in it.

example · coverage keeps growing

Brown et al. (2024) found that on coding and maths tasks, the fraction of problems solved by at least one of $k$ samples kept rising, roughly log-linearly, all the way to 10,000 samples. Majority voting and reward-model selection plateaued after a few hundred. The bottleneck is picking the right sample, not generating it.

Training a latent-variable model

To train by maximum likelihood on question–answer pairs, you'd maximise $\log p(y \mid x)$ , the log of a sum over all chains of thought. Its gradient is

\nabla \log p(y \mid x) = \mathbb{E}_{z \sim p(z \mid x, y)}\big[\nabla \log p(z, y \mid x)\big]

an average over the posterior $p(z \mid x, y)$ : chains of thought, weighted by how likely they are given that they reach the right answer. By Bayes, that posterior is proportional to $p(z \mid x)\, p(y \mid x, z)$ .

You can sample from it by rejection: generate chains, keep the ones that reach the correct answer. Then fine-tune on what you kept. That's exactly STaR and rejection-sampling fine-tuning, which turn out to be an approximate version of the classic expectation–maximisation (EM) algorithm:

E-step: infer the hidden variable (sample chains that reach the right answer).
M-step: train the model on them as if they had been observed.

And RL with a correct/incorrect reward is closely related: $\mathbb{E}_{z}[\mathbb{1}(\text{correct})]$ is $p(y^* \mid x)$ itself, so the policy gradient for that reward is the gradient of the marginal likelihood, scaled by $p(y^* \mid x)$ .

Where this shows up

Self-consistency and majority voting at test time: Does RL teach new skills?.
STaR, rejection-sampling fine-tuning, expert iteration: distillation.
RLVR: the reward is "the latent reasoning produced the right answer": RLVR.
Sampling the posterior over chains with MCMC (TRICE) or a learned sampler (GFlowNets): MCMC.

going deeper

Marginal likelihood and its gradient

\log p_\theta(y \mid x) = \log \sum_z p_\theta(z \mid x)\, p_\theta(y \mid x, z), \qquad \nabla_\theta \log p_\theta(y \mid x) = \mathbb{E}_{z \sim p_\theta(z \mid x, y)}\big[\nabla_\theta \log p_\theta(z, y \mid x)\big]

The identity follows from $\nabla \log f = \nabla f / f$ and Bayes' rule.

The evidence lower bound

For any distribution $q(z)$ over chains of thought, Jensen's inequality gives

\log p(y \mid x) \ \ge\ \mathbb{E}_{z \sim q}\big[\log p(z, y \mid x) - \log q(z)\big] = \log p(y \mid x) - \mathrm{KL}\big(q(z) \,\|\, p(z \mid x, y)\big)

This is the ELBO. It's tight when $q$ equals the posterior. EM alternates between making $q$ close to the posterior (E-step) and maximising the bound in $\theta$ (M-step). The same bound, with a noise-adding process as the latent variable, is the training loss of diffusion models.

Self-consistency

\hat y = \arg\max_y \sum_{i=1}^{n} \mathbb{1}[y_i = y], \qquad y_i \sim p(y \mid x) \text{ via } z_i \sim p(z \mid x)

As $n \to \infty$ , $\hat y \to \arg\max_y p(y \mid x)$ , the mode of the marginal.

Papers

Self-Consistency Improves Chain of Thought Reasoning in Language Models (Wang et al., 2022)
Large Language Monkeys: Scaling Inference Compute with Repeated Sampling (Brown et al., 2024)
STaR: Bootstrapping Reasoning With Reasoning (Zelikman et al., 2022)
Training Chain-of-Thought via Latent-Variable Inference (Phan, Hoffman et al., 2023): TRICE, MCMC-EM for chains of thought
Amortizing Intractable Inference in Large Language Models (Hu et al., 2023): GFlowNets to sample the posterior over reasoning
Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models (Chow et al., 2024): training the model for the selection rule it will be used with