Importance sampling in depth

Reweighting samples from one distribution to learn about another. Why it works, how to tell when it's failing (effective sample size), and the fixes LLM training uses for long answers.

Key idea: Importance weights, effective sample size, self-normalisation, truncation, and per-token vs per-sequence ratios
Real world: A survey reached twice as many over-60s as the population has. Count each over-60's answer at half weight and each younger person's at a bit more, and the survey speaks for everyone. But if only 3 people under 30 replied, their huge weights make the result fragile.
In an LLM: Answers generated by a slightly older copy of the model, or by a fast inference engine, train the current model. Each answer is reweighted by how much more or less likely the current model makes it. For a 10,000-token answer, that weight can be astronomically large or tiny.

Importance sampling in the RL course introduces the idea. This page covers the probability behind it: when it works, how to measure how badly it's failing, and the corrections you'll meet in papers.

The identity

You want an average under a target distribution $p$ , but your samples come from a proposal (or behaviour) distribution $q$ . Multiply each sample by its importance weight $w = p(x) / q(x)$ :

\mathbb{E}_{x \sim p}[f(x)] = \mathbb{E}_{x \sim q}\Big[\frac{p(x)}{q(x)} f(x)\Big]

It's exact, with one condition: $q$ must give some chance to everything $p$ can produce. If $q(x) = 0$ where $p(x) > 0$ , those outcomes are never sampled, and no reweighting can recover them.

example · a survey with too many retirees

The population is 25% over-60s; the survey got 50%. Weight each over-60 by $0.25/0.5 = 0.5$ and each younger person by $0.75/0.5 = 1.5$ . The reweighted average matches the population, without surveying anyone new.

example · one PPO step later

A token was sampled when the model gave it probability 0.20; after one gradient step the model gives it 0.24. Its weight is $0.24/0.20 = 1.2$ : this sample counts 20% more when estimating what the current model would get. That ratio is exactly PPO's $\rho$ .

When it fails: effective sample size

The estimate stays unbiased however different $p$ and $q$ are. What breaks is the variance. If $p$ puts weight where $q$ rarely samples, a few samples get enormous weights and dominate the average. Your 1,000 samples behave like 5.

The effective sample size (ESS) puts a number on this:

\text{ESS} = \frac{\big(\sum_i w_i\big)^2}{\sum_i w_i^2}

If all weights are equal, ESS equals the number of samples. If one weight dwarfs the rest, ESS is close to 1. It's the standard diagnostic in statistics, and the thing to look at when an off-policy method misbehaves.

example · three young respondents

If 3 of 1,000 respondents are under 30 but the population is 20% under 30, each of those three gets weight about 67, while everyone else gets about 0.8. The ESS is about 70, not 1,000, and the "young people" part of the result is the opinion of three individuals.

example · a stale batch

In asynchronous RL, a batch was sampled four model versions ago. Most token weights are near 1, but a few tokens the model has since learned to avoid now have weights near 0, and a few it has come to favour have weights of 3 or more. Logging the ESS of a batch shows how much its staleness actually costs.

Three ways to tame the weights

Method	What it does	Cost
Self-normalised IS	Divide by the sum of the weights instead of by n	Slightly biased; can't blow up past the largest $f$
Truncated IS (TIS)	Cap each weight at C	Biased toward the proposal; bounded variance
Masking (IcePop, MIS)	Drop samples whose weight is outside a trusted range	Throws data away; ignores those samples entirely

Try it

target shift μ1.50proposal width σ1.00cap C5.0

proposal q = N(0, σ²): where samples come fromtarget p = N(μ, 1); shaded: P(x > 1)

solid line: each row’s averagetruth

truth0.691

effective sample size21 of 100

ordinary: bias / sd-0.002 / 0.274

self-norm.: bias / sd-0.021 / 0.092

truncated: bias / sd-0.200 / 0.125

A few samples now carry most of the weight: see the tall lollipops on the right. Ordinary IS still centres on the truth, but spreads out: the effective sample size is the number of equally weighted samples this batch is really worth. Capping weights at 5.0 pulls the truncated estimate down by 0.20 but narrows it: bias traded for variance.

Estimate P(x > 1) under the target N(μ, 1) using 100 samples from the proposal N(0, σ²). Each estimate is repeated 1,000 times.

Long answers: per-token and per-sequence weights

An answer's probability is the product of its tokens' probabilities (chain rule), so the exact weight for a whole answer is the product of its token weights:

w(y) = \prod_{t=1}^{T} \frac{\pi_\theta(y_t \mid y_{\lt t})}{\pi_\text{old}(y_t \mid y_{\lt t})}

Take logs and it's a sum of $T$ small random terms. Sums of many random terms spread out like $\sqrt{T}$ (the central limit theorem), so the log-weight spreads like $\sqrt T$ and the weight itself spreads exponentially. For thousand-token answers the exact correction is unusable.

What methods do instead:

Per-token weights (PPO, GRPO): weight each token's gradient by its own ratio. Bounded, but it ignores that earlier tokens changed what later tokens see. It's an approximation that's fine while the two policies stay close, which clipping enforces.
Geometric mean (GSPO): one weight per answer, $w(y)^{1/T}$ . The spread now shrinks like $1/\sqrt T$ . Stable, at the price of no longer being an exact correction.
Truncate or mask per token (TIS, IcePop): for the sampler–trainer mismatch, cap or drop tokens whose ratio between engines is extreme.

example · compound interest

A 1% monthly difference in interest rate doesn't sound like much. Over 30 years, it's a factor of $1.01^{360} \approx 36$ . Small per-step ratios multiplied over many steps stop being small.

example · numerical noise in a long answer

The trainer and inference engine disagree by about ±0.01 in log-probability per token. Over 10,000 tokens, the sequence log-weight is off by about $0.01 \times \sqrt{10{,}000} = 1$ , a factor of $e \approx 2.7$ either way, purely from rounding differences.

Where this shows up

PPO and GRPO ratios: PPO, GRPO.
GSPO, CISPO, TIS and other ratio fixes: GRPO fixes, sampler–trainer mismatch.
Off-policy and asynchronous RL: async rollouts.
Speculative decoding uses the same ratio $p/q$ to accept or reject: rejection sampling.

going deeper

Estimators

With $x_i \sim q$ and $w_i = p(x_i)/q(x_i)$ :

\hat\mu_\text{IS} = \frac1n \sum_i w_i f(x_i), \qquad \hat\mu_\text{SNIS} = \frac{\sum_i w_i f(x_i)}{\sum_i w_i}, \qquad \hat\mu_\text{TIS} = \frac1n \sum_i \min(w_i, C)\, f(x_i)

Self-normalised IS also works when $p$ is only known up to a constant, $p(x) \propto \tilde p(x)$ , because the constant cancels. That matters for tilted distributions and sequential Monte Carlo, where the target is never normalised.

Variance and the χ² divergence

\operatorname{Var}_q[w] = \mathbb{E}_q[w^2] - 1 = \chi^2(p \,\|\, q), \qquad \frac{\text{ESS}}{n} \approx \frac{1}{1 + \chi^2(p \| q)}

If each token's log-ratio is roughly $\mathcal N(-\sigma^2/2, \sigma^2)$ under the sampler (so each token weight has mean 1), the sequence log-weight is $\mathcal N(-T\sigma^2/2,\, T\sigma^2)$ and $\text{ESS}/n \approx e^{-T\sigma^2}$ : exponential decay in length. The widget's "long answers" mode plots exactly this.

As written in papers

The PPO/GRPO token ratio and the GSPO sequence ratio:

r_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,\lt t})}{\pi_{\theta_\text{old}}(o_{i,t} \mid q, o_{i,\lt t})}, \qquad s_i(\theta) = \Big(\frac{\pi_\theta(y_i \mid x)}{\pi_{\theta_\text{old}}(y_i \mid x)}\Big)^{1/|y_i|} = \exp\Big(\frac{1}{|y_i|} \sum_t \log r_{i,t}(\theta)\Big)

For the sampler–trainer mismatch, $\pi_{\theta_\text{old}}$ is split in two: $\mu$ (what the inference engine computed) and $\pi_\text{old}$ (what the trainer computes). The TIS weight $\min\big(\pi_\text{old}/\mu,\ C\big)$ multiplies the usual loss.

Papers

Owen, Monte Carlo theory, methods and examples, chapter 9 (book): importance sampling and ESS
Group Sequence Policy Optimization (Zheng et al., 2025): length-normalised sequence ratio
Defeating the Training-Inference Mismatch via FP16 (Qi et al., 2025)
Every Step Evolves (Ling Team, 2025): IcePop masking
Yao et al., Your Efficient RL Framework Secretly Brings You Off-Policy RL Training (blog, 2025): TIS