The training objective

Key idea

targets = inputs shifted by one; loss = average of −log p(correct next token) over every position; perplexity = e^loss

Real world

A guessing game with a book: cover everything after a word, guess the next word, uncover it, score yourself, and move on one word. A good reader rarely gets a nasty surprise.

In an LLM

On “The cat sat on the mat because it was tired.”, GPT-2 small gives the real next token an average loss of 3.88: it is about as unsure as if it were choosing between 48 equally likely tokens at each step.

Free labels

Most machine learning needs labelled examples: photos tagged “cat”, emails tagged “spam”. Language models don’t. Any text already contains its own answers, because the label for each position is simply the token that comes next.

Take a sentence of $T$ tokens. The inputs are tokens 1 to $T-1$; the targets are tokens 2 to $T$, the same text shifted one place left. At position $t$ the model sees tokens $1 \ldots t$ and must predict token $t+1$.

Score every position at once

At each position the model outputs a probability for every token in the vocabulary (the unembedding). The loss at that position is

the negative log of the probability it gave the token that actually came next. The loss for the sequence is the average over positions:

This is the cross-entropy between the real next token and the model’s prediction. Here it is on real sentences, with GPT-2 small’s real probabilities:

Thanks to the causal mask, position $t$ can only see tokens up to $t$, so one forward pass produces a prediction at every position, each one honest about the future. A training batch of 1,024 tokens is 1,023 prediction problems solved in one go. That is a big reason transformers train so efficiently.

Teacher forcing

Look at the “GPT-2 guess” row. After “The cat sat on the”, GPT-2’s top guess is “ floor”, not “ mat”. During training that doesn’t matter: the next input is still the real “ mat”. Every position gets the true history, never the model’s own guesses. This is Teacher forcing.

It is what makes parallel training possible (no position has to wait for an earlier guess), and it keeps one early mistake from derailing the rest of the sequence. The price is a mismatch: at generation time the model does read its own guesses, which it never practised. In practice large models cope well.

Why −log p

The loss must reward putting probability on the right answer, and it must be something we can add up over positions. The negative log does both:

• Certain and right costs nothing: $-\log 1 = 0$.
• Confident and wrong is ruinous: as $p \to 0$, $-\log p \to \infty$.
• It adds up: the probability of a whole sequence is a product of next-token probabilities (the chain rule of probability), and the log of a product is a sum. So minimising the summed loss is the same as maximising the probability of the whole training text.

Perplexity

Losses are hard to picture, so people often report perplexity, $e^{\mathcal{L}}$. A perplexity of 48 means the model is, on average, as unsure as if it were choosing uniformly between 48 tokens. Lower is better. A model that guesses uniformly over GPT-2’s 50,257 tokens has loss $\ln 50{,}257 = 10.82$ and perplexity 50,257. More in perplexity.

What a loss curve looks like

Plot the loss against training steps and almost every language model draws the same shape:

• Step 0: random weights give close to uniform predictions, a loss near 10.8 for a 50,257-token vocabulary.
• The first few hundred steps: a steep drop as the model learns which tokens are common (“ the”, “,”, “.”). Just predicting token frequencies takes the loss a long way down.
• Then a long, slow slope: grammar, then facts, then longer-range patterns, each worth a little less loss. Bigger models and more data keep pushing it down, following smooth scaling laws.

For scale: the released GPT-2 small reaches a loss of about 3.1 on web text (perplexity about 22), and GPT-2 XL about 2.5. Watch a real curve being drawn in train a tiny transformer live.

The loss can never reach zero on real text. Language is genuinely uncertain: after “I had a cup of”, both “ tea” and “ coffee” are right, and the best a model can do is split its bets. The irreducible part of the loss is the entropy of language itself.

logits = model(tokens[:, :-1])                 # (B, T-1, vocab): predictions at every position
targets = tokens[:, 1:]                        # the same text shifted one place left
loss = F.cross_entropy(logits.reshape(-1, logits.size(-1)), targets.reshape(-1))