What an LLM actually does

A language model does one thing. It reads a list of tokens and gives a probability for every possible next token. Generation is that one step in a loop. Pick a token, append it, run again.

Key idea: Next-token prediction, the vocabulary, logits, probabilities and the generation loop
Real world: Your phone keyboard suggests three next words as you type. Tap one and it suggests three more, based on everything typed so far.
In an LLM: Given “The cat sat on the”, GPT-2 gives “ floor”, “ bed” and “ couch” the highest probabilities of its 50,257 tokens. A chatbot’s whole answer is this step repeated a few hundred times.

One job

Strip away the chat interface and a large language model is a function with one job:

In: a list of tokens. Out: a probability for every token in its vocabulary, saying how likely each one is to come next.

A is a chunk of text, usually a word or part of a word. The model has a fixed vocabulary of them (50,257 in GPT-2, about 150,000 in recent models). Each one has an ID number, and the model only ever sees those numbers. How text is cut into tokens is the subject of the tokenization section.

Generation is a loop

To write a sentence, you run that one job over and over:

Feed in the tokens so far.
The model produces a score (a logit) for every token in the vocabulary.
Softmax turns the scores into probabilities.
Pick one token, usually by sampling.
Append it and go back to step 1.

Step through it:

1. read the tokens2. run the model3. scores → probabilities4. pick one5. append it

the text so far

The·cat

↓

the model

a stack of layers

↓

what comes after “·cat”? (only the top 4 shown)

·sat…

·slept…

·is…

·ran…

score · probability (softmax of the scores ÷ temperature)

show token IDstemperature1.0

The text is 2 tokens. The model only ever sees the numbers (tick “show token IDs”). This toy looks only at the last token; a real LLM looks at all of them.

A toy model that only looks at the last token, but the loop is exactly the one every LLM runs. Step through one stage at a time, or press ‘Whole step’ to generate a token in one go.

Every answer from every chatbot is produced like this, one token at a time. The model never plans a whole sentence and writes it out; it only ever predicts the next token. Anything that looks like planning has to happen inside that one prediction, or be written out in tokens as it goes, which is why “thinking” models write their reasoning down.

What a real model predicts

The toy above only looks at the last word. A real model looks at all of them. Here is GPT-2 small, a real 124-million-parameter model from 2019, reading a sentence. At every position it has produced a probability for each of its 50,257 tokens; these are its top 5.

Loading GPT-2 data…

GPT-2 small’s real predictions.

Notice a few things. After “The cat sat on the”, “ mat” isn’t even in GPT-2’s top 5: “ floor”, “ bed” and “ couch” are just as plausible, and the probability is spread thin across them. In fact the top five together get only about 30%; the other 70% is spread over the remaining 50,252 tokens (the panel shows this), which is why sampling does not always pick the favourite. After “because”, GPT-2 splits almost evenly between “ it”, “ she” and “ he”: it can’t know how the writer will refer to the cat. And after “sat on”, it gives “ the” 45%: when the grammar pins the next word down, the model is confident.

example · phone keyboard suggestions

The three words above your phone keyboard are a next-word predictor. Type “I’ll be there in” and it offers “a”, “the”, “5”. Tap “5” and it offers “minutes”. It is the same loop: predict, pick, append. A phone keyboard uses a small model and only shows its top three; an LLM uses a huge one and samples from the full distribution.

example · a chatbot answer

You ask “What is the capital of France?”. The chat software wraps that in a template and sends the tokens to the model. The model predicts “The” (high probability), it is appended, then “ capital”, “ of”, “ France”, “ is”, “ Paris”, “.”, then a special end-of-answer token, and the loop stops. That is eight forward passes through the model: seven tokens of answer and one end token.

Where the work happens

Everything interesting is inside “the model” box: how the tokens become numbers, how they share information through attention, and how the result becomes a score for every token. The rest of this course opens that box. The next page shows the whole map at once.

going deeper

The model as a probability distribution

Formally, a language model with parameters $\theta$ defines the probability of a sequence $x_1, \dots, x_T$ through the chain rule:

p_\theta(x_1, \dots, x_T) = \prod_{t=1}^{T} p_\theta(x_t \mid x_1, \dots, x_{t-1})

The network computes one vector of logits $z \in \mathbb{R}^{|V|}$ per position, and

p_\theta(x_t = v \mid x_{<t}) = \operatorname{softmax}(z)_v = \frac{e^{z_v / \tau}}{\sum_{u \in V} e^{z_u / \tau}}

where $\tau$ is the sampling temperature ( $\tau = 1$ in training). The chain rule is covered in the chain rule and autoregressive models; softmax and temperature in softmax.

In code

tokens = tokenizer.encode("The cat sat on the")
for _ in range(20):
    logits = model(tokens)[-1]          # scores for the next token only
    probs = softmax(logits / temperature)
    next_token = sample(probs)
    tokens.append(next_token)
    if next_token == tokenizer.eos_id:
        break
print(tokenizer.decode(tokens))

In papers

GPT-2 (the model in the widget): Radford et al., 2019.
The transformer: Vaswani et al., 2017.