The MLP

Key idea

MLP(x) = W_down · act(W_up · x); SwiGLU adds a gate; neurons are keys (patterns) and values (what to write)

Real world

A reference librarian. You hand over one question; they check it against everything they know, the few matching entries light up, and they hand back a note built from those entries. They never talk to the other people in the queue.

In an LLM

At the word “ France”, neurons whose keys match “a country” fire, and their values add what the model knows about it, such as its capital. Later attention carries that to the end of “The capital of France is”, where “ Paris” is pushed up.

Widen, bend, narrow

The second half of every transformer block is a small neural network called the MLP (feed-forward layer) (multi-layer perceptron, also called the feed-forward layer). It works on one token’s vector at a time. In GPT-2:

1. Up-project. Multiply the 768-number vector by a matrix $W_{\text{up}}$ (768 × 3072). Now there are 3072 numbers, four times as many. Each one is a neuron: the dot product of the input with one column of $W_{\text{up}}$.
2. Activate. Apply an Activation function to each of the 3072 numbers separately.
3. Down-project. Multiply by $W_{\text{down}}$ (3072 × 768) to get back to 768 numbers, which are added to the residual stream.

The activation is essential. Two matrix multiplications in a row are just one matrix multiplication ($x W_1 W_2 = x (W_1 W_2)$), so without the bend in the middle the MLP could only compute a linear map. The activation lets each neuron act as a switch: on for some inputs, off for others.

• ReLU ($\max(0, x)$) was the default for years. It is exactly zero for every negative input, so a neuron that is always negative gets no gradient and can stop learning.
• GELU (Hendrycks and Gimpel, 2016) is a smooth ReLU used by GPT-2 and BERT: slightly negative inputs leak through a little.
• SiLU ($x \cdot \sigma(x)$, also called Swish) looks almost the same and is what Llama-family models use, inside SwiGLU.

Step through one token

Notice that the MLP never looks at other tokens. Change the token and only that token’s row is computed. All the movement of information between positions happens in attention; the MLP’s job is to process what has arrived.

SwiGLU: a gate

Most models since 2023 use SwiGLU (Shazeer, 2020) instead of a plain activation. It makes two projections of the input: a gate and an up vector. The gate goes through SiLU and is multiplied, number by number, with the up vector:

Switch the widget above to SwiGLU to see it. The gate decides how open each neuron is; the up vector decides what flows through. It costs a third matrix, so models shrink the hidden width from 4d to about 8/3·d to keep the parameter count the same (Llama 3 8B: d = 4096, hidden 14,336 ≈ 3.5d). In Shazeer’s experiments it gave a consistently lower loss for the same compute, and nobody has fully explained why.

Neurons as memories

Geva et al., 2021 suggested a clean way to read an MLP. Look at one neuron $i$:

• its column of $W_{\text{up}}$ is a key: the neuron fires when the input points in that direction;
• its row of $W_{\text{down}}$ is a value: when the neuron fires, that vector is added to the residual stream, scaled by how strongly it fired.

So the MLP is a big soft lookup table: thousands of “if the input looks like this, add that” rules, all checked at once. They found real neurons, in a trained language model, whose keys match human-readable patterns (text ending in a particular word, a topic like TV shows) and whose values push up the words that tend to come next. Here is a toy version with five readable neurons:

Real neurons are rarely this tidy. Most respond to several unrelated patterns at once (they are polysemantic), because the model has many more facts to store than it has neurons; see interpretability. But the key → value picture is a good first model, and it explains why editing a single MLP layer can change a fact the model “knows” (Meng et al., 2022).

Most of the parameters

The MLP is also where most of the weights are. Per block, attention has four $d \times d$ matrices ($4d^2$ weights) and a GPT-2 MLP has two $d \times 4d$ ones ($8d^2$). So the MLP holds two thirds of every block:

Llama’s share is higher because it uses grouped-query attention (only 8 key–value heads, which shrinks $W_K$ and $W_V$; see efficient attention) and a wide SwiGLU. This is also why mixture-of-experts models split the MLP, not attention: it is the biggest part, and each token only needs some of its memories.

class MLP(nn.Module):                  # GPT-2
    def __init__(self, d):
        super().__init__()
        self.up = nn.Linear(d, 4 * d)
        self.down = nn.Linear(4 * d, d)
    def forward(self, x):
        return self.down(F.gelu(self.up(x), approximate="tanh"))
 
class SwiGLU(nn.Module):               # Llama
    def __init__(self, d, hidden):
        super().__init__()
        self.gate = nn.Linear(d, hidden, bias=False)
        self.up = nn.Linear(d, hidden, bias=False)
        self.down = nn.Linear(hidden, d, bias=False)
    def forward(self, x):
        return self.down(F.silu(self.gate(x)) * self.up(x))