Linear layers

A layer of a neural network is a matrix multiplication plus a bias, y = xW + b. Stacking two such layers gives just one bigger matrix, so on their own they can only draw straight lines. A simple bend between them, like ReLU or GELU, is what lets a network learn curved shapes.

Key idea: y = xW + b; a neuron is a column of W; parameters = in × out + out; two linear layers collapse to one; a nonlinearity (ReLU, GELU) between them makes depth useful
Real world: A smoothie bar’s recipe card: each smoothie is a fixed mix of fruit amounts. Doubling the fruit doubles every smoothie, and two recipe cards used one after another are just one combined card.
In an LLM: Every GPT-2 block contains four linear layers. The MLP widens each token from 768 to 3,072 numbers, applies GELU, and narrows back to 768. Without the GELU, the two MLP matrices would multiply into one 768 × 768 matrix and the MLP would gain nothing from its 3,072-wide middle.

A layer is a matrix

A takes a vector $\mathbf{x}$ with $n_{\text{in}}$ numbers and returns a vector $\mathbf{y}$ with $n_{\text{out}}$ numbers:

\mathbf{y} = \mathbf{x} W + \mathbf{b}

$W$ is the weight matrix, shape $n_{\text{in}} \times n_{\text{out}}$ , and $\mathbf{b}$ is the bias, one number per output. Each output is a dot product of the input with one column of $W$ , plus its bias. In a network diagram, each output circle is a neuron and each line into it is one weight; the diagram and the matrix hold exactly the same numbers.

W3 × 2

y1y2x10.8−0.5x20.30.9x3−0.60.4

y1y20.1−0.2

x11.0x20.5x3−1.0bias b

y1 = 1.0×0.8 + 0.5×0.3 + −1.0×(−0.6) + 0.1 = 1.65

weights3 × 2 = 6

biases2

parameters8

Each output is a dot product: the inputs times one column of W, plus a bias. All outputs together are one vector–matrix product, y = xW + b. A “neuron” in a diagram is just one column of a matrix.

Three inputs, two outputs. Each line is one weight: blue positive, red negative, thicker for bigger. Hover a line or a cell of W: they are the same number.

The number of parameters (learned numbers) is

n_{\text{in}} \times n_{\text{out}} + n_{\text{out}},

the weights plus the biases. For the layer above that is $3 \times 2 + 2 = 8$ .

example · a smoothie recipe card

A smoothie bar has 3 fruits and 2 smoothies. The recipe card says how many grams of each fruit go into each smoothie: a $3 \times 2$ table, $W$ . Given today’s fruit $\mathbf{x}$ , the amount of each smoothie you can make is $\mathbf{x}W$ , and a fixed scoop of yoghurt per smoothie is $\mathbf{b}$ . The recipe card is a linear layer: double the fruit and you double the fruit part of every smoothie.

Real sizes

Layer in GPT-2 small (each block)	Shape of W	Parameters (with bias)
Q, K, V for all 12 heads together	768 × 2,304	1,771,776
Attention output	768 × 768	590,592
MLP up	768 × 3,072	2,362,368
MLP down	3,072 × 768	2,360,064

Together that is about 7.1 million parameters per block, 85 million for all 12 blocks: more than two thirds of GPT-2 small’s 124 million. The rest is mostly the embedding table. Larger models are the same picture with bigger matrices: one MLP matrix in Llama 3 8B is $4{,}096 \times 14{,}336$ , 58.7 million numbers.

Two linear layers are one linear layer

Stack two layers: $\mathbf{h} = \mathbf{x}W_1 + \mathbf{b}_1$ , then $\mathbf{y} = \mathbf{h}W_2 + \mathbf{b}_2$ . Substitute:

\mathbf{y} = (\mathbf{x}W_1 + \mathbf{b}_1)W_2 + \mathbf{b}_2 = \mathbf{x}\,\underbrace{(W_1W_2)}_{\text{one matrix}} + \underbrace{(\mathbf{b}_1W_2 + \mathbf{b}_2)}_{\text{one bias}}

As in the transformations lesson, two maps in a row are one map. However many linear layers you stack, the result is a single linear layer, and a linear layer can only separate things with a straight line (a flat plane in more dimensions). Depth would buy nothing.

The bend that makes depth work

The fix is to apply a simple, fixed, non-linear function to every number between the layers, called a or activation function. The simplest is ReLU: $\max(0, x)$ , which leaves positive numbers alone and turns negatives into 0. GPT-2 uses GELU, a smooth version of the same bend.

Below is a tiny network, 2 inputs → 3 hidden units → 1 output, trained to tell the inner blue points from the outer amber ring. Switch the activation off. The weights stay exactly the same; only the bend disappears.

Loading GPT-2 data…

With ReLU, each hidden unit contributes a “ramp” that switches on along a line, and the output layer adds the three ramps into a surface that is high only in a closed region around the centre. Without it, the network collapses to one matrix, $W_1W_2$ , and one straight boundary.

example · tax bands

Income tax is a bent function: nothing on the first slice of income, then a percentage of everything above a threshold, then a higher percentage above the next. Each band is a ReLU, $\max(0, \text{income} - \text{threshold})$ , times a rate. Add a few of them and you get a curve that no single straight-line rule could match. A network builds its curved decision boundaries the same way: many ReLUs, each switching on at its own threshold, added up by the next layer.

example · the MLP in GPT-2

Every token’s 768 numbers go through $W_{\text{up}}$ (768 × 3,072), then GELU on each of the 3,072 numbers, then $W_{\text{down}}$ (3,072 × 768). Interpretability work finds that many of the 3,072 middle units act like detectors that switch on for a pattern, such as a topic or a phrase, and then write a matching vector back. The switching on is the GELU. See the MLP.

going deeper

Conventions

PyTorch’s nn.Linear(n_in, n_out) stores its weight as $n_{\text{out}} \times n_{\text{in}}$ and computes $\mathbf{y} = \mathbf{x}W^\top + \mathbf{b}$ . GPT-2’s original code stores $n_{\text{in}} \times n_{\text{out}}$ (a “Conv1D”) and computes $\mathbf{x}W + \mathbf{b}$ . Same layer, transposed storage: a common source of shape bugs when porting weights.

Many modern LLMs (Llama, Mistral, Qwen) drop the biases from most linear layers.

Activations

\operatorname{ReLU}(x) = \max(0, x), \qquad \operatorname{GELU}(x) = x\,\Phi(x) \approx \tfrac{x}{2}\Big(1 + \tanh\big(\sqrt{2/\pi}\,(x + 0.044715\,x^3)\big)\Big)

where $\Phi$ is the standard normal cumulative distribution function. Llama-style models use SwiGLU, which multiplies a SiLU-activated branch by a second linear branch.

In code

import torch, torch.nn as nn
layer = nn.Linear(768, 3072)          # weight (3072, 768), bias (3072,)
sum(p.numel() for p in layer.parameters())   # 2362368
 
x = torch.randn(10, 768)              # 10 tokens
layer(x).shape                        # torch.Size([10, 3072])
 
mlp = nn.Sequential(nn.Linear(768, 3072), nn.GELU(), nn.Linear(3072, 768))
 
# Without the GELU the two layers collapse:
A, B = nn.Linear(768, 3072), nn.Linear(3072, 768)
W = A.weight.T @ B.weight.T           # (768, 768)
b = A.bias @ B.weight.T + B.bias
torch.allclose(B(A(x)), x @ W + b, atol=1e-4)   # True

In papers

ReLU in deep networks: Nair and Hinton, 2010; Glorot et al., 2011.
GELU: Hendrycks and Gimpel, 2016. SwiGLU: Shazeer, 2020.
Networks with one bent hidden layer can approximate any continuous function: Cybenko, 1989; Hornik, 1991.
MLP units as key–value memories: Geva et al., 2020.