Matrices as transformations

A matrix is also a machine that moves every point in space. Its columns say where the axis arrows land, and everything else follows, with straight lines staying straight. Scaling, rotating, shearing, reflecting and flattening are all matrices, and doing one after another is multiplying them.

Key idea: A matrix is a linear map; its columns are where the basis vectors land; grid lines stay straight and evenly spaced; determinant = area scale; composing maps = multiplying matrices
Real world: The shadow of a sheet of card on the ground, cast by the sun: a rectangle becomes a slanted parallelogram, but straight lines stay straight and evenly spaced lines stay evenly spaced.
In an LLM: A query matrix W_Q takes every token’s 768-number embedding to a 64-number query. It is one fixed linear map, applied to every token, that keeps the features this head cares about and throws the rest away.

A matrix moves every point

The last lesson showed that $M\mathbf{x}$ is a weighted sum of $M$ ’s columns. Take the two simplest inputs:

M \begin{bmatrix} 1 \\ 0 \end{bmatrix} = \text{column 1}, \qquad M \begin{bmatrix} 0 \\ 1 \end{bmatrix} = \text{column 2}.

The arrows $\hat\imath = (1, 0)$ and $\hat\jmath = (0, 1)$ are the basis vectors: one step along each axis. Every point $(x, y)$ is $x$ steps of $\hat\imath$ plus $y$ steps of $\hat\jmath$ , so after the matrix it is $x$ steps of column 1 plus $y$ steps of column 2. Knowing where the two basis vectors land tells you where everything lands.

That is the second way to read a matrix: as a , a transformation of space. Linear means two things: the origin stays put, and grid lines stay straight, parallel and evenly spaced. Curves never appear.

Warp the grid

Drag the green and red dots (the two columns). Then try the presets: each one animates from “nothing moved” to the new matrix.

M =1.00−0.500.501.00

determinant1.25

area of the square becomes1.25×

The column (1.00, 0.50) is where (1, 0) goes, and (−0.50, 1.00) is where (0, 1) goes. Grid lines stay straight, parallel and evenly spaced. The unit square becomes the shaded shape, and every area is multiplied by the determinant, 1.25.

Drag the green and red dots: they are where the arrows (1, 0) and (0, 1) land, and they are the two columns of the matrix. Every other point follows. Presets animate from no change to the new matrix.

Matrix	Columns	What it does	Determinant
scale	(1.5, 0), (0, 0.75)	stretch x, squash y	1.125
rotate 30°	(0.87, 0.5), (−0.5, 0.87)	turn everything	1
shear	(1, 0), (1, 1)	slide rows sideways	1
reflect	(−1, 0), (0, 1)	mirror left–right	−1
project	(1, 0.5), (0.5, 0.25)	flatten onto a line	0

The determinant: how areas change

The shaded square is the unit square, with sides $\hat\imath$ and $\hat\jmath$ . After the matrix it becomes a parallelogram, and its area is the :

\det \begin{bmatrix} a & b \\ c & d \end{bmatrix} = ad - bc

Every area in the plane is multiplied by the same number. Two special cases matter:

Negative determinant: space has been flipped over, like a mirror. The F reads backwards.
Zero determinant: the plane has been squashed flat onto a line (or a point). Different points land in the same place, so the map can’t be undone. Switch the shape to “GPT-2 words” and press “project”: whole groups of words merge.

example · a shadow in the sunlight

Hold a rectangular sheet of card in the sunlight, above flat ground. Its shadow is a slanted parallelogram. Straight edges stay straight, evenly spaced lines drawn on the card stay evenly spaced in the shadow, and the middle of the card lands in the middle of the shadow: the card-to-shadow step is a linear map (plus a shift). Turn the card edge-on to the sun and the shadow collapses to a single line: determinant 0, and you cannot tell from the shadow what the card looked like. (A projector pointed at a wall at an angle is not linear: it makes a trapezoid, because rays from one point spread out. Sunlight is parallel, which is why its shadows are.)

One map after another

Apply $A$ and then $B$ : each point goes to $A\mathbf{x}$ and then to $B(A\mathbf{x})$ . Because matrix multiplication is associative, that is $(BA)\mathbf{x}$ : doing two maps in a row is one map, their product. Note the order: $BA$ means “ $A$ first”, read right to left. And since $BA$ and $AB$ are usually different, the order of operations matters:

first

then

A: rotate 90°

[0 −1; 1 0]

A, then B = BA

[1 −1; 1 0]

B, then A = AB

[0 −1; 1 1]

“rotate 90°, then shear” and the reverse give different pictures, so BA ≠ AB. Doing one matrix after another is a single matrix, their product, and it is written right to left: BA means “A first”. Two layers of a network are combined the same way.

Pick two moves. The middle picture does the first, then the second; the right one does them the other way round. The dashed F is the original.

This is the fact behind the next lesson: stacking linear layers gives just another linear layer.

In an LLM: projecting into a new space

A transformer’s matrices are linear maps between spaces of different sizes. $W_Q$ in one head of GPT-2 small is $768 \times 64$ : it maps each 768-number embedding to a 64-number query. A map to fewer dimensions has to flatten something, like the “project” preset, and that is the point: the head keeps only the directions that matter for what it is looking for, and different heads keep different ones.

example · query space in GPT-2

GPT-2 small has 12 heads in each of 12 layers, and each has its own $W_Q$ , $W_K$ and $W_V$ . Each $W_Q$ projects the same token vectors into its own 64-dimensional “query space”. A head that tracks the previous token keeps position information; a head that resolves “it” keeps whatever separates “cat” from “mat”. The projection is learned: training adjusts the 768 × 64 = 49,152 numbers of $W_Q$ so that the right queries and keys end up pointing the same way. See multi-head attention.

going deeper

Rows or columns?

This page uses the maths convention: vectors are columns and the matrix acts from the left, $M\mathbf{x}$ , so the columns of $M$ are the images of the basis vectors. Transformer code stores vectors as rows and writes $\mathbf{x}W$ ; then the rows of $W$ are the images of the basis vectors. The two are transposes of each other: $\mathbf{x}W = (W^\top \mathbf{x}^\top)^\top$ . Same map, different bookkeeping.

Facts

$\det(AB) = \det A \cdot \det B$ : area scalings multiply.
$A$ can be undone (has an inverse $A^{-1}$ with $A^{-1}A = I$ ) exactly when $\det A \ne 0$ .
A linear map satisfies $M(\mathbf{x} + \mathbf{y}) = M\mathbf{x} + M\mathbf{y}$ and $M(c\mathbf{x}) = cM\mathbf{x}$ . Adding a constant, $M\mathbf{x} + \mathbf{b}$ , is called affine.

In code

import numpy as np
M = np.array([[1.0, 1.0],     # shear: columns (1, 0) and (1, 1)
              [0.0, 1.0]])
M @ np.array([1.0, 0.0])      # array([1., 0.])  first column
M @ np.array([0.0, 1.0])      # array([1., 1.])  second column
np.linalg.det(M)              # 1.0: areas unchanged
 
R = np.array([[0, -1], [1, 0]])   # rotate 90°
S = np.array([[1, 1], [0, 1]])    # shear
np.array_equal(S @ R, R @ S)      # False: order matters