Matrices

A matrix is a rectangle of numbers. Read it as a table, as a stack of row vectors or as a row of column vectors; all three views are used. Its shape, rows × columns, is the first thing to check. The transpose flips it over its diagonal.

Key idea: A matrix is a grid of numbers with shape rows × columns; entry A[i, j]; rows and columns are vectors; the transpose swaps rows and columns
Real world: A spreadsheet of daily weather: one row per day, one column per measurement (temperature, rain, wind). Each row is a day’s vector; each column is one measurement across the week.
In an LLM: A sentence of 5 tokens, each a vector of 768 numbers, is a 5 × 768 matrix. GPT-2’s whole vocabulary is a 50,257 × 768 matrix, the embedding table: one row per token.

A rectangle of numbers

A is numbers arranged in rows and columns. Its shape is written rows × columns: a matrix with 5 rows and 4 columns is “5 by 4”, $5 \times 4$ , and holds 20 numbers. The entry in row $i$ and column $j$ is written $A_{ij}$ or $A[i, j]$ .

X = \begin{bmatrix} 0.1 & 0.0 & 1.0 & 1.0 \\ 1.0 & 0.1 & 0.0 & 0.0 \\ 0.2 & 1.0 & 0.0 & 0.0 \\ 0.0 & 0.3 & 0.8 & 0.0 \\ 0.1 & 0.0 & 1.0 & 0.0 \end{bmatrix} \qquad X_{2,1} = 1.0

This is the toy input from the Transformers course: one row per token of “The cat sat on the”, one column per meaning (thing, action, small word, first token). $X_{2,1} = 1.0$ says that token 2, “cat”, is fully a thing.

Three ways to read it

A table. Look up one number by its row and column.
A stack of rows. Each row is a vector. Here each row is one token’s embedding, so the matrix is the whole sentence. This is how a transformer holds its input: one row per token.
A row of columns. Each column is also a vector: one meaning, measured for every token.

None of these is the “real” one. Matrix multiplication, in the next lesson, uses the rows of the left matrix and the columns of the right one, so you need both.

The transpose

The of $X$ , written $X^\top$ , flips it over its main diagonal (the cells $X_{11}, X_{22}, \dots$ from top left downwards). Row $i$ becomes column $i$ , and the shape turns round: a $5 \times 4$ matrix becomes $4 \times 5$ . In symbols, $(X^\top)_{ij} = X_{ji}$ .

Switch views, then press “transpose” and watch every cell move across the dashed diagonal:

thingactionsmallfirstThecatsatonthe0.10.01.01.01.00.10.00.00.21.00.00.00.00.30.80.00.10.01.00.0

shape5 × 4

numbers20

read it as

A grid of numbers with 5 rows and 4 columns. Rows are tokens, columns are meanings.

A matrix is a rectangle of numbers. Its shape is written rows × columns, here 5 × 4, and X[i, j] means row i, column j. Press “transpose” to watch it flip.

The toy embedding matrix from the Transformers course: 5 tokens × 4 meanings. Switch the view, and hover or tap a cell.

example · a weather spreadsheet

A week of weather in a spreadsheet has 7 rows (days) and 3 columns (temperature, rain, wind): a $7 \times 3$ matrix. Read along a row and you get one day’s vector, $[14, 2.5, 20]$ . Read down a column and you get the week’s temperatures. Transpose it and the days run across the top instead, a $3 \times 7$ matrix with the same 21 numbers. Spreadsheets call this “paste special, transpose”.

An embedding table is a matrix

A model’s vocabulary is stored as a matrix $E$ with one row per token: the embedding table. Looking a token up means taking its row. You can write that lookup as a multiplication: a row vector of zeros with a single 1 in the token’s position (a one-hot vector) times $E$ gives exactly that row.

token:

one-hot(“cat”)1 × 8

0123456701000000

E8 × 4

thingactionsmallfirstThe0.10.01.01.0cat1.00.10.00.0sat0.21.00.00.0on0.00.30.80.0the0.10.01.00.0mat0.90.00.10.0dog1.00.20.00.0.0.00.00.30.0

embedding of “cat”1 × 4

thingactionsmallfirst1.00.10.00.0

column “thing”

0×0.1 + 1×1.0 + 0×0.2 + 0×0.0 + 0×0.1 + 0×0.9 + 0×1.0 + 0×0.0 = 1.0

toy table8 × 4 = 32 numbers

GPT-2 small50,257 × 768

= numbers38,597,376

Every product is zero except the one where the 1 meets row “cat”, so the answer is exactly that row. The table has shape vocabulary × width. In GPT-2 it is 50,257 rows of 768 numbers, about 39 million parameters, almost a third of the whole 124-million-parameter model. Code skips the multiplication and just reads row 1: E[1].

A toy vocabulary of 8 tokens and an embedding table E with one row per token. Pick a token: a row of zeros with a single 1 (one-hot), times E, picks out that token’s row.

example · GPT-2’s embedding table

GPT-2 small’s embedding table has shape $50{,}257 \times 768$ : 38.6 million numbers, almost a third of the model’s 124 million parameters. Reading “The cat sat on the” takes rows 464, 3797, 3332, 319 and 262 and stacks them into a $5 \times 768$ matrix. That matrix is what the first transformer block receives. Because $E$ is so tall, real code reads rows by index (E[ids]) instead of multiplying by one-hot vectors, but the result is identical.

Shapes everywhere

From here on, almost every object in a transformer is a matrix, and the most common bug is a shape that doesn’t fit. The habit to build now: whenever you see a matrix, say its shape out loud and what its rows and columns mean.

Matrix	Shape in GPT-2 small	Rows are	Columns are
Embedding table $E$	50,257 × 768	tokens in the vocabulary	embedding dimensions
Input $X$ for $T$ tokens	$T$ × 768	positions in the text	embedding dimensions
Query weights $W_Q$ (one head)	768 × 64	input dimensions	query dimensions
Attention weights $A$	$T$ × $T$	the token looking	the token looked at

going deeper

Notation

An $m \times n$ matrix of real numbers belongs to $\mathbb{R}^{m \times n}$ . Row $i$ is written $A_{i,:}$ and column $j$ is $A_{:,j}$ , the same slicing that numpy uses. Maths counts rows and columns from 1; Python counts from 0, so $X_{2,1}$ above is X[1, 0] in code.

Useful facts about the transpose: $(A^\top)^\top = A$ , $(A + B)^\top = A^\top + B^\top$ , and $(AB)^\top = B^\top A^\top$ (the order reverses). A matrix with $A^\top = A$ is symmetric; the cosine-similarity table in the dot product lesson is one.

In code

import numpy as np
X = np.array([[0.1, 0.0, 1.0, 1.0],
              [1.0, 0.1, 0.0, 0.0],
              [0.2, 1.0, 0.0, 0.0],
              [0.0, 0.3, 0.8, 0.0],
              [0.1, 0.0, 1.0, 0.0]])
X.shape      # (5, 4)
X[1]         # row for "cat":  array([1. , 0.1, 0. , 0. ])
X[:, 0]      # column "thing": array([0.1, 1. , 0.2, 0. , 0.1])
X.T.shape    # (4, 5)
 
import torch
E = torch.nn.Embedding(50257, 768)       # the table, E.weight has shape (50257, 768)
E(torch.tensor([464, 3797, 3332, 319, 262])).shape   # torch.Size([5, 768])