Vectors

A vector is a list of numbers, and the same thing as an arrow. You can add vectors and stretch them, and both have a simple picture. Inside a language model every token is a vector of hundreds or thousands of numbers.

Key idea: A vector is an ordered list of numbers; drawn as an arrow; added number by number (tip to tail); scaled by multiplying every number
Real world: Directions to a café: “3 blocks east, 2 blocks north” is the vector [3, 2]. Two sets of directions in a row add up: [3, 2] + [1, −1] = [4, 1].
In an LLM: GPT-2 stores the word “cat” as a list of 768 numbers, its embedding. Every token the model reads becomes such a vector, and everything the model does afterwards is arithmetic on these vectors.

Two pictures of one thing

A is an ordered list of numbers, such as $[2, 1]$ . The order matters: $[2, 1]$ and $[1, 2]$ are different vectors. Each number is called a component or coordinate, and how many numbers there are is the vector’s dimension. $[2, 1]$ is a 2-dimensional vector.

A 2-dimensional vector can also be drawn: an arrow that starts at the origin (the point $[0, 0]$ ), goes 2 units right and 1 unit up. The list and the arrow are the same object. The list is how a computer stores it; the arrow is how you picture it.

Two operations make vectors useful:

Adding. Add the numbers position by position: $[2, 1] + [-1, 2] = [1, 3]$ . As arrows: put the second arrow’s tail on the first arrow’s tip; the sum goes from the origin to where you end up.
Scaling. Multiply every number by the same number, called a scalar: $2 \times [2, 1] = [4, 2]$ . The arrow keeps its direction and gets twice as long. A negative scalar flips it round.

Try it

Drag the arrow and watch its numbers change. Then switch to “add two” and press “tip to tail”, and to “scale one” and move the slider below zero.

a = [2.0, 1.0]

The arrow a goes 2.0 right and 1.0 up, so its list of numbers is [2.0, 1.0]. The arrow and the list are the same vector, drawn two ways. A word embedding is the same idea with 768 numbers instead of 2.

Drag the dots. A vector is an arrow from the origin, and the same thing as its list of coordinates. The grid squares are 1 unit.

example · directions to a café

From your door the café is 3 blocks east and 2 blocks north: the vector $[3, 2]$ . The bakery is 1 block east and 1 south of the café, $[1, -1]$ . To go from your door to the bakery via the café, add: $[3, 2] + [1, -1] = [4, 1]$ . Walking the whole route twice as far would be $2 \times [4, 1] = [8, 2]$ . Nothing about this needs a map; the numbers do the work.

Many more than two numbers

Nothing in the rules above depends on there being 2 numbers. A vector with 3 numbers is an arrow in 3D space. A vector with 768 numbers is an arrow in a 768-dimensional space. Nobody can picture that, but adding and scaling work exactly the same way, one position at a time, and every picture in this course built in 2D carries over.

A language model turns each into a vector called its embedding. In GPT-2 small every embedding has 768 numbers; in larger models it is several thousand. Here are real ones:

Loading GPT-2 data…

example · the word “cat” in GPT-2

GPT-2 small reads “ cat” as token number 3797 and looks up row 3797 of its embedding table: 768 numbers, the first few of which are shown above. The numbers were learned during training, not designed. Words used in similar ways end up with vectors pointing in similar directions, which the next lesson measures. From then on the model only adds, scales and multiplies such vectors: the residual stream is one vector per token that every layer adds its result to.

Why lists of numbers

Text is made of symbols, and you can’t add “cat” to “dog”. Turning every token into a vector gives the model something it can compute with. Two consequences follow:

Meaning becomes direction. Similar words get similar vectors, so the model can treat “cat” and “dog” alike without being told they are both animals.
Combining becomes adding. When a layer wants to add information to a token (“this word is the subject”, “the previous word was ‘the’”), it adds a vector to it.

Where a word’s embedding comes from is covered in the Transformers course: embeddings.

going deeper

Notation

A vector is usually written in bold or with an arrow, $\mathbf{a}$ or $\vec a$ , and its components with subscripts: $\mathbf{a} = [a_1, a_2, \dots, a_n]$ . The set of all vectors with $n$ real numbers is written $\mathbb{R}^n$ , so a GPT-2 embedding lives in $\mathbb{R}^{768}$ .

\mathbf{a} + \mathbf{b} = [a_1 + b_1,\ \dots,\ a_n + b_n], \qquad c\,\mathbf{a} = [c\,a_1,\ \dots,\ c\,a_n]

Maths books usually stand vectors up as columns; machine learning code usually lays them down as rows (one row per token). The numbers are the same.

In code

import numpy as np
a = np.array([2.0, 1.0])
b = np.array([-1.0, 2.0])
a + b        # array([1., 3.])
2 * a        # array([4., 2.])
a.shape      # (2,)
 
# A real embedding
from transformers import GPT2Model, GPT2TokenizerFast
tok = GPT2TokenizerFast.from_pretrained("gpt2")
wte = GPT2Model.from_pretrained("gpt2").wte.weight   # (50257, 768)
cat = wte[tok.encode(" cat")[0]]                      # a vector of 768 numbers