The dot product

Key idea

a · b = a₁b₁ + a₂b₂ + … ; equals |a| |b| cos θ; sign tells same or opposite direction; cosine similarity; projection

Real world

A shopping bill. Quantities [2, 1, 3] (bread, milk, apples) dotted with prices [1.20, 0.90, 0.40] gives 2×1.20 + 1×0.90 + 3×0.40 = 4.50 pounds.

In an LLM

To decide where to look, each token’s query is dotted with every earlier token’s key. In “The cat sat on the”, the last “the” has the largest dot product with “cat”, so it pays most attention to “cat”.

Multiply and add

The Dot product of two vectors of the same length multiplies them position by position and adds everything up:

For $\mathbf{a} = [3, 1]$ and $\mathbf{b} = [2, 2]$: $3 \times 2 + 1 \times 2 = 8$. Two lists in, one number out. Both vectors must have the same length, or there is nothing to pair up.

The arithmetic is simple. What makes it useful is what the number means.

What the number measures

The same number can be computed from the arrows’ lengths and the angle $\theta$ between them:

$|\mathbf{a}|$ is the length of the arrow (the norms lesson covers it). $\cos\theta$ is 1 when the arrows point the same way, 0 at a right angle and −1 when they point in opposite directions. So:

• Positive: roughly the same direction (angle under 90°).
• Zero: at right angles. The vectors are called orthogonal: they share nothing.
• Negative: roughly opposite directions.

There is also a picture. Shine a light straight down onto the line through $\mathbf{a}$: $\mathbf{b}$ casts a shadow on it, called its projection. The dot product is the shadow’s length times $\mathbf{a}$’s length, negative if the shadow falls backwards.

Drag both arrows and try the three buttons:

Similarity: cosine

The dot product grows with the lengths of the vectors as well as with their alignment. To measure direction alone, divide by both lengths. This is the Cosine similarity:

It always lies between −1 and 1. Here it is for 48 real GPT-2 word embeddings, each 768 numbers long. Nobody told GPT-2 that cats and dogs are both animals; the similarity comes from training.

Attention scores are dot products

Inside every attention head, each token makes a query vector (what it is looking for) and a key vector (what it contains). The score for “token i looks at token j” is the dot product of i’s query with j’s key. Here is the toy head from the Transformers course:

A real GPT-2 head does this with 64 numbers per query, for every pair of tokens. A 1,000-token text needs a million of these dot products per head, which is why they are organised as one matrix multiplication.

import numpy as np
a, b = np.array([3.0, 1.0]), np.array([2.0, 2.0])
a @ b                                    # 8.0  (same as np.dot(a, b))
cos = a @ b / (np.linalg.norm(a) * np.linalg.norm(b))   # 0.894
 
import torch.nn.functional as F
F.cosine_similarity(x, y, dim=-1)        # for torch tensors