Gradient descent

How every neural network learns. Measure which way is downhill on the loss, take a small step that way, repeat.

Key idea: Slope, gradient, learning rate, and stochastic gradients
Real world: Walking down a hill in thick fog. You can't see the valley, but you can feel which way the ground slopes under your feet. Step that way, feel again, repeat.
In an LLM: Every training step computes, for each of billions of weights, which direction would lower the loss, and moves every weight a tiny step that way.

The slope

The slope (or derivative) of a function at a point says how fast it changes as you move. Positive slope: going right goes uphill. Negative: going right goes downhill. Zero: you're at a flat spot, often the bottom.

With many inputs, the gradient is the list of slopes, one per input. It points in the direction of steepest increase.

To lower a loss, move against the gradient:

\text{new weight} = \text{old weight} - \text{learning rate} \times \text{slope}

The learning rate sets the step size. Too small and learning is slow; too large and each step overshoots.

Try it

learning rate0.30noisy slope (mini-batch)

step0

x-1.000

loss8.000

slope-4.000

A good learning rate: big steps while the slope is steep, smaller ones as it flattens near the minimum.

The loss is lowest at x = 3. Each step moves x against the slope (the orange tangent), scaled by the learning rate.

example · walking in fog

You feel the ground dropping steeply to your left, so you take a stride left. Near the bottom of the valley the slope is gentle, so the same rule gives you smaller strides. That's why gradient descent naturally slows down near a minimum. But if your strides are huge, you leap over the valley floor and land on the opposite slope, possibly higher than where you started.

example · one weight in a model

A weight is 0.50. The gradient of the loss with respect to it is +2.0: increasing it would increase the loss. With learning rate 0.01:

0.50 - 0.01 \times 2.0 = 0.48

The same update happens to every weight at once, each with its own gradient, billions of times over a training run.

Stochastic gradients

The true loss averages over all the training data, which is too expensive to compute every step. Instead each step uses a small random batch and computes the gradient on that. It's a noisy estimate, an expected value from samples, but it points roughly the right way, and it's thousands of times cheaper. Tick noisy slope in the widget: the path wobbles around the minimum instead of settling, which is why training usually lowers the learning rate towards the end.

Practical optimisers such as Adam add momentum and per-weight step sizes on top, but the core is this rule.

Gradient ascent in RL

RL wants to maximise expected reward rather than minimise a loss. That's the same thing with the sign flipped: step with the gradient of the reward, or equivalently descend on minus the reward. Code usually writes it as a loss, loss = -(advantage * logprob), so the usual minimiser does the right thing.

Where this shows up

The policy-gradient update is gradient ascent on expected reward: policy gradient.
PPO's clip works by making the gradient zero in certain regions: PPO.
A learning rate that's too high is a common cause of RL training collapsing: training an LLM with RL.

going deeper

Definitions

\theta \leftarrow \theta - \eta\, \nabla_\theta L(\theta) \qquad \nabla_\theta L = \Big(\frac{\partial L}{\partial \theta_1}, \dots, \frac{\partial L}{\partial \theta_n}\Big)

For the widget's loss $L(x) = \tfrac12 (x - 3)^2$ , the slope is $x - 3$ , so each step is $x \leftarrow x - \eta (x - 3)$ . The distance to 3 is multiplied by $(1 - \eta)$ each step: it shrinks for $0 < \eta < 2$ , lands exactly at $\eta = 1$ , and grows for $\eta > 2$ .

Neural networks compute the gradient with backpropagation, which applies the chain rule of calculus layer by layer.

Gradient descent

The slope