Gradient descent: changing the weights
A gradient points uphill in loss. An optimizer uses it to take a carefully sized step in the other direction.
In this lesson
A training update changes parameters; a normal generation step uses the parameters already learned.
A small change with a direction
Imagine a single adjustable number, w, controlling a prediction. The gradient tells us how the loss would change for a very small increase in w, with everything else fixed. A positive derivative means an increase raises the local loss; a negative derivative means it lowers it. This is local information, not a map of every possible setting.
The simplest update is w_new = w − η × gradient, where η is the learning rate. For many parameters, the gradient contains one derivative per parameter. Practical training commonly uses more elaborate optimizers that keep statistics of past gradients. The optimization textbook chapter distinguishes these methods from basic stochastic gradient descent.
A worked step
Consider the original toy objective L = (w − 3)². At w = 1, the derivative is 2(w − 3) = −4. With learning rate 0.1, the next value is 1 − 0.1(−4) = 1.4. Loss falls from 4 to 2.56. No guessing is involved: both the direction and resulting loss can be computed.
Now try learning rate 1. The same initial step lands at 5, equally far from the minimum on the opposite side. Repeating it bounces between 1 and 5. An even larger step can diverge. A direction that is useful locally does not guarantee that an arbitrary distance along it is useful.
One small step changes a weight.
- Starting weight2.0
- Subtract rate × gradient0.1 × 3.0
- Updated weight1.7
Show the update calculation
w_new = w_old − learning_rate × gradient = 2.0 − 0.1 × 3.0 = 1.7. A real training step obtains the gradient from the loss calculation rather than choosing it by hand.
Why real learning looks noisy
A large training run usually estimates a gradient from a batch rather than the entire corpus at once. Different batches contain different examples, so successive updates need not reduce every example’s loss. The useful question is whether the overall training process improves the intended objective and held-out performance.
A learning-rate schedule changes the step scale over time. Gradient clipping limits the magnitude or norm of gradients supplied to the optimizer; it does not directly impose the same bound on every optimizer’s eventual parameter update. It also does not mean clipping text or probabilities. These are controls on numerical optimization. They cannot repair a training objective that rewards the wrong behavior or a corpus missing the information a task needs.
Keep the two clocks separate
Training alternates predictions, loss computation, gradient calculation, and weight updates. Inference ordinarily performs predictions and token selection without this update loop. A response can adapt to a newly supplied instruction because the context changed, even though every parameter remains the same.
The local training demonstration deliberately has very few adjustable numbers. It makes the update inspectable and testable. Its smooth curve is not a forecast of a production run, and one successful step does not imply a neural network has found a global optimum. Use it to understand the operation, then carry that distinction into larger systems.
Try a sign check
A gradient of +2 with learning rate 0.05 subtracts 0.1 from the weight. A gradient of −2 adds 0.1. The minus sign in the update is what turns uphill sensitivity into a downhill step.
Make one weight learn
A trained scalar logistic model; much smaller than a language model.
The model learns new weights whenever you type a message.
Ordinary inference conditions on your message while weights stay fixed. Weight training is a separate process.
Take it with you
- A gradient gives local sensitivity.
- Learning rate controls update size.
- Improving a batch objective is not proof of generalization.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Backpropagation: tracing responsibility
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Deep Learning, chapter 8: Optimization ↗Ian Goodfellow, Yoshua Bengio and Aaron Courville · 2016 · checked 13 Nov 2026