Backpropagation: tracing responsibility
The chain rule carries a loss signal backward through the computations that produced it.
In this lesson
Backpropagation computes gradients efficiently; the optimizer decides how to use them.
A record of the forward computation
During a forward pass, a network turns inputs into intermediate values and eventually a loss. Think of those calculations as a graph: an arrow records that one value depends on another. Backpropagation travels this graph in reverse, combining local derivatives to find how each parameter affects the loss. PyTorch’s automatic differentiation guide illustrates this graph-based mechanism.
The word “backward” refers to differentiation through computations. It does not mean that text is read backward, that the model undoes its answer, or that a second model criticizes the first. The same numerical functions used in prediction provide the derivatives needed for learning.
Follow a two-step example
For an original scalar example, let z = wx and L = (z − y)². Choose x = 2, y = 6, and w = 1. The forward pass gives z = 2 and L = 16. The derivative of L with respect to z is 2(z − y) = −8; the derivative of z with respect to w is x = 2.
The chain rule multiplies these local sensitivities: dL/dw = −8 × 2 = −16. If the optimizer uses a learning rate of 0.01, it moves w to 1.16. A new forward pass then predicts 2.32 and gives loss 13.5424. The gradient and the update are separate operations, even when software runs them consecutively.
Shared paths add together
A parameter may contribute to many positions or branches. Each path supplies part of its total derivative, and those contributions add. Sharing a weight across token positions therefore does not give it a different value at every position; it gives one shared value a combined training signal.
Reverse-mode differentiation is useful when a scalar loss depends on many parameters. Rather than perturbing every weight separately and repeating the whole network, it reuses the structure of the original computation. Intermediate values often need to be retained or recomputed, which helps explain why training has memory costs beyond storing weights.
What it can and cannot explain
A derivative answers a precise local question: how would this numerical objective change under a tiny parameter change at this setting? It does not automatically identify a human-readable concept in a neuron or provide a complete causal account of an answer. Interpretation needs different experiments and a clearly stated question.
Automatic differentiation also depends on the graph actually containing the intended operations. Detached values, omitted loss terms, or incorrect masks can produce gradients for the wrong problem. A useful small-model check compares an analytic gradient with a finite difference: evaluate the loss at w + ε and w − ε, then divide their difference by 2ε.
Numerical cross-check
For L = (2w − 6)² at w = 1, the central difference [L(1 + ε) − L(1 − ε)] / (2ε) approaches −16. Agreement checks the derivative; it does not prove that the chosen loss is appropriate.
Backpropagation and gradient descent are the same thing.
Backpropagation computes derivatives. Gradient descent is one rule for changing parameters using those derivatives.
Take it with you
- The forward pass supplies values; the backward pass supplies derivatives.
- Multiple dependency paths contribute to a shared gradient.
- The optimizer performs the actual update.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Pretraining: learning from continuations
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Automatic differentiation with torch.autograd ↗PyTorch contributors · Living documentation; checked 2026-11-13 · checked 13 Nov 2026