Neural networks: composing simple calculations
Large models combine many ordinary numerical operations into a trainable function.
In this lesson
Complex behavior can come from composing layers; no individual multiplication needs to understand a sentence.
Start with one small calculation
A basic unit can multiply inputs by weights, add the results, and apply a nonlinear function. A layer performs many such calculations together. The arithmetic is ordinary; learning changes the coefficients so that the complete network becomes useful for its objective. Karpathy’s Zero to Hero course develops this construction from small examples.
For a toy unit, let s = 2x₁ − x₂ + 0.5. With input [1, 3], s = −0.5. A rectified linear function, ReLU, outputs max(0, s), giving zero. Change the input to [2, 1] and the same unit outputs 3.5. Its response depends on combinations of features.
Why nonlinearity matters
Stacking only affine transformations still gives an affine transformation. Nonlinear operations allow the composition to represent different behaviors in different regions of the input space. This mathematical distinction is more useful than imagining each unit as a miniature human neuron.
Our toy ReLU has a threshold: below zero it outputs zero; above zero it passes the score through. Two units can respond to different regions, and another layer can combine those responses. Real networks use many activation choices, so this is an accessible example of the principle rather than a universal architecture diagram.
Forward computation and learning
The forward pass applies the current parameters to an input and produces an output. A training step then compares output with a target, calculates gradients, and updates selected parameters. The forward pass is therefore part of training, but it can also run by itself during inference.
Imagine predicting the next token from a short prefix. The network produces scores, a loss measures how much probability went to the observed next token, and the update slightly adjusts future predictions. One training example usually affects many parameters. There is no requirement that a human assign a meaning to each intermediate unit.
From a small network to a transformer
A transformer is a particular arrangement of neural-network operations. It includes mechanisms for moving information between token positions and mechanisms for transforming features at each position. The resulting function remains numerical, even when its output is fluent language.
When reading a diagram, separate three questions: what is the shape of the data, what operation transforms it, and which numbers are learned? A linear layer’s documented input and output shapes provide a concrete example. Following shapes often resolves more confusion than attaching human traits to a box labeled “AI”.
A network without a hidden story
For x = [1, 3], w = [2, −1], and b = 0.5, the weighted sum is −0.5 and ReLU gives 0. For x = [2, 1], the sum is 3.5 and ReLU gives 3.5. The weights are unchanged; only the input differs.
Each artificial neuron represents one identifiable idea.
Useful information can be distributed across many units and directions. A unit’s interpretation depends on the surrounding computation.
Take it with you
- Layers compose numerical functions.
- Nonlinearities make the composition more expressive.
- A forward pass does not itself imply a parameter update.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Transformers: a repeated pattern of computation
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Neural Networks: Zero to Hero ↗Andrej Karpathy · Living course; accessed 2026-11-13 · checked 13 Nov 2026
- Linear — PyTorch 2.15 documentation ↗PyTorch contributors · Version 2.15; accessed 2026-11-13 · checked 13 Nov 2026