Parameters: the numbers learning changes
Weights shape a model’s computation. They are different from the text you put in a prompt.
In this lesson
Parameters persist across requests; activations are computed for a particular input.
A model contains adjustable numbers
A parameter is a value that a training procedure can adjust. In a simple linear layer, weights multiply input features and an optional bias shifts the result. PyTorch’s Linear documentation gives the operation as an affine transformation. Embedding tables and many other network components also contain parameters.
Consider a toy score s = 2x − 1. The numbers 2 and −1 are parameters; x is the input. For x = 3, the score is 5. Replacing x with 4 changes the score to 7 without changing either parameter. This is the central distinction between applying a model and updating it.
Weights are not the supplied context
A prompt provides input to a computation. The hidden vectors calculated from that prompt are activations. In ordinary inference, those activations change as the input changes while the weights stay fixed. Training uses an additional procedure to calculate and apply parameter updates.
If you tell a model that a fictional planet has two moons, it may use that statement in the current response. That does not establish that the statement has become a permanent weight change. A product could save memories or later train on permitted data, but those are separate mechanisms with their own behavior.
Using weights and changing them are different.
During ordinary generation
- New context
- Same saved weights
- New prediction
During training
- Examples + loss
- Optimizer update
- Changed weights
Count does not equal knowledge
A layer mapping three input features to two outputs has six weights. Adding one bias per output brings its total to eight parameters. This count describes numerical capacity and storage needs; it does not tell us how many facts the layer contains. Facts are not generally filed one per weight.
A parameter count also omits important differences: training data, training duration, architecture, numerical precision, and evaluation setting. A hypothetical model with more parameters can still perform worse on a task if its training did not prepare it well. Compare measured behavior, not a single size label.
An architecture is a recipe, a checkpoint a state
The architecture specifies which calculations exist and how they connect. A checkpoint stores a particular learned state, usually including the parameter values and sometimes training state. Two checkpoints can share the same architecture but behave differently because their numbers were learned from different examples or objectives.
For an inspectable historical implementation, Karpathy’s nanoGPT separates model structure from the training loop. It is linked here as a compact GPT example, not a recommendation that an older recipe describes every current model. The enduring question is: which values are fixed for this request, and which are being computed?
Eight parameters, changing activations
A 3-to-2 linear layer has 3 × 2 weights and 2 biases. Feed it two different three-number inputs: both outputs can change while all eight learned values remain identical. A training update is a separate operation on those eight values.
Every message permanently teaches the model new weights.
Ordinary inference changes activations and context, not weights. Storage and later training are separate product-level processes.
Take it with you
- Parameters are adjustable values learned during training.
- Activations depend on the current input.
- Parameter counts do not directly measure facts or competence.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Neural networks: composing simple calculations
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Linear — PyTorch 2.15 documentation ↗PyTorch contributors · Version 2.15; accessed 2026-11-13 · checked 13 Nov 2026
- nanoGPT: historical GPT implementation ↗Andrej Karpathy and contributors · Historical implementation; repository accessed 2026-11-13 · checked 13 Nov 2026