Logits: scores before probabilities
The network ends a prediction step with one real-valued score per vocabulary entry.
In this lesson
A logit is an unnormalized score. Only relative differences matter after softmax.
The output projection
After the transformer processes a prefix, an output projection maps the final representation to vocabulary-sized scores. These are logits. The matrix can be distinct from the input embedding table or share weights with it; weight tying is a studied architectural choice. Press and Wolf, 2017.
For a fictional three-token vocabulary, imagine the scores [2, 1, 0]. The first token ranks highest, but the values do not yet sum to one and are not percentages. Negative values would also be valid. A score of −3 is not a negative probability; it is simply a smaller score than −1.
Differences carry the meaning
Softmax converts scores zᵢ to exp(zᵢ) divided by the sum of all exponentials. Adding the same constant to every score cancels from numerator and denominator. Therefore [2, 1, 0] and [102, 101, 100] produce the same probabilities. The softmax definition supplies the operation; the shift result follows algebraically.
At temperature one, a logit gap of one gives an odds ratio of e, about 2.718, between the two corresponding tokens before additional filtering. This is a ratio, not a fixed probability. Other vocabulary entries still affect the denominator and therefore each token’s individual probability.
Scores become a distribution.
Show the softmax calculation
Subtract the largest logit for numerical stability: [2, 1, 0] − 2 = [0, −1, −2]. Exponentiate to approximately [1, 0.3679, 0.1353], then divide each value by their sum, approximately 1.5032.
Scores respond to the whole computation
A logit is not a standalone fact stored in the model. It is calculated from the current input and learned parameters. Change the prompt, include a retrieved passage, or append a generated token, and the next set of logits can change without any training update.
Imagine “Please write a color:” versus “Please write a number:”. The same vocabulary remains available, but useful scores should differ. The output layer need not gain new rows for the task to change; earlier computations can produce a different representation that projects differently onto existing rows.
Do not confuse prediction with truth
A high score for a token means the model favors that continuation under its learned behavior and supplied context. It does not establish that a statement containing the token is correct. A fictional story can strongly favor a made-up name. An inaccurate premise can strongly favor an inaccurate continuation.
Use the sampling demo to separate stages: inspect the raw logits, convert them to probabilities, then select a token. If temperature changes, the underlying raw logits remain the same in this illustration while the sampling distribution changes. Naming each stage prevents “the model’s confidence” from becoming an ambiguous label.
Same distribution, shifted scores
Softmax([2, 1, 0]) is approximately [0.6652, 0.2447, 0.0900]. Softmax([12, 11, 10]) is identical because the common exp(10) factor cancels. The values are illustrative; they are not measurements from a production model.
Turn scores into choices
Illustrative logits; the probability and sampling calculations are real.
A logit of 5 means a 50% chance.
Logits have no direct percentage interpretation. Probabilities depend on all competing scores and the conversion rule.
Take it with you
- There is one output score per vocabulary entry.
- A shared score offset leaves softmax unchanged.
- High continuation probability is not proof of factual truth.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Softmax: turning scores into a distribution
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Using the Output Embedding to Improve Language Models ↗Ofir Press, Lior Wolf · 2017-02-21 (v3) · checked 13 Nov 2026
- Softmax — PyTorch 2.15 documentation ↗PyTorch contributors · Version 2.15; accessed 2026-11-13 · checked 13 Nov 2026