Softmax: turning scores into a distribution
Exponentiate, normalize, and obtain a set of nonnegative probabilities that add to one.
In this lesson
Softmax makes relative scores comparable; it does not turn a prediction objective into a truth detector.
A precise recipe
For finite scores z₁ through zₙ, softmax assigns pᵢ = exp(zᵢ) / Σⱼ exp(zⱼ). Each result is positive and the total is one. PyTorch’s Softmax documentation states the definition. It is used here to convert next-token logits into a distribution over candidates.
For equal scores [0, 0, 0], each exponential is one, so each probability is one third. For [ln 2, 0], the exponentials are [2, 1], giving probabilities [2/3, 1/3]. These examples make normalization tangible without requiring a large vocabulary.
A stable way to calculate it
Directly exponentiating very large values can overflow. Subtracting the maximum score first preserves the probabilities while keeping the largest exponential at one. Calculate exp(zᵢ − m), where m is the maximum, and normalize those values instead. This follows from cancelling the same positive factor in every term.
With [1000, 999], direct exponentials exceed the range of ordinary floating-point numbers. Shifting to [0, −1] yields a stable result of about [0.7311, 0.2689]. Numerical stability is not a different model; it is a way to compute the same mathematical expression reliably.
Scores become a distribution.
Show the softmax calculation
Subtract the largest score: [0, −1, −2]. Exponentiate: [1, 0.3679, 0.1353]. Divide each number by 1.5032. This gives approximately [0.6652, 0.2447, 0.0900]. Subtracting the same constant from every score leaves softmax unchanged.
Normalization creates competition
Increasing one score changes not just that entry’s probability but the denominator shared by all entries. Suppose three equal candidates each receive one third. Raise only the first score and it takes more of the total probability mass. The other two remain tied but both lose probability.
This also explains why a probability is inseparable from the candidate set. If a filter removes options, the remaining probabilities must be renormalized. Reporting a probability before top-k filtering and then sampling from a filtered distribution without saying so would describe two different experiments as though they were one.
A distribution is not automatically calibrated
A normalized output is mathematically valid even when its confidence is poorly matched to real outcomes. Calibration asks whether probability estimates correspond to observed correctness frequencies under a defined task. Guo and colleagues studied calibration in classification networks; their findings should not be treated as an LLM factuality score. On Calibration of Modern Neural Networks.
For language generation, “the next token in this dataset” and “a factually correct answer” are different targets. A distribution can be useful for predicting text while still supporting false claims. Inspect the mathematical total in the demo, then separately ask what event those probabilities are intended to describe.
A normalized distribution you can compute
Scores [ln 3, ln 2, 0] exponentiate to [3, 2, 1]. Dividing by the total 6 gives [0.5, 0.3333…, 0.1666…]. The displayed rounded values may sum to 0.9999 or 1.0001; the underlying calculation still normalizes to one.
Turn scores into choices
Illustrative logits; the probability and sampling calculations are real.
Probabilities that sum to one must be trustworthy confidence estimates.
Normalization is a mathematical property. Calibration and factual accuracy require separate evaluation against an appropriate target.
Take it with you
- Exponentiation and normalization turn scores into probabilities.
- Subtracting the maximum improves numerical stability.
- A valid distribution can still be poorly calibrated.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Temperature: reshaping the odds
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Softmax — PyTorch 2.15 documentation ↗PyTorch contributors · Version 2.15; accessed 2026-11-13 · checked 13 Nov 2026
- On Calibration of Modern Neural Networks ↗Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger · 2017-08-03 (v2) · checked 13 Nov 2026