Temperature: reshaping the odds
A sampling setting makes an existing distribution sharper or flatter without teaching the model anything new.
In this lesson
Temperature changes relative sampling odds, not the facts available to the model.
Scale scores before softmax
For a positive temperature T, divide each logit by T before applying softmax. Lower T magnifies score differences; higher T reduces them. The generation configuration in Transformers documentation exposes temperature as a sampling control. Its exact product availability should be checked for a particular model or service.
Consider two logits [2, 0]. At T = 1 their probabilities are about [0.881, 0.119]. At T = 2 they become about [0.731, 0.269]. At T = 0.5 they become about [0.982, 0.018]. The order stays the same while the chance of selecting the less-favored token changes.
The boundary at zero
The formula z/T is undefined at T = 0. Interfaces that offer zero temperature commonly implement a separate greedy choice: select a highest-scoring token. Our explanation treats greedy selection as that separate rule, with a stated tie-break, rather than pretending division by zero is a valid calculation.
As positive T approaches zero, probability concentrates on the highest score. If several scores tie for the maximum, the mathematical limit shares mass among those ties. A greedy implementation may choose one using a deterministic ordering. These two details matter when testing edge cases.
Same scores. A different spread.
Lower · T = 0.5
Higher · T = 2
Show what temperature changes
Compute softmax(logits / T). At T = 0.5, [2, 1, 0] becomes [4, 2, 0]. At T = 2, it becomes [1, 0.5, 0]. The model’s original logits stay fixed; temperature acts on them after the model computation.
What temperature leaves unchanged
With positive temperature, candidate ranking is unchanged before other filters: dividing by a positive number preserves order. The prompt, vocabulary, raw logits, and weights are also unchanged in this demo. Only the distribution used to draw the next token is reshaped.
If every raw score is equal, temperature has no effect: the distribution stays uniform. If top-k retains only one candidate, changing a positive temperature also cannot change the selected candidate. Try both situations mentally before concluding that every slider movement must visibly alter the output text.
Diversity is not the same as correctness
A flatter distribution can admit less-common continuations, but calling this a universal “creativity” control oversimplifies the result. A less-common token might be an interesting choice, a grammatical error, or a factual error. Which outcome matters depends on the task and the model’s learned distribution.
Likewise, greedy generation can confidently repeat an incorrect continuation. Lowering temperature is not fact checking. When comparing settings, hold the prompt and model fixed, run multiple draws where relevant, and evaluate the property you actually care about—such as valid formatting, variety, or supported factual claims.
A logit gap changes its effect
For two tokens separated by a logit gap of 2, their odds ratio is exp(2/T). It is about 7.39 at T = 1, 2.72 at T = 2, and 54.60 at T = 0.5. These ratios describe relative sampling chances, not factual confidence.
Turn scores into choices
Illustrative logits; the probability and sampling calculations are real.
Low temperature makes the model factual; high temperature makes it intelligent.
Temperature reshapes a sampling distribution. It cannot supply missing evidence or guarantee correctness.
Take it with you
- Positive temperature rescales logits before normalization.
- Zero-temperature behavior needs a separate defined rule.
- Sampling diversity and factual reliability are different properties.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Sampling: choosing the next token
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Generation configuration — Transformers documentation ↗Hugging Face contributors · Main documentation; accessed 2026-11-13 · checked 13 Nov 2026