Sampling: choosing the next token
A distribution offers possibilities. A decoding rule turns those possibilities into one continuation.
In this lesson
The model’s scores and the procedure that chooses a token are distinct parts of generation.
One distribution, different decisions
Greedy decoding picks a highest-probability token. Multinomial sampling draws a token according to the distribution. These can produce different outputs from identical model scores. Transformers generation strategies documents both. Sampling is not selecting uniformly unless the probabilities happen to be uniform.
With probabilities [0.6, 0.3, 0.1], divide the unit interval into [0, 0.6), [0.6, 0.9), and [0.9, 1). A uniform random number 0.72 selects the second token. A draw of 0.95 selects the third. Over many independent draws, selection frequencies tend toward the specified probabilities.
Top-k keeps a fixed number
Top-k keeps the k highest-scoring candidates and renormalizes their probabilities. In the same toy distribution, k = 2 removes the third candidate and changes the survivors to [2/3, 1/3]. Setting k = 1 produces a single surviving candidate, assuming a defined tie-break.
A fixed k does not mean fixed uncertainty. If one candidate dominates, keeping five choices still concentrates probability near one. If many candidates are close, keeping five can discard substantial plausible mass. The filter changes the candidate set, while temperature changes relative odds within it.
One draw chooses one interval.
60%Green
30%Red
10%
Draw: 0.72 → green0.72 falls between 0.6 and 0.9
Show the selection calculation
Blue occupies [0, 0.6), green [0.6, 0.9), and red [0.9, 1). Because 0.6 ≤ 0.72 < 0.9, this draw selects green. These interval lengths match the probabilities 60%, 30%, and 10%.
Top-p keeps enough probability mass
Nucleus, or top-p, sampling retains a smallest leading set whose cumulative probability reaches a chosen threshold, then samples within that set. Holtzman and colleagues studied this approach for open-ended text generation. The Curious Case of Neural Text Degeneration. Its candidate count adapts to the shape of the distribution.
For [0.6, 0.3, 0.1] and p = 0.8, keep the first two entries because one reaches only 0.6 while two reach 0.9. The threshold is not “keep tokens individually above 0.8”. Tie handling and filter order should be specified by an implementation, especially when combining settings.
Repeatability and limits
A deterministic pseudo-random seed lets a toy experiment reproduce the same draws for the same algorithm and inputs. It does not make a real service universally deterministic across hardware, implementations, or model revisions. In this site’s browser demo, seeds are an educational control for comparing changes fairly.
A single generated sentence is a sample, not a complete picture of the distribution. Compare repeated draws and inspect the probabilities. Also distinguish the hand-authored candidate scores used in this demo from scores produced by a trained neural model. The sampling arithmetic is real even though the miniature probability source is illustrative.
Filtering requires renormalization
Start with A: 0.6, B: 0.3, C: 0.1. Top-k with k = 2 removes C. The remaining mass is 0.9, so A becomes 0.6/0.9 = 2/3 and B becomes 1/3. Sampling from unnormalized [0.6, 0.3] as though it totaled one would be incorrect.
Turn scores into choices
Illustrative logits; the probability and sampling calculations are real.
The model always outputs the most likely next token.
That is greedy decoding. A sampling rule can choose another token according to a possibly filtered distribution.
Take it with you
- Decoding chooses from model-produced scores.
- Top-k and top-p filter candidates in different ways.
- Seeds help compare toy experiments but do not prove universal determinism.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Autoregression: one choice becomes the next input
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Generation strategies — Transformers documentation ↗Hugging Face contributors · Main documentation; accessed 2026-11-13 · checked 13 Nov 2026
- The Curious Case of Neural Text Degeneration ↗Ari Holtzman et al. · 2020-02-14 (v2) · checked 13 Nov 2026