Greedy vs sampling
A probability distribution does not dictate a single output. Decoding decides how the system turns that distribution into a sequence.
| Question | Greedy selection | Sampling |
|---|---|---|
| Choice | Select a highest-scoring next token | Draw a token using probabilities |
| Repeatability | Repeatable with fixed scores and tie handling | Repeatable in a deterministic toy with a fixed seed |
| Temperature | Positive scaling does not change the highest score | Changes how concentrated the distribution is |
| Diversity | One local choice per step | Can produce different continuations |
| Truthfulness | Not guaranteed | Not guaranteed |
If “blue” has probability 0.6 and “green” has 0.4, greedy always selects blue under fixed scores. Sampling can select either; neither number is a probability that a factual claim is true.
Follow the evidence
Read the connected lessons for detailed mechanisms, assumptions, and primary references.
Logits: scores before probabilities
The network ends a prediction step with one real-valued score per vocabulary entry.
Softmax: turning scores into a distribution
Exponentiate, normalize, and obtain a set of nonnegative probabilities that add to one.
Temperature: reshaping the odds
A sampling setting makes an existing distribution sharper or flatter without teaching the model anything new.
Sampling: choosing the next token
A distribution offers possibilities. A decoding rule turns those possibilities into one continuation.