Autoregression: one choice becomes the next input
Generation repeatedly predicts, selects, appends, and predicts again.
In this lesson
A response unfolds through a feedback loop in which earlier generated tokens become context for later ones.
The basic loop
Start with a prompt. Compute next-token scores, apply a decoding rule, append the chosen token, and repeat. This is ordinary autoregressive generation. The Transformers generation guide distinguishes how the token is selected; the loop then feeds that selection into the next prediction.
For an invented prompt “A small”, suppose the first step chooses “bird”. The next prediction is conditioned on “A small bird”, not the original two-token prefix. If the first step instead chooses “boat”, later probabilities can change dramatically even though the model weights remain exactly the same.
Sequence probability multiplies conditionals
The probability of a particular continuation is the product of its conditional token probabilities. For two steps, P(a,b | prefix) = P(a | prefix) × P(b | prefix,a). Log probabilities turn that product into a sum, which is easier to calculate for long sequences.
In a toy tree, A has first-step probability 0.6 and B has 0.4. If the best continuation after A has probability 0.5, that two-token path has probability 0.3. If the best continuation after B has probability 0.9, its path has probability 0.36. Greedy first-token choice therefore need not find the most probable full sequence.
The output becomes part of the next input.
- Contextthe
- Append “cat”the cat
- Append “sat”the cat sat
Stopping is part of the system
A generator needs a stopping rule. It may stop after a designated end token, a configured output limit, or another supported condition. The existence of a fluent sentence does not itself tell a basic loop to stop. The application and model configuration determine which rules apply.
A clear educational demo should show when it has stopped and why. If a token limit ends the loop halfway through a sentence, that is different from sampling an end marker. Restarting should restore the initial prefix and seed so the same experiment can be repeated.
Sequential dependence and efficient execution
The basic loop has a dependency: the next selected token is needed before the following ordinary decoding step. Implementations can still reuse previous computations, batch requests, or use more advanced decoding schemes. A key-value cache preserves past attention information rather than recalculating it from scratch. How caching works.
The browser generator is a small transparent illustration of the dependency, not a performance benchmark. Its probabilities come from a fixed hand-authored bigram table, not learned weights or a production model. Only the last token chooses the next table row. What transfers is the loop’s structure: a choice changes the sequence, then another prediction is made.
Local best is not always global best
Path A→x has probability 0.6 × 0.5 = 0.30. Path B→y has probability 0.4 × 0.9 = 0.36. Greedy decoding starts with A because 0.6 > 0.4, even though B→y is the more probable two-token path in this invented tree.
Grow a sequence
An explicit hand-authored bigram table, not a trained language model.
The model writes a complete answer internally, then reveals it word by word.
In ordinary autoregressive decoding, each selected token is used to compute the next step. The exact serving strategy may add optimizations, but the output is not simply a hidden finished paragraph being uncovered.
Take it with you
- Generated tokens become later context.
- Sequence probabilities multiply conditional probabilities.
- Stopping and caching are explicit parts of generation.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Context windows: what is available right now
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Generation strategies — Transformers documentation ↗Hugging Face contributors · Main documentation; accessed 2026-11-13 · checked 13 Nov 2026
- How caching works — Transformers documentation ↗Hugging Face contributors · Main documentation; accessed 2026-11-13 · checked 13 Nov 2026