Next-token prediction: a small objective with broad demands
The model assigns probabilities to what could follow a prefix. Training rewards probability on the observed continuation.
In this lesson
Predicting text can reward useful representations, but the objective is not a guarantee of truth or understanding.
The prediction target
Given a prefix x₁ … xₜ, a causal language model estimates a distribution P(xₜ₊₁ | x₁ … xₜ). A next-token training example supplies the actual following token as its target. GPT-3 is a historical example of an autoregressive language model trained and evaluated in this setting. Brown and colleagues, 2020.
Use an invented sequence “the kite is blue”. The prefix “the kite is” makes “blue” the observed target in this example. “red” could also form a sensible sentence, but it is not the token observed at this position. The objective concerns the dataset’s continuation, not every possible correct answer to a question.
Many predictions from one sequence
A sequence can provide several shifted targets. The input tokens “the kite is” align with targets “kite is blue”. With causal masking, each input position uses only the prefix available to it, even when the positions are processed together. This makes training efficient without allowing future-token leakage.
The loss can average the negative log probability of the observed targets. If a target receives probability 0.5, its loss is about 0.693 nats; if it receives 0.1, the loss is about 2.303 nats. The learner receives a stronger penalty for assigning very little probability to what actually occurred.
Each position predicts what follows it.
Why prediction can require structure
To predict a continuation well, a model may benefit from representing grammar, topic, relationships, and patterns that extend beyond neighboring words. A bracket-closing task requires tracking earlier brackets; a story continuation can depend on a character introduced earlier. These examples explain why the objective can reward more than local word frequency.
This is an argument about what could help prediction, not proof that a particular model has learned a specific internal algorithm. Capability must be measured, and internal explanations need evidence. Fluent output alone does not tell us which representations or computations produced it.
One objective, several limitations
Text includes fiction, uncertainty, contradictions, and errors. A model can learn plausible continuations from all of them. The next-token objective does not automatically supply a fact checker, a source citation, or a rule for abstaining when evidence is missing. Those behaviors require suitable learning and system design.
The sequence generator on this site isolates the loop with a hand-authored bigram table: only the last token selects the next distribution. Its table is small enough to inspect. Follow a selected token into the next step and notice how the condition changes. It is not a trained transformer.
Inputs and targets are shifted
For the toy token sequence [the, kite, is, blue], the input positions [the, kite, is] have targets [kite, is, blue]. At the “kite” input position, only “the kite” is available; the correct training target is “is”.
Grow a sequence
An explicit hand-authored bigram table, not a trained language model.
Next-token prediction is the same as choosing a factually true answer.
It predicts continuations under a learned distribution. Truthfulness is a separate property requiring evidence and evaluation.
Take it with you
- A prefix produces a distribution over the following token.
- Training targets are shifted relative to the inputs.
- A useful prediction objective does not guarantee factual reliability.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Logits: scores before probabilities
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Language Models are Few-Shot Learners ↗Tom B. Brown et al., OpenAI · 2020-07-22 (v4) · checked 13 Nov 2026