Pretraining: learning from continuations
Large collections of examples teach a model reusable patterns before any particular assistant conversation.
In this lesson
Pretraining fits a predictive model to a chosen data distribution; it is neither an exact archive nor a truth filter.
The text supplies the target
In autoregressive language-model pretraining, a sequence can supervise itself: earlier tokens are input and the next token is the target. A separate person need not label each word. The term self-supervised describes where the targets come from, not the absence of human decisions about data or design. The GPT-3 paper documents one influential example of this approach.
A corpus might contain prose, code, mathematical notation, and dialogue. Predicting their continuations rewards learning structures that help across many contexts. A closing parenthesis depends on syntax; a plausible paragraph depends on topic and style. The same objective can press the network to represent several kinds of regularity.
What one training batch does
Data preparation selects and tokenizes examples. A batch of token sequences enters the model. Causal predictions are compared with shifted targets, losses are aggregated, gradients are computed, and an optimizer changes the weights. Repeating this loop is the core learning process. Data order, mixing, masking, and update settings all affect what is learned.
The final checkpoint is an array of learned parameters plus the information needed to use it correctly. It is not a folder containing every training document. Nevertheless, a model can sometimes reproduce specific training passages, so “not an archive” must not be confused with “incapable of memorizing.”
Prediction becomes a training signal.
- Forward passPredict
- Observed targetMeasure loss
- Backward passFind gradients
- OptimizerUpdate weights
Pretraining is a phase, not a universal recipe
Autoregressive prediction is the focus of this guide. Other language-model objectives include predicting masked text or reconstructing corrupted input, and systems can combine objectives or modalities. Calling something a pretrained model does not fully specify its architecture, objective, training mix, or later adaptation.
After initial pretraining, developers may continue training on a narrower domain, longer sequences, or different data. Instruction tuning and preference methods are additional choices, not magical conversions of a text model into a complete product. Separate the learned checkpoint from retrieval, tool execution, and the interface wrapped around it.
Read the evidence at the right level
If a model produces a competent explanation, that is evidence about its behavior on that prompt. It does not by itself reveal which documents caused the behavior or prove that the model stores the explanation as a discrete fact. Different parameter configurations can support similar outputs.
For an original thought experiment, train only on invented weather reports where rain is always followed by “bring a coat.” The model may learn that continuation very well. A low held-out loss on the same artificial world does not establish real meteorological expertise. Data distribution and evaluation setting determine what the result means.
One sequence, several targets
For the illustrative token sequence [the, sky, is, blue], training can score sky after the, is after the sky, and blue after the sky is. These targets come from the same sequence, while the causal mask prevents seeing each target early.
Pretraining means reading the internet and checking which claims are true.
A predictive objective rewards matching selected training data. Factual verification is not automatically part of that objective.
Take it with you
- Targets can be obtained directly from text.
- Data preparation is part of the learning system.
- A pretrained model and a deployed assistant are different objects.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Scaling laws: trends, not guarantees
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Language Models are Few-Shot Learners ↗Tom B. Brown et al. · 2020-05-28 · checked 13 Nov 2026