Generalization: beyond the examples seen
A learned pattern is useful when it transfers to inputs that were not simply repeated from training.
In this lesson
Generalization is measured on a specified distribution; success on one kind of novelty does not imply success on every kind.
From fitting to transferring
A model can fit training examples extremely well without behaving well on new ones. Generalization concerns that transfer. An unseen example may be very similar to training data, or it may differ in topic, language, structure, or required reasoning. Those are different tests, even when all are called “new.”
The distinction between minimizing a training objective and achieving good performance on unseen data is fundamental in Deep Learning’s discussion of optimization. The training algorithm only sees its chosen data and objective. Evaluation supplies evidence about whether the learned behavior is useful beyond those observations.
An original pattern test
Suppose a toy model sees lists of two colors and learns to return the last one. Testing on new color pairs checks whether it learned the position rule rather than memorized the exact pairs. Testing on a list of fifty colors introduces a different challenge: the same abstract rule now appears at an unfamiliar length.
A good test deliberately separates competing explanations. If every training example with “blue” also has “blue” last, the model could succeed by copying blue rather than selecting the final item. Design examples where those shortcuts disagree. This is as important in a classroom demonstration as in a benchmark.
Distribution shift changes the question
A model trained mostly on one writing style may struggle when the same task appears in another style. A factual question can change meaning as the world changes. A tool interface can introduce errors that were absent in a text-only test. Generalization is therefore not a single permanent property attached to a parameter count.
The T0 study evaluated transfer to held-out tasks after multitask prompted training. It illustrates how an experiment can define a particular kind of novelty. Reading the definition matters more than assuming that the phrase zero-shot covers every situation a user might encounter.
Avoid the false choice
Memorization and generalization are not mutually exclusive descriptions of an entire model. The same checkpoint can reproduce a familiar phrase, apply a learned grammatical pattern to new text, and fail a novel combination. Ask which behavior was measured and what evidence rules out a shortcut.
Aggregate improvements can also conceal uneven task changes. Research on predictability and surprise in large models discusses why broad loss trends do not fully specify individual capabilities. The practical response is to test the transfer you need: new users, new documents, changed formats, longer inputs, or unfamiliar combinations, with clear success criteria for each.
A stronger held-out split
Randomly splitting near-duplicate documents can make a test look easier than it is. Holding out document sources or time periods may better test the intended transfer, though it also creates a harder and different distribution. State which question the split answers.
A model either memorizes or generalizes.
Both can happen within the same model. Different tasks and input distributions reveal different behavior.
Take it with you
- Specify what kind of novelty the test contains.
- Design cases that distinguish rules from shortcuts.
- Do not infer universal transfer from one benchmark.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Compute: the work behind the words
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Deep Learning, chapter 8: Optimization ↗Ian Goodfellow, Yoshua Bengio and Aaron Courville · 2016 · checked 13 Nov 2026
- Multitask Prompted Training Enables Zero-Shot Task Generalization ↗Victor Sanh et al. · 2021-10-15 · checked 13 Nov 2026
- Predictability and Surprise in Large Generative Models ↗Deep Ganguli et al. · 2022-02-15 · checked 13 Nov 2026