Training data: what the model gets to learn from
A dataset is a designed mixture of examples, not a perfect copy of the world.
In this lesson
The training distribution shapes the patterns a model has opportunities to learn.
A corpus is a collection of choices
A text corpus can mix documents, code, dialogue, scientific writing, and many other formats. The Pile is a historical, openly described example of a deliberately mixed corpus with component-level analysis. Its composition is a case study, not a recipe shared by every model. Gao and colleagues, 2020.
Imagine building a tiny weather assistant from one hundred documents. If ninety are travel advertisements and ten are measurement reports, the model receives far more examples of persuasive travel language. Counting documents alone also hides length: one long report may contribute more training tokens than many short advertisements.
Text becomes training examples
After collection and preparation, text is encoded and arranged into sequences. For a next-token objective, the sequence supplies its own targets: a prefix is the input and the following token is the answer to predict. No person needs to label each ordinary word as “the correct next word”.
In the invented sequence “clouds bring rain”, one training position asks what follows “clouds”; another asks what follows “clouds bring”. Implementations must still decide how to mark document boundaries and which positions contribute to loss. A formatting choice can change what the learner sees as a plausible continuation.
Mixture and coverage
Suppose two topics each appear in ten files. If one topic repeats an identical paragraph in every file, it supplies less varied evidence than ten independent accounts. A dataset needs inspection at several levels: language, source, length, duplication, formatting, and intended use. “A lot of text” is not a quality specification.
A useful original audit question is: which question would this dataset make easy to answer, and which would it barely prepare a model for? If a corpus includes formal proofs but almost no dialogue, conversational behavior may need additional examples. If it excludes recent events, those events cannot be learned from that excluded material.
Data is not a live reference shelf
Training transforms examples into parameter updates. It does not give ordinary inference an automatic search connection to the original files. Some sequences can be memorized, but a generated sentence is not proof that the model retrieved a particular document. Provenance requires a separate evidence path.
The same dataset also raises questions beyond prediction accuracy: permission, privacy, representativeness, and evaluation overlap. Those decisions must be made for the actual collection and intended use. This site uses invented examples and links to research; it distributes no research training corpus and makes no claim that a cited corpus is licensed for every downstream use.
A mixture changes the practice schedule
A toy collection has 900 tokens of poetry and 100 tokens of recipes. Uniform token sampling exposes the learner to poetry roughly nine times as often. Sampling each category equally would create a different training distribution even though the stored files are unchanged.
Training data is a database that the model searches for each answer.
Ordinary training changes weights. Retrieval from documents at answer time is a separate system component.
Take it with you
- Dataset mixture is an engineering and editorial choice.
- Next-token targets can be derived from text itself.
- Training exposure and live retrieval are different.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Data quality: more than removing typos
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling ↗Leo Gao et al., EleutherAI · 2020-12-31 · checked 13 Nov 2026