Data quality: more than removing typos
Duplicates, contamination, format errors, and missing perspectives affect what a model learns and how it is measured.
In this lesson
A useful dataset needs both suitable examples and trustworthy boundaries between learning and evaluation.
Quality depends on purpose
A legal contract, a poem, and a chat message can all be high-quality examples for different purposes. A filter that keeps only polished formal prose would remove useful conversational language. Before cleaning, define the intended capabilities, languages, and failure modes. Then inspect what the rules keep as carefully as what they discard.
For a toy arithmetic corpus, a beautifully formatted wrong answer remains a bad target. For a style-transfer task, unusual wording might be the point. This is why “quality” should be broken into concrete dimensions such as correctness, coverage, consistency, permission, and formatting instead of represented by one unexplained score.
Duplicates alter the effective mixture
Repeated documents increase repeated exposure. Lee and colleagues showed that deduplication reduced memorized output and train–test overlap in their studied datasets and models. That is evidence for a particular intervention, not a promise that one cleanup pass eliminates memorization. Deduplicating Training Data Makes Language Models Better.
Imagine ten independent recipes plus one recipe copied ninety times. A file count says there are one hundred examples; the learner’s practice is dominated by one recipe. Exact duplicate detection would catch identical copies. Near-duplicate detection is harder because page headers, dates, punctuation, or small edits can hide shared content.
Evaluation must stay separate
If an examination’s answers appear repeatedly in the study material, a high score becomes hard to interpret. In machine learning, training–evaluation overlap creates the same problem. A model may genuinely learn useful patterns and still receive an inflated score from contamination. The question is what the test actually establishes.
A concrete workflow is to reserve evaluation examples before development, compare them with training sources, record known overlaps, and keep a final set untouched while choosing settings. A single random split is insufficient if near-identical versions of the same document land on both sides. Split by meaningful groups when appropriate.
Inspect what filters change
A simple language filter might misclassify short passages. A profanity filter might remove discussions of discrimination as well as abuse. A length filter might discard brief but important instructions. These are illustrative failure modes of rule-based selection: the rule’s name is not evidence that it captures the intended concept.
Try auditing a small sample from each stage: original, accepted, rejected, and transformed. Count reasons for rejection and read examples. Keep the processing recipe and version alongside the dataset. This makes later performance changes easier to investigate: did the model architecture change, or did its practice material quietly change first?
A split that looks clean but is not
“The answer is seven.” goes into training; “The answer is 7.” goes into evaluation. The strings are different, but the underlying example is nearly the same. A trustworthy evaluation needs more than an exact string comparison.
A larger cleaned dataset is automatically a better dataset.
Cleaning can remove valuable coverage, and size can conceal duplication or contamination. Evaluate the specific choices against the intended use.
Take it with you
- Define quality dimensions before choosing filters.
- Deduplication changes exposure and evaluation credibility.
- Inspect rejected data and keep a reproducible preparation record.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Choose your next learning path
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Deduplicating Training Data Makes Language Models Better ↗Katherine Lee et al. · 2022-03-24 (v2) · checked 13 Nov 2026