Scaling laws: trends, not guarantees
Empirical curves help allocate training resources, but they do not certify the behavior of a finished assistant.
In this lesson
Scaling laws fit trends within a measured setup. Their domain and objective matter as much as the curve.
What is being predicted?
A scaling study varies quantities such as parameter count, training tokens, and compute, then measures an outcome such as held-out cross-entropy. Kaplan and colleagues’ 2020 study found approximate power-law relationships across the ranges they tested. These are empirical patterns, not mathematical laws requiring every future model to behave identically.
The vertical axis matters. A curve predicting average next-token loss does not directly predict reliability in a medical conversation, tool-use accuracy, or the quality of a citation. A small improvement averaged over billions of tokens can coexist with very different changes on individual tasks.
An allocation problem
With a fixed training budget, resources spent on a bigger model cannot also be spent processing more tokens. A very large model trained briefly and a smaller model trained longer represent different points in this tradeoff. Data quality, architecture, optimization, and hardware add further constraints.
Hoffmann and colleagues’ 2022 Chinchilla study revisited this allocation and argued for scaling model size and training data together under its compute-optimal setup. Treat the dated conclusion as evidence about a particular objective and experimental regime, rather than a timeless ratio that every production system should follow.
Training optimum is not deployment optimum
Imagine two hypothetical checkpoints that cost the same to train. One has fewer parameters but was trained on more examples. If both meet an application’s quality target, the smaller checkpoint may be easier to serve repeatedly. Once millions of future requests enter the calculation, minimum pretraining loss for a fixed initial budget is not the only sensible objective.
Conversely, a model that is inexpensive per generated token may require more tokens, retries, or tool calls to finish a task. The meaningful cost can be cost per successful task. This is an accounting distinction, not a universal claim that either larger or smaller models always win.
How to read a scaling claim
Ask what was held fixed, which data were used, whether runs were fully trained, and where the fitted line is being extrapolated. Curves that agree within observed ranges can diverge outside them. Changing tokenization or dataset composition can also change what a unit on the plot means.
Specific capabilities may be less predictable than aggregate loss. Ganguli and colleagues’ 2022 analysis emphasizes this tension. A useful response is to pair resource forecasting with task-specific evaluation: estimate the broad trend, then measure the behavior you actually need rather than treating the extrapolation as a deployment test.
An illustrative tradeoff
A fictional project can afford either a larger network for one pass or a smaller one for several passes. The better choice depends on measured validation behavior, data repetition, and future serving needs. Parameter count alone does not resolve the choice.
A smooth scaling curve guarantees every capability improves smoothly.
An average-loss trend and a specific task result are different measurements. Extrapolations require validation.
Take it with you
- Always identify the metric behind a scaling claim.
- Compute allocation couples model size and training data.
- Serving cost can change the preferred training decision.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Supervised fine-tuning: demonstrations of behavior
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Scaling Laws for Neural Language Models ↗Jared Kaplan et al. · 2020-01-23 · checked 13 Nov 2026
- Training Compute-Optimal Large Language Models ↗Jordan Hoffmann et al. · 2022-03-29 · checked 13 Nov 2026
- Predictability and Surprise in Large Generative Models ↗Deep Ganguli et al. · 2022-02-15 · checked 13 Nov 2026