Loss: a number to learn from
A prediction becomes a training signal when we score how much probability it gave the observed token.
In this lesson
Cross-entropy rewards placing probability on the observed next token, not merely picking a plausible word.
From prediction to a score
A model can produce a smooth, convincing probability distribution and still assign little probability to the token that actually came next. Training needs a numerical way to distinguish those cases. For a single observed target, the usual next-token cross-entropy is L = −ln p(target). PyTorch’s loss reference describes the corresponding logits-based calculation.
The logarithm makes a confident miss costly. In an original three-outcome example, assigning the observed token probability 0.5 gives a loss of about 0.693; probability 0.1 gives about 2.303. A lower number is better for this target. These values use natural logarithms, so their unit is nats. They are arithmetic examples, not measurements of an LLM.
Many positions, one objective
A training sequence supplies many next-token targets. Each position predicts the following token using only the allowed prefix. We can average the resulting losses over the positions selected for training, then average across a batch. Padding or other excluded positions must not accidentally count as genuine targets.
This is why the model need not generate a whole answer during every pretraining example. The known sequence provides the inputs and targets together. The computation can score many positions in parallel while preserving the causal restriction. An average conceals variation: an excellent score on ordinary punctuation can coexist with poor predictions on a rare technical phrase.
Less probability on the target means more loss.
Show the loss calculation
For one observed target, loss = −ln(p). At p = 0.5, −ln(0.5) ≈ 0.693 nats. At p = 0.1, −ln(0.1) ≈ 2.303 nats. Smaller target probability gives a larger penalty.
What a lower loss tells you
Suppose a toy corpus uses “blue” after one prefix 80% of the time and “green” 20%. Always allocating all probability to blue would punish the green examples severely. Matching the mixture can be better than making an overconfident single prediction. Learning a distribution is different from always selecting its most likely outcome.
Training loss measures fit to the examples used for updates. Validation loss uses held-out examples to ask whether that fit transfers. Comparing the two is more informative than watching one line fall. The distinction between an optimization objective and the performance we ultimately care about is central to Deep Learning’s optimization chapter.
Probability is not verification
A low loss does not establish truth. The next observed token may come from a mistaken document, a fictional conversation, or a biased source. The score says how well the model predicts that data under this setup. It does not check the world outside the dataset.
When comparing loss numbers, keep the tokenizer, dataset, masking rules, and averaging convention fixed. A model that splits text into more tokens can report a different per-token loss for reasons unrelated to usefulness. Perplexity is the exponential of mean cross-entropy under a consistent convention; it does not remove these comparison problems.
Check the direction
If the target probability rises from 0.25 to 0.50, the loss falls from 1.386 to 0.693 nats. It is halved in this special example, but doubling probability does not generally halve loss; it subtracts ln 2.
Make one weight learn
A trained scalar logistic model; much smaller than a language model.
Loss measures how factually correct an answer is.
Next-token loss measures probability assigned to observed training targets. Factual evaluation requires a separate check.
Take it with you
- Loss turns probability into an optimization signal.
- The loss is computed before the optimizer updates weights.
- Compare losses only under compatible data and tokenization.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Gradient descent: changing the weights
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- CrossEntropyLoss documentation ↗PyTorch contributors · Living documentation; checked 2026-11-13 · checked 13 Nov 2026
- Deep Learning, chapter 8: Optimization ↗Ian Goodfellow, Yoshua Bengio and Aaron Courville · 2016 · checked 13 Nov 2026