Causal masking: no peeking at the answer
A next-token learner must only use the prefix available at the moment it predicts.
In this lesson
Causal masking controls information access, making parallel training consistent with left-to-right generation.
The training problem
A training sequence already contains the tokens that serve as targets. Without an access restriction, a position could read the future answer it is supposed to predict. Causal attention prevents a position from attending to later positions. PyTorch’s causal attention definition implements the square case with a lower-triangular permitted region.
Use the toy sequence “birds can fly”. At the position for “birds”, the model should predict “can” using “birds” and any earlier context. If that position could inspect “can” directly, the exercise would no longer test prediction from the available prefix. The mask closes this shortcut.
What gets masked
For query position i, ordinary causal self-attention permits key positions j ≤ i. Scores for j > i are excluded before softmax, conventionally represented as negative infinity. Their exponential contribution is zero. The remaining entries are normalized, so a row still sums to one when it has at least one permitted position.
Notice that the current position can attend to itself. Its representation is used to predict the following token, so reading itself is not cheating. Confusing the input token with the shifted target is a common source of off-by-one mistakes in explanations and implementations.
A position can use its past, not its future.
| From ↓ / To → | the | cat | sat |
|---|---|---|---|
| the | Can use | Masked | Masked |
| cat | Can use | Can use | Masked |
| sat | Can use | Can use | Can use |
Parallel training, sequential generation
During training, all known positions can be processed together because the mask separately restricts each position’s information. During ordinary autoregressive generation, the next token does not yet exist, so it must be selected before the following prediction can use it. This is a dependency difference, not a contradiction.
Draw three rows on paper. Row one may read column one. Row two may read columns one and two. Row three may read all three. Every row can be calculated in the same batch while respecting its own prefix. Later layers must maintain the same restriction to avoid leaking future information indirectly.
A mask is not a safety policy
The word “causal” here describes sequence access. It does not mean the model has discovered real-world causal relationships, and it does not filter harmful content. Nor does every attention task require this mask: an encoder that reads an entire supplied passage can legitimately use both directions.
In the demo, turn causal masking off and inspect the newly available cells. This comparison illustrates information flow. It does not turn the toy into another trained model or show how a real model would perform if its training mask were changed after the fact. Training and inference rules must be compatible.
A three-token permission table
For positions A, B, C, the permitted keys are A → {A}, B → {A,B}, and C → {A,B,C}. The forbidden cells are A→B, A→C, and B→C. Those cells have zero attention weight after masking and normalization.
See attention add up
A small numerical attention calculation, not a window into reasoning.
A causal mask stops a model from saying unsafe things.
It limits access to future sequence positions. Content policies and safeguards are separate system mechanisms.
Take it with you
- Targets must not leak into their own predictions.
- The current input position is allowed; future positions are not.
- Parallel training can preserve left-to-right information constraints.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Attention heads: several mixtures in parallel
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Scaled dot-product attention — PyTorch 2.15 documentation ↗PyTorch contributors · Version 2.15; accessed 2026-11-13 · checked 13 Nov 2026