Attention heads: several mixtures in parallel
Different projections let a layer combine information through more than one attention pattern.
In this lesson
A head is one attention computation, not a miniature expert with a guaranteed human-readable job.
Why have more than one mixture?
One attention head produces one weighted combination of value vectors per query. Multiple heads use different projections, allowing different mixtures to be computed in parallel and then combined. The original Transformer introduced multi-head attention as part of its architecture. Vaswani and colleagues, section 3.2.
Imagine editing a sentence while tracking both the subject and a nearby adjective. Two different mixtures could preserve information useful for those two relationships. That is a teaching analogy for why parallel projections might help; it does not assert that a particular trained head is permanently “the subject head”.
A tiny construction
Suppose head A mixes scalar values [2, 8] with weights [0.9, 0.1], producing 2.6. Head B uses weights [0.2, 0.8], producing 6.8. Concatenating gives [2.6, 6.8]. A learned output transformation can combine both numbers before the layer updates the residual representation.
This construction can express something different from simply averaging both attention maps first. The heads may have different value projections as well as different weights. Looking only at two heatmaps throws away part of the information that makes their outputs useful.
Head count is not a capability score
Research by Michel and colleagues found that many heads could be pruned with limited performance impact in their studied models, while some components depended more strongly on multiple heads. Are Sixteen Heads Really Better than One?. This shows why head count alone is an incomplete description of effective computation.
A hypothetical design with more heads but smaller per-head vectors changes several tradeoffs at once. Training, dimensionality, routing, and the task all influence the result. Compare measured behavior and computational cost rather than treating an architectural count as a leaderboard.
Keys and values can be shared
Multi-query attention shares keys and values across query heads. Grouped-query attention sits between fully separate and fully shared keys and values. The GQA paper studies this tradeoff to reduce decoding costs while retaining quality in its experimental setting. Ainslie and colleagues, 2023.
For a concrete bookkeeping example, eight query heads might share two key-value groups. There are still eight query-based attention calculations, but fewer separate past key and value tensors to store. This is why a cache estimate needs the number of key-value heads, not automatically the number of query heads.
Two heads, two outputs
Head A: 0.9 × 2 + 0.1 × 8 = 2.6. Head B: 0.2 × 2 + 0.8 × 8 = 6.8. Their concatenated output is [2.6, 6.8], before an output projection. These hand-selected values illustrate the operation rather than measured specialization.
Each head is a separate reasoning agent.
Heads are parallel numerical operations with learned projections. Their roles can overlap, vary with input, or resist simple labels.
Take it with you
- Heads compute different mixtures in parallel.
- Weights and value projections both matter.
- Shared key-value heads change cache costs.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Feed-forward layers: transforming each position
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Attention Is All You Need ↗Ashish Vaswani et al., Google and University of Toronto · 2017; revised 2023-08-02 (v7) · checked 13 Nov 2026
- Are Sixteen Heads Really Better than One? ↗Paul Michel, Omer Levy, Graham Neubig · 2019-11-04 (v3) · checked 13 Nov 2026
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints ↗Joshua Ainslie et al. · 2023-12-23 (v3) · checked 13 Nov 2026