The KV cache: reusing the past
Causal generation can retain attention keys and values from previously processed tokens.
In this lesson
Caching saves repeated computation by storing input-dependent state; it does not update model weights.
Why old work can be reused
In ordinary causal attention, an earlier token’s representation does not depend on future tokens. Its already-computed keys and values can therefore be reused as generation advances, provided the prefix and relevant configuration remain compatible. Transformers documentation on caching describes this reuse for inference.
Imagine processing the prefix “the blue” and then appending “kite”. Without reuse, an implementation could redo work for the old prefix. With a cache, it retains the relevant past keys and values, computes the new token’s state, and attends over the combined past and present information.
What is stored—and what is not
The cache stores derived attention tensors for particular sequences and layers. It is distinct from the model parameters, which are shared learned values. It is also distinct from a database of factual sources. A stored key or value is a numerical representation, not a document citation.
Changing the prefix can invalidate cached state. If “blue” becomes “red”, later states based on “blue” are no longer valid for the modified sequence. Reuse requires matching computation, not merely similar-looking text. A cache can also be discarded and recomputed without retraining the model.
Memory grows with the retained sequence
A simplified dense-cache estimate is 2 × layers × tokens × key-value heads × head dimension × bytes per value, per sequence. The factor two accounts for keys and values. This estimate omits allocation overhead and assumes compatible fixed dimensions; actual implementations may quantize, page, or restrict retained state.
For an invented model with 2 layers, 10 tokens, 2 key-value heads, head dimension 4, and 2 bytes per value, the estimate is 2 × 2 × 10 × 2 × 4 × 2 = 640 bytes. Doubling retained tokens doubles this simplified cache size. It does not double the model’s parameter count.
Architecture and serving choices matter
Multi-query attention reduces distinct key-value heads, and grouped-query attention shares them among groups of query heads. Both are relevant to decoding memory and bandwidth. Shazeer’s 2019 work and Ainslie and colleagues, 2023 study these tradeoffs; their measured results belong to their experimental settings.
For a practical comparison, ask whether a memory figure covers weights only or also cache and temporary buffers. Ask how many concurrent sequences it assumes. A model that fits its weights into memory may still lack room for long or numerous requests. Caching is an efficiency technique with a storage cost, not free unlimited context.
A small cache budget
2 (keys and values) × 2 layers × 10 tokens × 2 KV heads × 4 dimensions × 2 bytes = 640 bytes per sequence in a deliberately simplified dense cache. At 20 tokens, it becomes 1,280 bytes. No learned parameter is added.
A KV cache is the model permanently learning the conversation.
It is reusable inference state tied to a compatible sequence. Discarding it does not erase or change the model’s learned weights.
Take it with you
- Causality makes reuse of past keys and values possible.
- Cache memory and parameter memory are different.
- KV head count and retained sequence length affect cache size.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Choose your next learning path
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- How caching works — Transformers documentation ↗Hugging Face contributors · Main documentation; accessed 2026-11-13 · checked 13 Nov 2026
- Fast Transformer Decoding: One Write-Head is All You Need ↗Noam Shazeer · 2019-11-06 · checked 13 Nov 2026
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints ↗Joshua Ainslie et al. · 2023-12-23 (v3) · checked 13 Nov 2026