Context management: choosing what stays in view
A finite input budget makes selecting, preserving, and refreshing context part of the application’s job.
In this lesson
A context window is a capacity limit; useful memory requires deliberate selection and evidence management.
Account for the whole input
The user’s latest message is only one possible contributor to context. Application instructions, conversation history, retrieved passages, tool descriptions, and tool results may all occupy tokens. Output also needs room under the relevant model’s limits. Exact accounting rules are model- and API-specific, so a product must use its actual documented limits.
The browser demonstration uses an explicitly simplified token budget. Its purpose is to make allocation visible, not to estimate the exact capacity of a named commercial system. When the budget is exceeded, the demonstration shows what is excluded rather than silently pretending everything remains available.
Four different ways to reduce the load
Truncation removes material. Summarization replaces it with a shorter representation. Retrieval selects likely relevant material from an external store. Compression changes representation in another way. These operations have different failure modes: a summary can omit a qualification, while retrieval can miss the relevant source altogether.
An original example is a long project conversation containing a changed deadline. Keeping only the oldest messages preserves the obsolete date; keeping only the newest may lose why it changed. A good summary records the current deadline and the source of the revision, while retaining a path back to the original evidence.
Capacity does not guarantee use
Even when information fits, the model may not use it reliably. Lost in the Middle, published in 2023, found position-sensitive performance in the evaluated long-context tasks and models. This is a dated experimental finding, not a fixed law that every current model ignores every middle passage.
The practical implication is to test your actual information layout. Vary where the relevant passage appears, how many distractors surround it, and whether several sources conflict. A successful single lookup near the beginning of a prompt is not sufficient evidence of robust long-context performance.
Treat memory as a system feature
If an application promises to remember a preference across sessions, it needs a defined storage and retrieval mechanism or a clearly documented training process. Ordinary context conditioning alone does not make a fact persist after it leaves the input. The interface should explain what is stored, how it is used, and how a user can change it.
For a practical context policy, preserve authoritative current instructions, retain evidence for important claims, mark summaries as summaries, and leave room for the response. When information is missing, retrieve it again or ask for clarification. Silent loss is especially damaging because the assistant may continue fluently after the crucial constraint has disappeared.
A transparent toy budget
In a fictional 100-unit budget, instructions use 15, conversation uses 45, retrieved evidence uses 25, and the planned answer needs 20. Total demand is 105. The application must reduce or revise something; a larger-looking chat window does not create five extra units.
Fit the conversation
An illustrative word-piece budget, not any provider’s token count.
Everything ever said in a chat remains available to the model forever.
Only the material actually supplied to a model call can be used directly. Applications may truncate, summarize, or retrieve older content.
Take it with you
- Budget all supplied material, not only the latest message.
- Summaries and retrieval have different failure modes.
- Test whether relevant information is used, not merely whether it fits.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Choose your next learning path
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Lost in the Middle: How Language Models Use Long Contexts ↗Nelson F. Liu et al. · 2023-07-06 · checked 13 Nov 2026