In-context learning: adapting without weight updates
Instructions and examples in the input can change a model’s behavior while its learned parameters stay fixed.
In this lesson
A model can infer a pattern from the current context without permanently training on it.
Learning in two different senses
The phrase “in-context learning” describes behavior: a model uses examples or instructions in its input to perform a task. It does not ordinarily mean an optimizer changes the model’s parameters during the request. The 2020 GPT-3 study evaluated tasks supplied through text without task-specific gradient updates, helping establish this distinction.
Think of the context as the material available for the current computation. The weights implement the function that processes that material. Supplying a new table changes the function’s input, not the numbers defining the function. A system can therefore adapt its answer immediately without undergoing a training run.
An invented mapping you can inspect
Suppose the prompt says “In this puzzle, dax means red and wug means blue. Translate wug.” A correct answer can use the mapping in the prompt, even if these associations were never part of pretraining. The model need not memorize them for the future; it only needs to use the current evidence appropriately.
Few-shot examples extend the idea. A prompt might show “small → S” and “large → L” before asking for another conversion. The examples can identify the task, demonstrate the output format, and resolve ambiguity. Their usefulness depends on what the pretrained model can infer from them; a few examples are not guaranteed to teach any arbitrary algorithm.
Context effects have limits
If the mapping disappears from the supplied context, an ordinary stateless model call no longer receives it. An application may preserve the mapping by retaining history, storing a note, or retrieving it later. Those are surrounding memory mechanisms. They should not be confused with the model having permanently absorbed the conversation into its weights.
Examples can also mislead. A pattern supported by two demonstrations may admit several interpretations. Order, formatting, and irrelevant details can influence the continuation. For a useful test, vary the examples and evaluate new cases that distinguish the intended rule from a superficial shortcut.
Choose the right adaptation mechanism
In-context examples are convenient when a task changes frequently or the needed information is specific to one request. Fine-tuning may be useful when a stable behavior must be reinforced across many requests. Retrieval can supply external evidence without placing an entire document collection into every prompt. These techniques can be combined.
The central diagnostic question is “What changed?” If only the input changed, call it context conditioning. If parameters changed through updates, call it training. If an external store gained a record, call it application memory. Keeping these operations separate makes persistence, privacy, and reproducibility easier to reason about.
A persistence test
Give a toy model a made-up mapping inside one request, then start a fresh request without it. If the second input contains no mapping and the weights are unchanged, the first success does not imply that the second request can recover it.
A model that learns from an example in a prompt has permanently memorized it.
In-context adaptation uses current input. Persistence requires retained context, external storage, or a separate training process.
Take it with you
- Examples can specify a task without changing weights.
- Contextual adaptation is bounded by the supplied input.
- Distinguish context, external memory, and parameter training.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Retrieval: giving the model evidence
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Language Models are Few-Shot Learners ↗Tom B. Brown et al. · 2020-05-28 · checked 13 Nov 2026