The residual stream: updating a running representation
Residual connections let sublayers add changes to a representation that continues through the network.
In this lesson
An additive path carries information across depth; the values on that path are activations, not permanent memory.
Add a change instead of replacing everything
A residual update has the basic form x′ = x + f(x). The sublayer computes a change and adds it to the incoming vector. Residual learning was developed to ease optimization of deep networks, including the influential image-recognition work by He and colleagues. Deep Residual Learning, 2015.
Take x = [1, 2] and f(x) = [0.5, −1]. The updated representation is [1.5, 1]. The original vector contributes directly, while the sublayer contributes an adjustment. The result is still one vector; the two components are not stored as separately labeled text notes.
A communication path through a transformer
Transformer-circuits research uses “residual stream” for the representations read and updated by successive components. That view helps track how information can pass between layers. The cited framework explicitly simplifies some model details, so it should not be mistaken for a complete universal implementation. Elhage and colleagues, 2021.
Suppose an early component adds a feature useful for tracking a subject. A later component can read a direction containing that feature and use it in a new calculation. This is a conceptual example of communication through a vector, not a claim that the model dedicates one permanent coordinate to grammatical subjects.
Addition can preserve or cancel
An additive update is not necessarily an accumulation of more information. If x = [1, 2] and an update is [−1, 0], the first coordinate becomes zero. Components can reinforce or counteract earlier values. Their effect depends on direction as well as magnitude.
Nor does a large update automatically matter most to the output. A final projection may be insensitive to one direction and highly sensitive to another. To assess an update, consider what later computations read, rather than judging its significance from the length of its vector alone.
Keep normalization and time straight
Practical transformer blocks include normalization around their sublayers; the exact arrangement varies. Layer normalization computes statistics within a representation and applies learned scale and shift parameters. Ba and colleagues, 2016. An educational equation that omits this should be read as a simplified account of the residual connection.
The stream also belongs to a particular forward computation. It is not the training dataset and not a permanent conversation memory. Changing the input creates different activations. Updating weights is a separate learning step. This distinction keeps a useful “running representation” metaphor from turning into an inaccurate story about a model remembering every interaction.
An update can erase as well as add
Start with x = [1, 2]. Add [0.5, −1] to obtain [1.5, 1]. Then add [−1.5, 0] to obtain [0, 1]. Residual addition allows later components to counteract earlier coordinates; it does not guarantee monotonic accumulation of information.
The residual stream is a permanent memory shared across all chats.
It is an intermediate representation in a forward pass. Persistent product memory and learned parameters are different mechanisms.
Take it with you
- Residual connections add sublayer outputs to an existing vector.
- Direction matters; updates can reinforce or cancel.
- The stream contains input-dependent activations.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Choose your next learning path
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Deep Residual Learning for Image Recognition ↗Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun · 2015-12-10 · checked 13 Nov 2026
- A Mathematical Framework for Transformer Circuits ↗Nelson Elhage et al., Anthropic · 2021-12-22 · checked 13 Nov 2026
- Layer Normalization ↗Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton · 2016-07-21 · checked 13 Nov 2026