Feed-forward layers: transforming each position
Attention moves information between positions. Feed-forward sublayers transform the features available at each one.
In this lesson
A transformer does more than attention; nonlinear feature transformations are central to its computation.
One function, applied at every position
A position-wise feed-forward sublayer applies the same learned function independently to each position’s vector. It does not directly mix token positions at that step. The vector it receives can already contain information brought in by attention. Thus “position-wise” does not mean “unaware of context”.
Imagine three workbenches using the same set of tools. Each workbench receives different materials, so the outputs differ even though the procedure is shared. This analogy separates shared parameters from shared activations: the weights repeat across positions; the actual numbers being transformed need not.
Expand, transform, project
A simple design projects a vector into an intermediate space, applies a nonlinear activation, then projects back. Alternative designs use gates that multiply one transformed branch by another. Shazeer’s GLU variants paper studies such choices in transformer feed-forward sublayers; no single activation defines the entire architecture family.
For a toy scalar x, consider h = max(0, 2x − 1) and y = 3h. At x = 0, y = 0; at x = 1, y = 3. The threshold makes the function behave differently in different regions. Real layers perform analogous compositions with vectors and matrices, not one scalar.
What is stored in the weights?
Geva and colleagues analyzed feed-forward layers as a form of key-value memory, relating learned directions to input patterns and output tendencies. Their 2021 study is an interpretation of tested models, not evidence that every neuron is a cleanly labeled fact or that a layer contains a searchable document database.
A useful distinction is between a learned association and a retrieved record. The former influences a computation through parameter values. The latter can retain an identifiable source and be supplied as context. Both can affect an answer, but they offer different evidence about where the answer came from.
Why this changes how to read a diagram
A diagram that shows only attention arrows leaves out a substantial part of a standard transformer’s computation. After information arrives, feature transformations can amplify, suppress, or combine it. Later attention layers then work on those changed representations, so the two kinds of sublayer interact across depth.
To reason about a toy block, follow one position’s vector through attention, the feed-forward function, and the residual additions. Keep a record of dimensions and values. If the final result changes while the attention weights stay the same, a feed-forward update or another path may explain the difference.
A nonlinear response
For h = max(0, 2x − 1) and y = 3h, x = 0.25 gives y = 0, while x = 0.75 gives y = 1.5. The change is not described by one global straight line. This scalar example illustrates nonlinear transformation, not a trained language model.
Attention performs all the meaningful work in a transformer.
Feed-forward transformations, residual connections, normalization, and output projections also affect predictions.
Take it with you
- The same feed-forward function is applied across positions.
- Its input can already contain contextual information.
- Memory-like interpretations are not literal document retrieval.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
The residual stream: updating a running representation
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- GLU Variants Improve Transformer ↗Noam Shazeer · 2020-02-12 · checked 13 Nov 2026
- Transformer Feed-Forward Layers Are Key-Value Memories ↗Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy · 2021-09-05 (v2) · checked 13 Nov 2026