Transformers: a repeated pattern of computation
Token representations repeatedly exchange information and transform it before producing next-token scores.
In this lesson
A transformer is an architecture family, not a synonym for the entire chatbot product.
Follow one representation
Begin with a sequence of token representations. A transformer block combines an attention operation with a position-wise feed-forward operation, plus surrounding connections and normalization. Repeating blocks lets later computations use representations produced by earlier ones. The 2017 paper introduced the architecture for sequence-to-sequence tasks. Attention Is All You Need.
For this site’s main pipeline, imagine a decoder-style language model. After the final block, the representation at the last input position is converted into vocabulary scores for the next token. The diagram follows that common route; it is not a complete specification of every architecture called a language model.
Two complementary jobs
Attention can bring information from permitted token positions into the representation at the current position. A feed-forward sublayer then transforms features at that position using shared parameters. A useful first mental model is communication followed by computation, while remembering that both operations are themselves computations.
In “the red balloon rose”, a representation near the end may need information associated with earlier words. The architecture allows such interactions, but a drawn arrow from “balloon” to “rose” is only an illustrative possibility. To claim an actual model used that connection would require examining its computed values.
Connections help information travel
Residual connections add a sublayer’s output to its input. Normalization controls aspects of the representation’s scale according to a defined rule. The exact ordering and normalization method vary; diagrams should not silently mix historical and modern recipes. Layer Normalization is a primary reference for one widely used normalization idea.
Think of a block as making an update to a working representation, rather than destroying a token and replacing it with an unrelated object. This makes it easier to understand why multiple layers can successively refine the same position. The metaphor is bookkeeping, not a claim that the model writes explicit human-readable notes.
Architecture versus system
A transformer does not automatically browse, retrieve documents, run code, remember users, or enforce a product policy. An application can surround a model with these components. Keeping the model’s computation separate from the application’s workflow makes capabilities and failure modes easier to identify.
To inspect a complete compact implementation, the historical nanoGPT model is a useful companion. Read it as one recipe: token and position representations, repeated blocks, an output projection, and generation logic. Compare that structure with this site’s pipeline before moving into architecture-specific variations.
Three levels of description
“Transformer” describes an architecture family. “This trained checkpoint” identifies a particular set of learned values. “A chat assistant with document search” describes a surrounding product system. These labels answer different questions and should not be treated as interchangeable.
Every current language model uses exactly the original 2017 Transformer.
Many models adapt the architecture, and some use other approaches. This site explains a common decoder-style path while naming variations where they matter.
Take it with you
- Blocks combine information exchange and feature transformation.
- Residual and normalization details are architecture choices.
- A chatbot adds system behavior beyond the model architecture.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Attention: a computed weighted mixture
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Attention Is All You Need ↗Ashish Vaswani et al., Google and University of Toronto · 2017; revised 2023-08-02 (v7) · checked 13 Nov 2026
- Layer Normalization ↗Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton · 2016-07-21 · checked 13 Nov 2026
- nanoGPT: historical GPT implementation ↗Andrej Karpathy and contributors · Historical implementation; repository accessed 2026-11-13 · checked 13 Nov 2026