Positions: giving order to the sequence
The same tokens in a different order can mean something different. A model needs a way to represent that order.
In this lesson
Position information helps distinguish who came before whom; the implementation varies by architecture.
A bag of tokens is not a sentence
“The dog chased the cat” and “The cat chased the dog” use the same words but describe different events. A computation that knew only which items were present would miss that reversal. Position mechanisms make sequence location available to a transformer alongside token identity.
The original Transformer used sinusoidal positional encodings and also tested learned alternatives. That is one historical design, not a rule that every current model adds the same kind of vector. Vaswani and colleagues, section 3.5. When reading an architecture diagram, ask exactly where and how order enters.
Absolute and relative views
An absolute position says “this is slot 12”. A relative position says “this item is three slots before that one”. Both can matter. Absolute location may distinguish the beginning of a sequence; relative distance may help recognize a nearby modifier or a repeated pattern. A model can combine different sources of ordering information.
In a simple additive illustration, the token vector [1, 0] plus position vector [0, 1] becomes [1, 1]. At another slot, adding [0, 2] yields [1, 2]. These invented vectors show how the same token can start with different representations; they do not describe the numerical encodings used by a named model.
Same token, different position.
Show the position calculation
At position i, this illustration adds [0.05 × i, −0.03 × i]. For i = 1, [1 + 0.05, 0 − 0.03] = [1.05, −0.03]. This is the teaching rule used in the site’s pipeline, not a universal positional encoding.
Rotations offer another route
Rotary position embeddings apply position-dependent rotations to query and key representations, connecting their interaction to relative position. RoPE was introduced in the RoFormer work. Su and colleagues, 2021. This differs from simply adding a position vector to the input embedding.
Imagine two arrows on a plane. Rotating them according to their sequence positions changes their dot product in a structured way. Real RoPE acts across pairs of coordinates using a defined set of frequencies. The arrow analogy explains the operation’s shape, while the linked paper gives the actual formulation.
Longer does not automatically mean reliable
A position scheme that accepts a larger index does not by itself prove a model will handle longer sequences well. Its learned behavior, training lengths, attention design, and inference configuration still matter. Extending a numerical coordinate system is different from demonstrating robust use of the extra context.
For a practical thought experiment, insert a long paragraph between a question and its supporting sentence. The tokens still have positions, but solving the task now requires finding and using information across a larger gap. Evaluate that behavior directly instead of treating a larger nominal window as a complete capability claim.
Identity plus location
In “very very small”, both occurrences of “very” can share a token ID, while occupying different positions. A position mechanism allows the computation to distinguish the first occurrence from the second without assigning them different vocabulary entries.
Every transformer adds a fixed position embedding to every token.
Architectures use different methods, including learned or sinusoidal additions and rotations applied inside attention.
Take it with you
- Token identity does not fully specify sequence order.
- Absolute and relative position information are distinct ideas.
- Long-context reliability requires behavioral evidence.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Parameters: the numbers learning changes
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Attention Is All You Need ↗Ashish Vaswani et al., Google and University of Toronto · 2017; revised 2023-08-02 (v7) · checked 13 Nov 2026
- RoFormer: Enhanced Transformer with Rotary Position Embedding ↗Jianlin Su et al. · 2021; revised 2023-11-08 (v5) · checked 13 Nov 2026