Embeddings: giving tokens coordinates
A lookup turns discrete IDs into vectors that a neural network can transform.
In this lesson
An embedding is a learned numerical representation, not a dictionary definition written in hidden English.
A row for each token
An embedding table has one row per vocabulary entry and one column per feature dimension. Looking up an ID returns its row. The table’s entries can be learned during training. This is the basic operation documented by PyTorch’s Embedding layer; the same token initially selects the same row in an ordinary lookup table.
Use a fictional vocabulary of four tokens and three dimensions. The table contains twelve numbers. ID 2 chooses the third row, perhaps [0.4, −0.2, 0.9]. The ID has served its purpose. The network now works on coordinates, where additions and matrix products have useful, trainable effects.
Similarity is a relationship, not a label
Vectors let us define numerical similarities. Cosine similarity measures the angle between nonzero vectors: their dot product divided by the product of their lengths. It ranges from −1 to 1. In our original two-dimensional demo, moving one vector changes the angle and therefore the score.
Historical word-vector work demonstrated useful syntactic and semantic relationships in learned vector spaces. Mikolov and colleagues, 2013. But a high cosine score is not a universal guarantee of equivalence. Its meaning depends on the training objective, the representations being compared, and the task for which similarity is useful.
An ID selects a row of numbers.
- Tokenrain
- Vocabulary lookupID 17
- Embedding row[0.2, 0.8]
Context changes the representation
The initial lookup for “bank” can be the same in “river bank” and “bank account”. Subsequent layers combine it with different surrounding information, so later representations can differ. Distinguish a token’s initial embedding from its contextual hidden state; both are vectors, but they occupy different points in the computation.
This is also why a two-axis illustration cannot reveal every dimension of a large model. Our demo uses invented coordinates with no linguistic interpretation. If another teaching diagram labels axes “water” and “money”, those labels need separate justification. Real representations may distribute a feature across many directions, and any projection can hide important differences.
Read vector pictures with care
In the demo, compare vectors pointing in the same direction but with different lengths. Cosine similarity remains the same, while their dot product changes. That distinction matters because different systems choose different similarity measures. A distance ranking should always name the measure it uses.
A zero vector has no defined direction, so its cosine is mathematically undefined. An educational interface should explain this boundary rather than silently inventing a score. This tiny edge case captures a broader lesson: pictures of vectors are useful only when the underlying calculation and its limits stay visible.
A cosine calculation you can check
For a = [1, 0] and b = [1, 1], a·b = 1, |a| = 1, and |b| = √2. Their cosine similarity is 1/√2 ≈ 0.707. These hand-chosen vectors illustrate geometry; they were not extracted from a language model.
Find a direction in meaning
Hand-authored two-dimensional vectors, not learned word meanings.
Each embedding coordinate has one clear human meaning.
Coordinates are learned jointly. A neat semantic axis in a toy chart is an illustration, not a claim about an actual model’s representation.
Take it with you
- A token ID selects a learned vector.
- Similarity depends on the chosen metric and representation.
- Initial embeddings become context-dependent hidden states.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Positions: giving order to the sequence
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Embedding — PyTorch 2.15 documentation ↗PyTorch contributors · Version 2.15; accessed 2026-11-13 · checked 13 Nov 2026
- Efficient Estimation of Word Representations in Vector Space ↗Tomas Mikolov et al. · 2013-09-07 (v3) · checked 13 Nov 2026