Tokens: the pieces a model reads
A model works with a vocabulary of numbered pieces. Those pieces do not have to line up with words.
In this lesson
A token is a vocabulary entry, and a token ID is its address—not a measurement of meaning.
From writing to a sequence
Before a text model can do numerical work, text is represented as a sequence of token IDs. A token may correspond to a word, part of a word, punctuation, whitespace, or bytes. Subword approaches were developed in part to handle words missing from a fixed word vocabulary. Sennrich and colleagues, 2016.
Imagine a fictional vocabulary with entries for “rain”, “bow”, “rainbow”, and a space. The text “rainbow” could be represented by one entry or two. Neither choice changes the visible spelling. It changes the sequence the model must process and the predictions it must learn. Token boundaries belong to the tokenizer, not to English grammar.
IDs are labels, not quantities
In our imaginary vocabulary, “rain” might have ID 17 and “sun” ID 904. The large numerical gap does not mean these concepts are far apart. An ID simply selects a row from an embedding table. The learned numbers in that row are what enter the network. PyTorch’s embedding definition makes this lookup explicit.
This distinction prevents a common mistake: averaging token IDs does not produce an average meaning. Renumbering the entire vocabulary consistently would preserve the model’s behavior. Changing its embedding values would not. Think of a library’s shelf labels versus the contents of the books on those shelves.
Same text. Different pieces.
Why the pieces matter
Suppose a task asks for the third letter of “rainbow”. The visible word is seven letters, but its model representation might be a single token. The network must connect a token-level representation to a character-level task. This does not make such tasks impossible; it explains why counting characters is different from counting input tokens.
Token counts also govern how much text fits in a context window. An unfamiliar name, source code, or another writing system may split differently from a familiar English sentence. There is no universal conversion from words to tokens. Measure with the tokenizer attached to the actual model when the count matters.
Read the demo honestly
The explorer on this site uses an explicitly simplified algorithm. Its colorful pieces make boundaries visible, but they are not advertised as the output of a production model’s tokenizer. Change spaces, punctuation, and repeated fragments to notice how a segmentation rule affects the result.
A useful check is reconstruction: concatenating the displayed pieces should recover the input under the demo’s stated rules. A second check is whether a piece has a stable ID within the vocabulary being used. Neither check says anything about factual understanding; they test representation before learning begins.
One string, two valid toy vocabularies
A character vocabulary encodes “cat!” as c · a · t · !, four tokens. A vocabulary containing “cat” and “!” encodes the same string as cat · !, two tokens. These are illustrative segmentations, not counts from a named production tokenizer.
Break text into tokens
A simplified longest-match tokenizer, not a product tokenizer.
One token always equals one word.
Token boundaries depend on the vocabulary and encoding rules. Words can span several tokens; a token can also contain whitespace or punctuation.
Take it with you
- Token IDs are discrete labels.
- The tokenizer determines the units the model predicts.
- Use the actual tokenizer for real token budgets.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Tokenization: choosing the alphabet
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Neural Machine Translation of Rare Words with Subword Units ↗Rico Sennrich, Barry Haddow, Alexandra Birch · 2016-08 · checked 13 Nov 2026
- Embedding — PyTorch 2.15 documentation ↗PyTorch contributors · Version 2.15; accessed 2026-11-13 · checked 13 Nov 2026