Break text into tokens
Type, inspect the pieces, and reconstruct your input with a transparent toy vocabulary.
TRY TO FIND OUT
Can the same word require more than one token?
WHAT YOU ARE LOOKING AT
A simplified longest-match tokenizer, not a product tokenizer. The result is an educational calculation, not a claim about a commercial model’s internal state.
CONNECT THE IDEA
Tokenization: choosing the alphabet
How a tokenizer turns text into reusable pieces—and why its training is separate from the model’s.