What happens when a word is not in the vocabulary? Tokenizers split it into subword pieces so a fixed vocabulary can cover any text: common words stay whole, while rare or long ones break into familiar fragments. For agents handling technical jargon or code, one long term can cost several tokens, more than a plain-English instruction of the same length.
⬇ Subword Tokens worksheet (1 page)
Next:
3. Non-Word Tokens