Subword Splitting

Token problem problems: 2 of 20

Token problems: the series

1. Count the Tokens

2. Subword Splitting

3. Punctuation Counts

4. Tokens per Word

5. Will It Fit?

6. The Price of a Call

7. Two Kinds of Token

8. Input or Output?

9. The Two-Part Bill

10. End-to-End Call Cost

11. Comparing Three Calls

12. The Average Call

13. The System Prompt

14. The System Prompt Tax

15. Calls in a Budget

16. Tokens on a Budget

17. How Long Will It Take?

18. The Split Matters

19. Pricing per Million Tokens

20. Tokens from the Bill

What happens when a word is not in the vocabulary? Tokenizers split it into subword pieces so a fixed vocabulary can cover any text: common words stay whole, while rare or long ones break into familiar fragments. For agents handling technical jargon or code, one long term can cost several tokens, more than a plain-English instruction of the same length.

Problem

A tokenizer splits `preprocessing` into three pieces: `pre`, `process`, and `ing`. The agent receives the pipeline command `run preprocessing now`. How many tokens does it count?

Step 1: Write each token in a box

Step 2: Count the filled boxes

Practice 1

A tokenizer splits `embeddings` into `embed` and `dings`, and `retrieval` into `retriev` and `al`. The agent receives the query `run embeddings retrieval`. How many tokens does it count?

Step 1: Write each token in a box

Step 2: Count the filled boxes

Practice 2

A tokenizer splits `debugging` into `debug` and `ging`, and `codebase` into `code` and `base`. The agent receives the task `start debugging codebase issues`. How many tokens does it count?

Step 1: Write each token in a box

Step 2: Count the filled boxes

---

Next:

3. Punctuation Counts