Count the Tokens

Token problems: 1 of 20

Token problems: the workbook

1. Count the Tokens

2. Subword Splitting

3. Punctuation Counts

4. Tokens per Word

5. Will It Fit?

6. The Price of a Call

7. Two Kinds of Token

8. Input or Output?

9. The Two-Part Bill

10. End-to-End Call Cost

11. Comparing Three Calls

12. The Average Call

13. The System Prompt

14. The System Prompt Tax

15. Calls in a Budget

16. Tokens on a Budget

17. How Long Will It Take?

18. The Split Matters

19. Pricing per Million Tokens

20. Tokens from the Bill

How does a model read text? Not word by word, but token by token. Splitting on spaces is the simplest tokenizer: one word, one token, one box. Every instruction you send to an agent gets split this way before the model sees it.

Problem

You type the command fetch the latest price data into a task-planning agent. The agent splits the text on spaces, counting each word as one token. How many tokens does it count?

Write each token in a box

Count the filled boxes

Practice 1

You send the sentence Agents think step by step to an AI agent. The agent splits the text on spaces, counting each word as one token. How many tokens does it count?

Write each token in a box

Count the filled boxes

Practice 2

You send the instruction plan and run the next step to an automation agent. The agent splits the text on spaces, counting each word as one token. How many tokens does it count?

Write each token in a box

Count the filled boxes

---

Next:
2. Subword Splitting