Count the Tokens

Token problem problems: 1 of 20

Token problems: the series

1. Count the Tokens

2. Subword Splitting

3. Punctuation Counts

4. Tokens per Word

5. Will It Fit?

6. The Price of a Call

7. Two Kinds of Token

8. Input or Output?

9. The Two-Part Bill

10. End-to-End Call Cost

11. Comparing Three Calls

12. The Average Call

13. The System Prompt

14. The System Prompt Tax

15. Calls in a Budget

16. Tokens on a Budget

17. How Long Will It Take?

18. The Split Matters

19. Pricing per Million Tokens

20. Tokens from the Bill

How does a model read text? Not word by word, but token by token. Splitting on spaces is the simplest tokenizer: one word, one token, one box. Every instruction you send to an agent gets split this way before the model sees it.

Problem

You type the command `fetch the latest price data` into a task-planning agent. The agent splits the text on spaces, counting each word as one token. How many tokens does it count?

Step 1: Write each token in a box

Step 2: Count the filled boxes

Practice 1

You send the sentence `Agents think step by step` to an AI agent. The agent splits the text on spaces, counting each word as one token. How many tokens does it count?

Step 1: Write each token in a box

Step 2: Count the filled boxes

Practice 2

You send the instruction `plan and run the next step` to an automation agent. The agent splits the text on spaces, counting each word as one token. How many tokens does it count?

Step 1: Write each token in a box

Step 2: Count the filled boxes

---

Next:

2. Subword Splitting