Punctuation Counts

Token problems: 3 of 20

Token problems: the workbook

1. Count the Tokens

2. Subword Splitting

3. Punctuation Counts

4. Tokens per Word

5. Will It Fit?

6. The Price of a Call

7. Two Kinds of Token

8. Input or Output?

9. The Two-Part Bill

10. End-to-End Call Cost

11. Comparing Three Calls

12. The Average Call

13. The System Prompt

14. The System Prompt Tax

15. Calls in a Budget

16. Tokens on a Budget

17. How Long Will It Take?

18. The Split Matters

19. Pricing per Million Tokens

20. Tokens from the Bill

Does punctuation count as a token? Yes. Most tokenizers treat each mark as its own token, carrying structure the model needs: sentence boundaries, pauses, questions. Agents that read or write code, JSON, or structured data pay this cost on every bracket and comma.

Problem

An AI agent counts each punctuation mark as its own token. The agent receives Hello, world!. How many tokens does it count?

Write each token in a box

Count the filled boxes

Practice 1

An AI agent counts each punctuation mark as its own token. The agent receives Wait, what?. How many tokens does it count?

Write each token in a box

Count the filled boxes

Practice 2

An AI agent counts each punctuation mark as its own token. The agent receives Stop, look, and listen!. How many tokens does it count?

Write each token in a box

Count the filled boxes

---

Next:
4. Tokens per Word