Tokens per Word

Token problems: 4 of 20

Token problems: the workbook

1. Count the Tokens

2. Subword Splitting

3. Punctuation Counts

4. Tokens per Word

5. Will It Fit?

6. The Price of a Call

7. Two Kinds of Token

8. Input or Output?

9. The Two-Part Bill

10. End-to-End Call Cost

11. Comparing Three Calls

12. The Average Call

13. The System Prompt

14. The System Prompt Tax

15. Calls in a Budget

16. Tokens on a Budget

17. How Long Will It Take?

18. The Split Matters

19. Pricing per Million Tokens

20. Tokens from the Bill

Does every language tokenize the same way? No. The ratio of tokens to words varies because tokenizers are trained mostly on English. Agents serving multilingual users will pay different costs for the same message depending on the language.

Problem

An AI agent processes a 20-word passage of English text and counts 26 tokens. What is the token-to-word ratio?

Divide tokens by words

Practice 1

An AI agent processes a 24-word passage of Spanish text and counts 36 tokens. What is the token-to-word ratio?

Divide tokens by words

Practice 2

An AI agent processes a 15-word passage of Hindi text and counts 45 tokens. What is the token-to-word ratio?

Divide tokens by words

---

Next:
5. Will It Fit?