Transformer Ops (Jun 23, 2026)

Counting AI by Hand ✍️ Lecture Series by Mohsena

Library › Counting AI by Hand ✍️

1. Transformer Ops (Jun 23, 2026)

2. Agent Tokens (Jul 14, 2026)

3. Loop Engineering (coming soon, Sep 3, 2026)

Thank you for attending the first lecture of Counting AI by Hand ✍️. Mohsena and I have been working on this series for months, and I am genuinely surprised by how many of you turned up for a topic like this one. This recording is for those who could not join us live.

Although the series is motivated by questions like "why are tokens costing so much?", we decided to go back and start from the foundational layer of counting the FLOPs, or the operations of the transformer architecture behind every model we are using today.

Here are some of the counting questions Mohsena works through in this lecture.

• [Dot Product] Why does a single dot product cost more operations than the vectors have numbers?

• [Context Length] What happens to the cost of attention when you double the length of your prompt?

• [Concatenation] How many operations does it take to concatenate the outputs of six attention heads?

• [Attention vs MLP] Which costs more, attention or the MLP, and does that flip as models get bigger?

• [Backpropagation] How much more does it cost to train a block than to run it once?

She built Excel worksheets to go over how you can "count" by hand to answer these questions. A matrix is a block of cells, and counting the operations is just counting the cells.

Outline

1. What are FLOPs (00:00)

   1. Why FLOP matters

2. Visualizing simple FLOP costs (03:09)

   1. Vector dot product

   2. Matrix-vector product

   3. Matrix-matrix product

   4. FLOPs get huge

3. Flow of the transformer architecture (11:51)

   1. Sequence length (n), embedding size (d), heads (h), layers (L)

4. Counting attention FLOPs (17:27)

   1. Softmax scaling (27:02)

   2. Value aggregation (27:40)

   3. Concat and output projection (29:26)

   4. Attention total (31:52)

5. FLOP counting after attention (33:37)

   1. Residual add and LayerNorm

6. FLOP counting for the MLP (40:48)

   1. Layer 1 and ReLU

   2. Layer 2 and the block total (44:52)

The workbook carries two things the hour did not reach: a full backpropagation FLOP count on its own sheet, and the multi-head attention and full-stack sheets underneath it.

About the instructor

Mohsena Ashraf is a PhD student in computer science at CU Boulder, with a research focus in HCI. Her work is on explainable AI, and specifically on making AI's cost legible to people who are not engineers.

Recording & Workbook

⬇ Excel Workbook

---

Next:
2. Agent Tokens (Jul 14, 2026)