Library › Counting AI by Hand ✍️
Transformer Ops (Jun 23, 2026)
Thank you for attending the first lecture of Counting AI by Hand ✍️. Mohsena and I have been working on this series for months, and I am genuinely surprised by how many of you turned up for a topic like this one. This recording is for those who could not join us live.
Although the series is motivated by questions like "why are tokens costing so much?", we decided to go back and start from the foundational layer of counting the FLOPs, or the operations of the transformer architecture behind every model we are using today.
Here are some of the counting questions Mohsena works through in this lecture.
[Dot Product] Why does a single dot product cost more operations than the vectors have numbers?
[Context Length] What happens to the cost of attention when you double the length of your prompt?
[Concatenation] How many operations does it take to concatenate the outputs of six attention heads?
[Attention vs MLP] Which costs more, attention or the MLP, and does that flip as models get bigger?
[Backpropagation] How much more does it cost to train a block than to run it once?
She built Excel worksheets to go over how you can "count" by hand to answer these questions. A matrix is a block of cells, and counting the operations is just counting the cells.
Outline
What are FLOPs (00:00)
Why FLOP matters
Visualizing simple FLOP costs (03:09)
Vector dot product
Matrix-vector product
Matrix-matrix product
FLOPs get huge
Flow of the transformer architecture (11:51)
Sequence length (n), embedding size (d), heads (h), layers (L)
Counting attention FLOPs (17:27)
Softmax scaling (27:02)
Value aggregation (27:40)
Concat and output projection (29:26)
Attention total (31:52)
FLOP counting after attention (33:37)
Residual add and LayerNorm
FLOP counting for the MLP (40:48)
Layer 1 and ReLU
Layer 2 and the block total (44:52)
The workbook carries two things the hour did not reach: a full backpropagation FLOP count on its own sheet, and the multi-head attention and full-stack sheets underneath it.
About the instructor
Mohsena Ashraf is a PhD student in computer science at CU Boulder, with a research focus in HCI. Her work is on explainable AI, and specifically on making AI's cost legible to people who are not engineers.