Like me, Dandekar's most preferred teaching method is sketching and writing by hand. ✍️ His AI teaching channel on YouTube has more than 200K+ subscribers https://www.youtube.com/@vizuara

We will start from a small decision we solve on paper, get deeper, and connect all the way to writing kernels to run on a GPU.

Frontier AI Seminar Series ~ Fall 2026

Title: From Jev to Kernels

Speaker: Rajat Dandekar, Co-Founder, Vizuara, PhD. MIT.

Host: Prof. Tom Yeh, Department of Computer Science, CU Boulder, AI by Hand ✍️

Date: Friday, October 9

Time: 9am Pacific Time / 10am Mountain

Luma Event: https://luma.com/byhand-v8nj

Register

Read more

Library › RNNs by Hand, in Excel

1. Recurrence

2. Size

3. Outputs

4. Depth

5. Seq2Seq

6. Tokens

7. Autoregressive

8. Loss

9. Small Language Model

This is the last article of the series. It has one example: the small language model from the start of the series, with every piece now in place.

words, d=6, h=6, layers=4, vocab=30, seq=30, softmax, sample, tied

It reads a word, looks up its embedding, and passes it through four stacked layers. Then it scores every word in the vocabulary, samples one, and reads that word next. Thirty steps later, it has written five sentences.

Now you are rewarded with an understanding of everything important to a language model, including:

• recurrence: the hidden state, and the same weights reused at every step

• hidden size, input size, and batches

• several outputs per step, softmax, and many-to-one

• stacked layers and bidirectional RNNs

• the encoder, the decoder, and the bottleneck of the context between them

• tokens, one-hot vectors, embeddings, and tied embeddings

• argmax and sampling

• autoregressive generation and prompts

• loss and its gradients

I stop short of backpropagation through time. I am still in the middle of doing research on how to represent it in Excel.

The download below has the whole language model in Excel, with live calculations.

Read more

Library › RNNs by Hand, in Excel

1. Recurrence

2. Size

3. Outputs

4. Depth

5. Seq2Seq

6. Tokens

7. Autoregressive

8. Loss

9. Small Language Model

So far, every weight was chosen by hand. How do we choose them from data instead? We need a measure of how wrong a network is. That measure is the loss. We give each output a target, the answer we want, and the loss compares the two. Then we ask which way each output should move to make the loss smaller. That direction is its gradient.

d=1, h=1, seq=5, L2 loss

Let's start with the d=1, h=1, seq=5 network of the Recurrence article. Its weights and inputs are the same, so every yₜ is too. Under each output, we write its target, y\*ₜ, in cream, because it is given, like an input. The loss at step t is lₜ = ½(yₜ − y\*ₜ)². It is half the squared difference, called the L2 loss. It is zero when the output hits its target, and it grows the further away it is. The loss of the whole sequence, L, is the mean of the five, in purple, under l₁. In practice, a sequence runs hundreds or thousands of steps. The mean keeps L on the same scale however long it gets. The largest lₜ is at step 2, where the output is furthest from its target. In the download, type another target, or change a weight, and watch L move.

Read more

Library › RNNs by Hand, in Excel

1. Recurrence

2. Size

3. Outputs

4. Depth

5. Seq2Seq

6. Tokens

7. Autoregressive

8. Loss

9. Small Language Model

In the last article, the network read tokens and wrote them. But which token it picked had no influence on what came next. How does a network read what it writes? Each output becomes the next step's input. A network that feeds on its own outputs is called autoregressive.

d=1, h=2, y=1, seq=5, one-to-many

We start with a one-to-many network, like the decoder of the Seq2Seq article, reading a single number at its first step. The first input, x₁ = 1, starts it. After that, every input is 0, dimmed, so each step runs on its state alone. The five outputs are 0.76, −0.43, −0.24, 0.58, and −0.28. In the download, change x₁, and all five outputs follow.

Read more

Library › RNNs by Hand, in Excel

1. Recurrence

2. Size

3. Outputs

4. Depth

5. Seq2Seq

6. Tokens

7. Autoregressive

8. Loss

9. Small Language Model

In the last article, every sequence was made of numbers. How does a network read a word, and write one? It reads and writes tokens. A token is a unit of text: a letter, a word, or part of a word. I use letters, so every example fits on a screen.

cab, d=2, h=2, seq=3, many-to-one, sigmoid

Let's start with a vocabulary of three letters, a b c, and read the word cab. Each letter becomes a vector of two numbers. We keep the vectors in a table, E, with one column per letter. Each column is an embedding. In these examples, E is fixed, so it has no fill. In practice, E is usually trained along with the weights. To read a letter, we look it up, finding the column headed by that letter. So cab becomes three vectors, x₁ to x₃, in cream. From there, the RNN reads them many-to-one, as in the Outputs article. A sigmoid gives one probability for the whole word. In the download, type another word made from a b c, and every step recomputes.

Read more

Library › RNNs by Hand, in Excel

1. Recurrence

2. Size

3. Outputs

4. Depth

5. Seq2Seq

6. Tokens

7. Autoregressive

8. Loss

9. Small Language Model

In the last article, our network read a sequence and gave its answers along the way. In this article, the answer is a sequence too. How does a network turn one sequence into another? We build it from two halves. An encoder reads the input sequence into one vector. A decoder turns that vector into an output sequence. Together, they form a sequence-to-sequence network, or seq2seq.

d=1, h=2, y=3, seq=5, many-to-one

We start with the encoder. It is a many-to-one RNN, as in the Outputs article. We label it Encoder, at the top left. Its one output, y, is what the encoder knows about the whole sequence. Compare y with the Outputs article, and you find the last output of its y=3 example.

Read more

Library › RNNs by Hand, in Excel

1. Recurrence

2. Size

3. Outputs

4. Depth

5. Seq2Seq

6. Tokens

7. Autoregressive

8. Loss

9. Small Language Model

In the last three articles, our network had one hidden layer. How do we make it deeper? We can stack layers, each reading the hidden state of the layer below. Or we can read the sequence in both directions at once. Each example builds on one we have already seen, so we keep an eye on what stays the same.

d=1, h=1, h′=1, seq=5

We stack a second layer on top of the first. We mark it with a prime. Its hidden state is h′, and its weights are W′x, W′h and b′. At each step, layer 2 reads the hidden state hₜ from layer 1. It also reads its own state from the step before, h′ₜ₋₁. The output now comes from layer 2. Why does the h row match the last example of the first article? Because layer 1 is exactly that network.

Read more

Library › RNNs by Hand, in Excel

1. Recurrence

2. Size

3. Outputs

4. Depth

5. Seq2Seq

6. Tokens

7. Autoregressive

8. Loss

9. Small Language Model

In the last two articles, every step gave one output. In this article, we change what comes out. What if each step gave several outputs, or the whole sequence gave just one? Most examples start from the h=2 network of the last article. They change only what comes after the hidden state.

d=1, h=2, y=2, seq=5

We ask each step for two outputs. The hidden states do not change. Each output reads the same hₜ, with its own row of weights. Compare the first output with the h=2 example, and you will see they match.

Read more

Library › RNNs by Hand, in Excel

1. Recurrence

2. Size

3. Outputs

4. Depth

5. Seq2Seq

6. Tokens

7. Autoregressive

8. Loss

9. Small Language Model

In the last article, every input and every memory had size one. In this article, we grow them. The hidden size, h, is how many numbers the hidden state holds. The input size, d, is how many numbers each step reads. We also read a batch, several sequences at once. Within this article, we keep the weights fixed, so each example differs from the last in one way only.

d=1, h=2, seq=5

We give the memory a second number, in green. Every step still does the same thing. At step t, it reads its input xₜ and the hidden state from the step before, hₜ₋₁. What grows is the weights. Wx, b and Wy each gain an entry, one per memory number. Wh becomes a matrix. Why a matrix? Because each new memory number reads both old ones. Follow the arrows in the download; each one now carries the whole memory to the next step.

Read more

Library › RNNs by Hand, in Excel

1. Recurrence

2. Size

3. Outputs

4. Depth

5. Seq2Seq

6. Tokens

7. Autoregressive

8. Loss

9. Small Language Model

How does a network remember? A recurrent neural network, or RNN, reads a sequence one step at a time. At each step, it takes one input and updates a hidden state. The hidden state is what the network remembers. We build an RNN in Excel, one change at a time. I keep the weights fixed, so each example differs from the last in one way only.

d=1

Let's start with the smallest network there is. It reads one input, x, in the cream cell at the top. The weight Wx and the bias b, in light red on the left, turn x into F. The bias is the darker red. We write the bias as one more weight, and put a constant 1 under x for it to multiply. So [Wx b] · [x; 1] is the same as Wx · x + b. A tanh turns F into the hidden value h, in green. Then Wy and by turn h into the output y, in blue. There is no memory yet.

Read more

September 5th is India’s Teachers' Day. We are pleased to announce the AI By Hand ✍️ Yantra Jnana Award, which is to recognize extraordinary work teaching AI in an Indian language, by hand, without relying on expensive technologies.

Please share this announcement with people you know who may be interested in this award!

To enter the award, record a video of you teaching AI by hand in one of the following languages:

• Hindi

• Punjabi

• Marathi

• Gujarati

• Kannada

• Tamil

• Telugu

• Malayalam

We know there are a lot more languages in India. But please allow us to start with this initial round of 8 languages, which I believe have the most number of native speakers. If this round is well-received, we will definitely give more awards for cover other languages in the future.

Awards

We currently have funding to give 10 awards: one in each of the eight languages, and two more to the strongest entries remaining, in any language.

Each award carries:

• A fellowship of ₹40,000 (to support you to teach AI by hand to more people)

• An award certificate (you can hang on the wall)

• A year of Packt subscription

• A year of paid subscription for my AI by Hand newsletter

• Your video featured on my social media channels and YouTube

Format Requirements

These requirements are what make this an AI By Hand ✍️ award, so please read it closely.

1. Print the material for your topic on paper, and work from it.

2. Or use a blackboard or a whiteboard. No iPad, no tablet, no digital ink.

3. Keep your hand in the frame for the entire video.

Judging Criteria

Every entry is read by a panel of expert reviewer who work in your language.

Every entry is read by an expert reviewer who works in your language, and scored on five things:

• Conceptual accuracy (35%): is the idea exactly right, however you chose to explain it?

• Pedagogical value (25%): does it build real intuition, or only state facts?

• Language quality (20%): is the language fluent and natural?

• Originality of approach (15%): is the teaching style your own, rather than a copy of the source’s framing?

• Production quality (5%): is it clear enough to watch and hear?

These criteria and weights are subject to change until the submission window opens on September 12.

Timeline

• Saturday, September 5: Registration opens

• Friday, September 18: Registration closes

• Friday, October 16: Videos are due

• Friday, October 30: Awards are announced

Register

Registering is the only step to take right now. Once you have registered, we will email you the submission instructions.

Register by Friday, September 18: byhand.ai/yantra/register

Register

The method is the same in every lecture. Instead of equations and Greek symbols, operations and tokens become 2D rectangles in a spreadsheet, and the area of a rectangle is the cost. Once you can see it, you can measure it, compare it, and count it by hand. Everything happens in Excel, so every formula is visible and every workbook is yours to keep and to modify.

1. Transformer Ops

2. Agent Tokens

3. Loop Engineering [coming Sep 3]

This series is sponsored by By Hand Research.

1. Transformer Ops (Jun 23, 2026) ^(free)

2. Agent Tokens (Jul 14, 2026) ^(free)

3. Loop Engineering (coming soon, Sep 3, 2026) ^(free)

Library › Counting AI by Hand ✍️

1. Transformer Ops (Jun 23, 2026)

2. Agent Tokens (Jul 14, 2026)

3. Loop Engineering (coming soon, Sep 3, 2026)

The third lecture of Counting AI by Hand ✍️ is on September 3.

The first lecture counted the operations inside the model, and the second counted the tokens moving through an application. This one counts what happens when nobody is re-prompting: the agent reads the goal, picks an action, checks whether it is done, and goes around again. You used to be the loop. Now the loop runs on its own, and every turn of it is billed.

Here is a preview of the workbook.

About the instructor

Mohsena Ashraf is a PhD student in computer science at CU Boulder, with a research focus in HCI. Her work is on explainable AI, and specifically on making AI's cost legible to people who are not engineers.

Workbook

Three exercises, all of them the same job search agent looping until it collects five good opportunities.

1. Loop Cost. Eight opportunities have to be checked before five turn out to be eligible, so how many LLM calls is that, and what do the input tokens, the output tokens and the bill come to?

2. Expected Loops and Context Growth. Each opportunity has only a 50% chance of being eligible, so how many loops should you expect to run, and how much of what you pay for is the context that grows with every one of them?

3. Loop Optimization. The same three steps in two different orders, priced side by side. One order is cheaper, and the reason is that the expensive step runs fewer times.

There is a Solution sheet with every answer worked out, so you can fill the tables in yourself first or read straight through.

⬇ Excel Workbook

The recording will be posted here after the lecture.

Library › Counting AI by Hand ✍️

1. Transformer Ops (Jun 23, 2026)

2. Agent Tokens (Jul 14, 2026)

3. Loop Engineering (coming soon, Sep 3, 2026)

Thank you for attending the second lecture of Counting AI by Hand ✍️, and for those who could not join us live, here is the recording.

The first lecture counted the operations inside the model. This one counts the tokens moving through an application, which is where the question that started the series finally gets a direct answer.

Here are some of the counting questions from this lecture.

• [Input vs Output] Why does the text the model writes cost more than the text you typed?

• [Conversation History] What makes the same short question cost more late in a chat than at the start?

• [Tool Calls] Which costs more, three turns of plain chat or two turns that call a tool?

• [RAG] How many times do you pay for a chunk your retriever keeps finding?

• [Multi-Agent] What does it cost when every agent in a system keeps its own conversation?

Mohsena works through every one of them in Excel, counting by hand. The cells hold tokens this time, and the total is your bill.

Outline

1. Introduction (00:00)

2. Tokens basics, and why output costs more (01:51)

   1. FLOPs vs tokens

   2. What a user sees

   3. What actually happens: system prompt + history + tool results = input

3. Chat history token counting (08:01)

   1. Basic chat, priced per turn

4. Tool calls and token billing (18:07)

   1. Tool definitions, the call, the output

   2. Why tool calls cost more (30:24)

5. RAG workflow and token math (32:15)

   1. The same chunk, retrieved twice and paid for twice

   2. Naive vs optimized: what has to be in the next request

6. RAG variations and other inputs (47:15)

   1. Image + text

7. Memory and multi-agent costs (48:27)

   1. Two sessions, and what the application stores between them

   2. Planner and specialists, each with their own conversation

About the instructor

Mohsena Ashraf is a PhD student in computer science at CU Boulder, with a research focus in HCI. Her work is on explainable AI, and specifically on making AI's cost legible to people who are not engineers.

Recording & Workbook

⬇ Excel Workbook

Next:
3. Loop Engineering (coming soon, Sep 3, 2026)

Library › Counting AI by Hand ✍️

1. Transformer Ops (Jun 23, 2026)

2. Agent Tokens (Jul 14, 2026)

3. Loop Engineering (coming soon, Sep 3, 2026)

Thank you for attending the first lecture of Counting AI by Hand ✍️. Mohsena and I have been working on this series for months, and I am genuinely surprised by how many of you turned up for a topic like this one. This recording is for those who could not join us live.

Although the series is motivated by questions like "why are tokens costing so much?", we decided to go back and start from the foundational layer of counting the FLOPs, or the operations of the transformer architecture behind every model we are using today.

Here are some of the counting questions Mohsena works through in this lecture.

• [Dot Product] Why does a single dot product cost more operations than the vectors have numbers?

• [Context Length] What happens to the cost of attention when you double the length of your prompt?

• [Concatenation] How many operations does it take to concatenate the outputs of six attention heads?

• [Attention vs MLP] Which costs more, attention or the MLP, and does that flip as models get bigger?

• [Backpropagation] How much more does it cost to train a block than to run it once?

She built Excel worksheets to go over how you can "count" by hand to answer these questions. A matrix is a block of cells, and counting the operations is just counting the cells.

Outline

1. What are FLOPs (00:00)

   1. Why FLOP matters

2. Visualizing simple FLOP costs (03:09)

   1. Vector dot product

   2. Matrix-vector product

   3. Matrix-matrix product

   4. FLOPs get huge

3. Flow of the transformer architecture (11:51)

   1. Sequence length (n), embedding size (d), heads (h), layers (L)

4. Counting attention FLOPs (17:27)

   1. Softmax scaling (27:02)

   2. Value aggregation (27:40)

   3. Concat and output projection (29:26)

   4. Attention total (31:52)

5. FLOP counting after attention (33:37)

   1. Residual add and LayerNorm

6. FLOP counting for the MLP (40:48)

   1. Layer 1 and ReLU

   2. Layer 2 and the block total (44:52)

The workbook carries two things the hour did not reach: a full backpropagation FLOP count on its own sheet, and the multi-head attention and full-stack sheets underneath it.

About the instructor

Mohsena Ashraf is a PhD student in computer science at CU Boulder, with a research focus in HCI. Her work is on explainable AI, and specifically on making AI's cost legible to people who are not engineers.

Recording & Workbook

⬇ Excel Workbook

Next:
2. Agent Tokens (Jul 14, 2026)

Library › Seminar Series 2026

1. Manifold-Constrained Hyper Connections (mHC) from DeepSeek (Jan 9, 2026)

2. How Small Models Learn Tool Use (Jan 11, 2026)

3. Generative AI (Jan 11, 2026)

4. Attention (Jan 15, 2026)

5. Google Ironwood TPU: From Bits to HBM (Jan 19, 2026)

6. 9 AI Eval Formulas You Must Know (Jan 24, 2026)

7. Meta Superintelligence Labs vs Facebook AI Research (Jan 30, 2026)

8. Transformer: Six Levels of Understanding (Feb 13, 2026)

9. OpenClaw Seminar (Feb 23, 2026)

10. PPO → DPO → GRPO → Rubrics (Mar 2, 2026)

11. Gemma 4 (May 7, 2026)

12. Qwen 3.6 (May 21, 2026)

13. Kimi 3 (Aug 6, 2026)

Thank you to everyone who joined the Frontier AI Seminar on Kimi 3.

It has been a while since I gave the last seminar on Qwen. Happy to get together with many of you again to study frontier model math and architecture by hand in Excel!

We are blessed with Nathan Lambert, the author of Interconnects, some of the most authoritative writing on open models, as a special guest. Nathan also wrote the RLHF book which has just been published by Manning.

Here are some of my favorite moments:

• Nathan shared his behind-the-scenes story about his trip to Moonshot AI.

• Nathan's famous dog Phoebe had a surprise appearance on camera.

• I got to finally talk about Linear Transformer line of frontier model advances.

• I got to mention LeBron James in my explanation of the attention mechanism.

Thanks for your feedback!

Before the summer, I did plan to talk about Linear Transformer line and wanted to use NVIDIA Nemotron as the backdrop. However, when Kimi 3 was released, I found an even stronger storyline, one where I can track the evolution much further from Linear Attention, DeltaNet, to the latest Kimi Delta Attention (KDA).

Outline

1. Interview and Discussion

   • Nathan Lambert

   • Moonshot AI Tour

   • Open vs Closed Models

   • Audience Q&A

2. Kimi 3 by Hand

   • Attention

   • Quadratic Attention

   • Inference and KV Cache

   • Local Attention

   • Linear Attention

   • DeltaNet

   • Kimi Delta Attention (KDA)

   • Convolution and Gating

   • Chunkwise Parallel Computation

   • Seq2Seq LSTM

Recording & Workbook

Limited-time free: the full recording and the Excel workbook are open to everyone for now.

⬇ Excel Workbook

Every model has a context window, and every window is a budget. These problems ask you to spend it. You will mark out a window and fill it turn by turn, work out whether the next message still fits, and count how many turns you get before it does not. Then retrieval arrives and takes its share, and you will see what a single retrieved document does to the turns you had left.

Halfway through, the window runs out and you have to decide what to do about it. Dropping the oldest turn is the simple answer, and problem 10 asks what that costs you. Compaction is the better one, and five problems work through what it spends, when to reach for it, and what survives.

The last five changed how I think about long conversations. A stateless API resends the entire context every turn. Add that up across a session and the cost is not the length of the conversation, it is the square of it. You will do the addition by hand, which is the only way I know to make that number feel real.

Twenty problems again. Bring a pen.

1. The Context Window

2. Window Occupancy

3. Growth per Turn

4. Remaining Space

5. Turn Budget

6. Retrieval Footprint

7. Retrieval per Turn

8. Search Rounds

9. Window Sizing

10. Truncation

11. Compaction

12. The Compaction Bill

13. Compacted Retrieval

14. The Compaction Threshold

15. Verbatim Tail

16. Stateless API

17. Retrieval Cost

18. Context Tax

19. Quadratic Cost

20. Cost Forecast

Library › Context Problems

1. The Context Window

2. Window Occupancy

3. Growth per Turn

4. Remaining Space

5. Turn Budget

6. Retrieval Footprint

7. Retrieval per Turn

8. Search Rounds

9. Window Sizing

10. Truncation

11. Compaction

12. The Compaction Bill

13. Compacted Retrieval

14. The Compaction Threshold

15. Verbatim Tail

16. Stateless API

17. Retrieval Cost

18. Context Tax

19. Quadratic Cost

20. Cost Forecast

One turn's input grows by a fixed step, so the tenth turn sends ten times what the first did. That much is a straight line. But the bill is not the last turn, it is every turn added together, and adding up a straight line gives a curve. That is why doubling the length of a conversation does not double what it costs to run, it roughly quadruples it. The output never joins that curve. It is the same size every turn, so adding it up stays a straight line, and the input half of the bill runs away from it however dearly the output is priced.

Paid members: the worksheet and its printable PDF are below ↓

Read more

Library › Context Problems

1. The Context Window

2. Window Occupancy

3. Growth per Turn

4. Remaining Space

5. Turn Budget

6. Retrieval Footprint

7. Retrieval per Turn

8. Search Rounds

9. Window Sizing

10. Truncation

11. Compaction

12. The Compaction Bill

13. Compacted Retrieval

14. The Compaction Threshold

15. Verbatim Tail

16. Stateless API

17. Retrieval Cost

18. Context Tax

19. Quadratic Cost

20. Cost Forecast

Adding four numbers is easy. Adding a hundred is not, and a conversation does not stop at four. The input goes up by the same step every turn, and a run of numbers like that has an equation: multiply the number of turns by the next number up, halve it, then scale by the step. Work the small case by hand first, then check the equation against what you got. Two numbers decide the total: k, what one turn adds, and n, how many turns there have been. k is fixed by the agent, so shortening its answers drops every total, while n is the conversation itself, and because it appears twice, doubling it does not double the total.

Paid members: the worksheet and its printable PDF are below ↓

Read more