Kimi 3 ~ New Seminar Recording

AI by Hand ✍️ Seminars

Library › Seminar Series 2026

1. Manifold-Constrained Hyper Connections (mHC) from DeepSeek (Jan 9, 2026)

2. How Small Models Learn Tool Use (Jan 11, 2026)

3. Generative AI (Jan 11, 2026)

4. Attention (Jan 15, 2026)

5. Google Ironwood TPU: From Bits to HBM (Jan 19, 2026)

6. 9 AI Eval Formulas You Must Know (Jan 24, 2026)

7. Meta Superintelligence Labs vs Facebook AI Research (Jan 30, 2026)

8. Transformer: Six Levels of Understanding (Feb 13, 2026)

9. OpenClaw Seminar (Feb 23, 2026)

10. PPO → DPO → GRPO → Rubrics (Mar 2, 2026)

11. Gemma 4 (May 7, 2026)

12. Qwen 3.6 (May 21, 2026)

13. Kimi 3 (Aug 6, 2026)

Thank you to everyone who joined the Frontier AI Seminar on Kimi 3.

It has been a while since I gave the last seminar on Qwen. Happy to get together with many of you again to study frontier model math and architecture by hand in Excel!

We are blessed with Nathan Lambert, the author of Interconnects, some of the most authoritative writing on open models, as a special guest. Nathan also wrote the RLHF book which has just been published by Manning.

Here are some of my favorite moments:

• Nathan shared his behind-the-scenes story about his trip to Moonshot AI.

• Nathan's famous dog Phoebe had a surprise appearance on camera.

• I got to finally talk about Linear Transformer line of frontier model advances.

• I got to mention LeBron James in my explanation of the attention mechanism.

Thanks for your feedback!

Before the summer, I did plan to talk about Linear Transformer line and wanted to use NVIDIA Nemotron as the backdrop. However, when Kimi 3 was released, I found an even stronger storyline, one where I can track the evolution much further from Linear Attention, DeltaNet, to the latest Kimi Delta Attention (KDA).

Outline

1. Interview and Discussion

   • Nathan Lambert

   • Moonshot AI Tour

   • Open vs Closed Models

   • Audience Q&A

2. Kimi 3 by Hand

   • Attention

   • Quadratic Attention

   • Inference and KV Cache

   • Local Attention

   • Linear Attention

   • DeltaNet

   • Kimi Delta Attention (KDA)

   • Convolution and Gating

   • Chunkwise Parallel Computation

   • Seq2Seq LSTM

Recording & Workbook

Limited-time free: the full recording and the Excel workbook are open to everyone for now.

⬇ Excel Workbook