Library › Seminar Series 2026
This Foundation seminar focused on rebuilding intuition for attention, starting from the famous paper Attention Is All You Need, which introduced the Transformer and reshaped how modern language models are built.
I began by framing tokens as vectors, units of information, and explained attention as a way to produce new tokens by combining existing ones with different weights. Each output token is simply a weighted mixture of earlier tokens, not a mysterious operation.
We then walked through scaled dot-product attention step by step. Tokens are projected into queries, keys, and values; dot products compare what a token is looking for with what others can provide; scaling keeps the numbers stable; and softmax turns scores into probabilities. I emphasized that matrix multiplication is just many dot products happening in parallel.
From there, we extended the idea to multi-head attention. Instead of one comparison space, multiple heads run in parallel, each learning a different way to relate tokens. Their outputs are concatenated and projected back to the original dimension so the next layer can proceed cleanly.
The main takeaway was simple: attention looks intimidating on paper, but once you unpack it, it’s a clear and systematic way to mix information across tokens, the core idea that makes Transformers work.
Recording & Workbook
Limited-time free preview: the full recording and Excel workbook are open to everyone for now.
Next:
5. Google Ironwood TPU: From Bits to HBM (Jan 19, 2026)
The "Frontier: Small Language Model & Agentic Tool Calling" session is missing.
Will there be a record for SLM part?