​This time, I’ll cover four frontier open-source models, two from the US and two from China. I’ll calculate the math and sketch the architectures by hand ✍️.

​We start with Gemma 4, released by Google DeepMind just a few weeks ago (April 2). This model is the perfect baseline: it runs on a phone, shares its recipe with Gemini 3, and its design choices show exactly which transformer techniques have finally matured into the modern stack. From there, we head to the frontier.

Date: Thursday, May 7

Time: 10am (Pacific Time)

Upcoming Seminars

​Week 2 — Qwen 3.5 (Alibaba, China): long-context scaling techniques and attention optimizations improving practical context length.

​Week 3 — Nemotron 3 Super (NVIDIA, US): large-scale transformer systems, with emerging directions toward state-space alternatives like Mamba.

​Week 4 — DeepSeek-V4 (DeepSeek, China): a 1.6T-parameter MoE with Compressed Sparse Attention for 1M-context and three test-time reasoning modes, advancing chain-of-thought at the open-source frontier, released just in-time for this special seminar series. 😉

Previous Seminars

• 🔥 PPO→DPO→GRPO→Rubrics (2/26/2026)

• 🔥 OpenClaw - 12 Stages of Evolution from the Transformer (2/19/2026)

• 🔥 Transformer - Six Levels of Understanding (2/12/2026)

• Meta Superintelligence Labs vs Facebook AI Research (2/5/2026)

• 9 AI Eval Formulas (1/29/2026)

• Google Ironwood TPU: From Bits to HBM (1/22/2026)

• How AWS Uses Small Models Learn Tool Use (1/15/2026)

• Attention (1/15/2026)

• DeepSeek’s Manifold-Constrained Hyper Connection (mHC) (1/8/2026)

• Introduction to Generative AI (1/8/2026)

• Gated Attention (NeurIPS 2025 Best Paper) (12/16/2025)

One important fact about multi-head attention: it has the same parameter count as single-head attention. The difference is purely structural — same total Wqkv weights, partitioned into smaller q–k–v triples.

Look at the two diagrams below. Both Wqkv matrices have the same height — same number of weight rows, same number of parameters. What changes is how that single tall block is sliced.

• Left. One head. The full Wqkv produces one big QKV: a tall Q (36 rows), a tall K, a tall V. One scoring computation runs over those full-width tensors.

• Right. 3 heads. The same-height Wqkv is sliced into 3 smaller q–k–v triples — each 12 rows tall. 3 scoring computations run in parallel, each a thinner version of the left.

The compute trade-off — kind of. Same Wqkv weights. Multi-head runs the attention scoring `S = Kᵀ × Q` once per head, so the dot-product count multiplies by H.

• Single-head: seq × seq = 80² = 6400 dot products

• Multi-head: seq × seq × H = 80² × 3 = 19200 dot products (3×)

But each multi-head dot product is narrower — its inner dimension is `head_dim` instead of `H × head_dim`. So when you count actual scalar multiplications, the totals are equal:

• Single-head: seq² × (H × head_dim) = 80² × 36 = 230400

• Multi-head: seq² × H × head_dim = 80² × 3 × 12 = 230400

Same FLOPs. Multi-head buys you H independent attention patterns at no extra weight cost and no extra arithmetic cost — it's the same total compute, sliced into H finer-grained heads.

Two attention mechanisms, side by side. Both project X into queries; both compute attention via S = Kᵀ × Q and F = V × A. The only difference is the source of K and V.

Self attention uses X for everything. Q, K, and V all come from projecting X. Each X token attends to every other X token. The score matrix S is square — 128 × 128.

Cross attention uses X for queries and a second sequence E for keys and values. Each X token attends to every E token instead. The score matrix S is rectangular — 64 × 128.

Notice what's shared and what's not:

• X is the same in both — same 36 × 128 input.

• Q and K share the 16 dimension — that's what makes the dot product Kᵀ × Q valid in either case.

• V dimensions are independent: self-attention uses 12, cross-attention uses 12. The choice doesn't depend on which mechanism you're using; it depends on what output dimension your downstream layer expects.

Library › Models › Fine-Tuning

The Fine-Tuning series

Fine-tuning is how you turn a general-purpose pretrained model into something that actually does your task — and getting it right means knowing which weights to update.

Fittingly, this series is itself a fine-tune: you bring what you already know about basic MLP neural networks, and each lesson specializes that foundation into one fine-tuning technique.

I teach this the way I teach my master's students — through higher-education metaphors. The pretrained network is someone who's finished their bachelor's.

Each of the eight lessons below shows one way to specialize further — retake every subject, refresh just the advanced course, add a certificate, pursue a PhD, invite a private tutor.

Pick any lesson. They form a sequence, but each stands on its own.

1. Weight Update

When a neural network learns, what actually changes? Not the architecture — the shape of the network stays fixed. Not the inputs — those come from outside. What moves is the weights.

Read → https://www.byhand.ai/p/library-models-fine-tuning-weight-update

2. Pretrain vs Fine-Tune

The previous lesson showed W + Δ W = W1 as a single abstract step. That same step shows up in two very different settings — and the setting is what separates pretraining from fine-tuning.

Read → https://www.byhand.ai/p/library-models-fine-tuning-weight-fine-tune

3. Full Fine-Tuning

In the previous lesson, fine-tuning meant updating one weight matrix. A real network has many — three layers in this example, billions of parameters in a production model. What does fine-tuning look like when you update all of them?

Read → https://www.byhand.ai/p/library-models-fine-tuning-full-fine-tuning

4. Freezing Layers

In the previous lesson, full fine-tuning reviewed every prerequisite — Linear Algebra, Probability, Advanced ML — to refresh each subject with the latest topics. Effective, but exhausting.

Read → https://www.byhand.ai/p/library-models-fine-tuning-frozen-layers

5. Linear Probe

In Freezing Layers, we kept the foundational prerequisites fixed and refreshed only the advanced course on top — W3. But even refreshing one course is still a whole course. W3 is a full 40 × 40 matrix — 1600 weights to update — weeks of lectures, assignments, and exams to work through. What if we don't re-take any existing course at all, and instead pick up a single new one-credit certificate — the kind you can finish in a month?

Read → https://www.byhand.ai/p/library-models-fine-tuning-linear-probe

6. Feature Extraction + Head

A feature head is a small trainable MLP bolted onto a frozen pretrained backbone. Think of it as pursuing a PhD on top of a master's degree. The master's — your pretrained backbone — stays exactly as it was, with no review. You aren't re-taking Linear Algebra or Probability; you're building something specialized on top of it: the PhD adds its own coursework, its own nonlinearity, and its own thesis layer.

Read → https://www.byhand.ai/p/library-models-fine-tuning-feature-head

7. Adapter Layers

Houlsby et al., 2019 proposed a different strategy: instead of choosing which layers to freeze, insert small trainable modules — adapters — between every frozen layer.

Read → https://www.byhand.ai/p/library-models-fine-tuning-adapter-layers

8. LoRA

Adapters showed that a low-rank bottleneck can specialize a frozen layer with far fewer parameters than a full ΔW.

Read → https://www.byhand.ai/p/library-models-fine-tuning-lora-intro

Library › Models › Fine-Tuning › LoRA

Adapters showed that a low-rank bottleneck can specialize a frozen layer with far fewer parameters than a full ΔW.

LoRA (Hu et al., 2021) takes the same low-rank idea and makes two changes:

1. Drop the nonlinearity.

"Down-project + ReLU + up-project" collapses into a single low-rank matrix product: Δ W = B A.

2. Add to the weight matrix, not between layers.

The effective weight is W' = W + Δ W = W + B A.

At inference, Δ W disappears — meaning it gets folded directly into W.

Below, the left column shows an adapter (a new module in the forward path). The right column shows LoRA: same low-rank idea, but expressed as a correction to the weight matrix.

The benefit of dropping the nonlinearity is that B and A can be merged into a single low-rank ΔW matrix at inference time. During training, B and A live alongside W as extra matrices.

You compute the weight update in two separate paths so gradients can flow into both matrices during training.

At inference you merge them once and for all: compute the combined weight, ship it, and discard the individual matrices. The resulting model has the exact same architecture and inference cost as the pretrained one. Same shape, same number of matrix multiplications, no extra latency, no extra memory.

An adapter can't do this. Its down-project–ReLU–up-project module stays in the forward path forever, and every inference call pays for those extra operations.

This is why LoRA is the default for large models today. You can keep thousands of tiny (B, A) pairs — one per fine-tuning task, a few megabytes each. Swap them in and out of a shared frozen backbone without changing its runtime shape.

← Previous: Adapter Layers

Paid subscribers: open the interactive diagram below ↓

Read more

Library › Models › Fine-Tuning › Adapter Layers

Houlsby et al., 2019 proposed a different strategy: instead of choosing which layers to freeze, insert small trainable modules — adapters — between every frozen layer.

Each adapter has a down-projection (d → r), a nonlinearity, and an up-projection (r → d). The bottleneck dimension r is much smaller than d, so each adapter adds very few parameters. The pretrained weights stay completely frozen.

With rank r = 8 and hidden size d = 64, each adapter has only 2 × r × d = 1024 trainable parameters.

Across the two layers of this network:

• The first layer projects from input to hidden, so W₁ has d × input = 2048 weights.

• The second layer has d × d = 4096 weights.

• Full fine-tuning would update all 6144 of them.

• Adapter fine-tuning adds only 2 × r × d = 1024 new trainable weights — about a 3× reduction.

← Previous: Feature Extraction + Head | LoRA →

Paid subscribers: open the interactive diagram below ↓

Try the interactive version ↗

Library › Models › Fine-Tuning › Feature Extraction + Head

A feature head is a small trainable MLP bolted onto a frozen pretrained backbone. Think of it as pursuing a PhD on top of a master's degree. The master's — your pretrained backbone — stays exactly as it was, with no review. You aren't re-taking Linear Algebra or Probability; you're building something specialized on top of it: the PhD adds its own coursework, its own nonlinearity, and its own thesis layer.

This is one step richer than a linear probe, which bolts on a single linear projection — like earning one certificate after the master's. Certificates are quick and cheap, but they can only form linear combinations of subjects you already know. A feature head, with multiple trainable layers, can form nonlinear connections and capture task-specific structure the probe can't.

And it's a tighter budget than Freezing Layers, where we still refreshed a top layer's weights directly. Here, every weight in the backbone is permanently frozen — no ΔW anywhere. All trainable parameters live in the head.

In the diagram, the backbone (gray, dashed) extracts features without changing. The head (red border) is the trainable MLP: a nonlinear mapping from those frozen features to task predictions. This is the standard recipe in computer vision — take a pretrained ResNet or ViT, freeze it, and train a task-specific head on top.

How much did we save?

Full fine-tuning would update every weight in both the backbone and the head — W1 through W5:

64 × 32 + 2 × 64 × 64 + 20 × 64 + 10 × 20 = 11720

parameters.

Freezing the backbone leaves only the head trainable — W4 and W5:

20 × 64 + 10 × 20 = 1480

parameters. That's about 7.92× fewer weights to train, and — because the backbone is shared — the same frozen model can support dozens of downstream tasks, each with its own tiny PhD head.

The next lesson takes the idea in a different direction: instead of bolting a head onto the end, we'll sprinkle small trainable modules throughout the network — the adapter layers.

← Previous: Linear Probe | Adapter Layers →

Paid subscribers: open the interactive diagram below ↓

Read more

Library › Models › Fine-Tuning › Linear Probe

In Freezing Layers, we kept the foundational prerequisites fixed and refreshed only the advanced course on top — W3. But even refreshing one course is still a whole course. W3 is a full 40 × 40 matrix — 1600 weights to update — weeks of lectures, assignments, and exams to work through. What if we don't re-take any existing course at all, and instead pick up a single new one-credit certificate — the kind you can finish in a month?

That's a linear probe. The master's degree — your pretrained network — stays exactly as it was, untouched. On top of it, you add one small linear layer: no homework for the old material, just a single combination of subjects you already know, tuned for the new task.

That new layer is Wn — a thin 10 × 40 matrix, just 400 weights. That's 4× fewer trainable parameters than refreshing W3 alone. Everything in W1, W2, and W3 is permanently frozen.

Why bother with something so tiny? Because it's also a probe: if one thin linear layer can already do the downstream task well, that tells you the pretrained features are rich enough on their own. If it can't, you know you need something bigger — and the pretrained features still got you most of the way.

A one-credit certificate is quick and cheap, but it can only form linear combinations of what you already know. The next lesson adds a whole PhD on top — a multi-layer head that can form nonlinear connections too.

← Previous: Freezing Layers | Feature Extraction + Head →

Paid subscribers: open the interactive diagram below ↓

Read more

Library › Models › Fine-Tuning › Freezing Layers

In the previous lesson, full fine-tuning reviewed every prerequisite — Linear Algebra, Probability, Advanced ML — to refresh each subject with the latest topics. Effective, but exhausting.

Then you realize something. The prerequisites haven't actually changed that much. Linear Algebra is still Linear Algebra; the matrix decompositions you learned still hold. Probability is still Probability; the distributions and Bayes' rule haven't moved. Almost all the new material — the new ideas, the recent discoveries — lives in the advanced layer at the top.

That's freezing layers: keep the prerequisite layers fixed at their pretrained state, and only update the advanced one. In the diagram below, W1 and W2 — the foundational prerequisites — stay frozen. Only W3 — the layer closest to your task-specific output — gets a ΔW.

Its update flows through the equation:

W'3 = W3 + Δ W3

The right column shows the network after the update. W'1 and W'2 are identical to W1 and W2 — nothing changed there. Only W'3 is new.

How much did we save?

Full fine-tuning would train all three layers:

40 × 30 + 40 × 40 + 40 × 40 = 4400

parameters.

Freezing layers 1 and 2 leaves only:

40 × 40 = 1600

parameters trainable. The other 2800 parameters are skipped — no ΔW, no gradient, no storage.

For a real-world model with billions of parameters, freezing the first 80% of the network can shrink fine-tuning cost by an order of magnitude.

There's a second, subtler benefit. The frozen prerequisites can't drift, so the model can't forget its foundations — Linear Algebra stays Linear Algebra. This problem has a name: catastrophic forgetting — the new task overwrites old knowledge. Frozen layers prevent it simply by not letting those weights move.

The next lesson takes the idea even further: freeze all the prerequisites and bolt a brand-new advanced course on at the end.

← Previous: Full Fine-Tuning | Linear Probe →

Paid subscribers: open the interactive diagram below ↓

Read more

Library › Models › Fine-Tuning › Full Fine-Tuning

In the previous lesson, fine-tuning meant updating one weight matrix. A real network has many — three layers in this example, billions of parameters in a production model. What does fine-tuning look like when you update all of them?

That's full fine-tuning: continue training every weight in the pretrained network on your new task. Every layer's W gets its own ΔW. Nothing is frozen — every parameter is in play.

Think of an MLP as a chain of prerequisites leading to an advanced course. Layer 1 might be Linear Algebra, layer 2 Probability, layer 3 Advanced Machine Learning — each one building on what came before.

Fine-tuning is what happens during graduate study: the foundations are already there from undergrad, so you're not re-learning. Full fine-tuning is reviewing every prerequisite to see what new topics have appeared and what discoveries the field has made since the last time you sat through them. Effective — but exhausting.

This diagram shows the same three-layer MLP twice, side by side.

On the left, the pretrained network runs on input X: three weight matrices W₁, W₂, W₃, each followed by a ReLU activation.

In the middle, one update equation per layer:

W'i = Wi + Δ Wi

On the right, the same network after the update — same shape, same flow, but every weight is now its updated version W'i. The arrows trace each Wi flowing down into its update, and each W'i flowing back up into its layer on the right.

Full fine-tuning gives the model the most freedom to specialize. Every parameter can move — and every parameter that can move must be stored.

In this small example, the trainable count is 40 × 30 + 40 × 40 + 40 × 40 = 4400 values. For a production model with billions of parameters, fine-tuning for one task means saving a billion-parameter ΔW set. Ten downstream tasks means ten complete model copies.

But not every prerequisite needs revisiting. The further you go back in the chain, the less the material has changed since pretraining — the linear-algebra basics under your computer-vision course are largely the same as they ever were. The next lesson does exactly that: freeze the prerequisites that haven't moved, and only refresh the advanced one closest to your specialization.

← Previous: Pretrain vs Fine-Tune | Freezing Layers →

Paid subscribers: open the interactive diagram below ↓

Try the interactive version ↗

Library › Models › Fine-Tuning › Pretrain vs Fine-Tune

The previous lesson showed W + Δ W = W1 as a single abstract step. That same step shows up in two very different settings — and the setting is what separates pretraining from fine-tuning.

In pretraining, the dataset D is massive and general-purpose — say 200 examples per batch, multiplied across billions of steps. Each ΔW is tiny, but the cumulative effect is a weight matrix that has absorbed broad patterns of the data: W is now the model's foundational knowledge.

In fine-tuning, the dataset is much smaller — maybe just 12 task-specific examples. You don't want to throw out everything W has learned, and 12 examples is nowhere near enough to relearn it from scratch anyway. You want to add a small specialization on top.

I once told my master's students, jokingly, "You are all fine-tuning here." You don't go back to re-learn calculus — you build on the undergraduate foundation and specialize. Fine-tuning does the same for a model: keep the foundational W, and add a small correction on top.

Concretely, freeze W and run task data X through it:

F = W × X

The output F is reasonable but generic — W has never seen your task, so the prediction is whatever a general-purpose model would say. To specialize, we add an adjustment — a new matrix ΔW, same shape as W (40 × 40), trained only on the task data while W stays put:

W1 = W + Δ W

Now the forward pass uses the fine-tuned weight:

F1 = W1 × X = (W + Δ W) × X

ΔW is the entire artifact of fine-tuning — the per-task adjustment learned from 12 examples, layered on top of months or years of pretraining.

← Previous: Weight Update | Full Fine-Tuning →

Try the interactive version ↗

Library › Models › Fine-Tuning › Weight Update

When a neural network learns, what actually changes? Not the architecture — the shape of the network stays fixed. Not the inputs — those come from outside. What moves is the weights.

A training step takes the current weight matrix W0, computes a small nudge Δ W from a batch of examples, and adds the two together to produce the new weights:

W1 = W0 + Δ W

This single equation — add a correction to the existing weights — underlies everything in this chapter. Pretraining, full fine-tuning, frozen layers, adapters, LoRA: they're all variations on who gets a ΔW and how big that ΔW is allowed to be.

The diagram below shows it for one layer. Before the step, the forward pass uses the old weight:

F0 = W0 × X

After the step, the same input flows through the new weight:

F1 = W1 × X

ΔW always has the same shape as W0 — for a 40 × 30 weight matrix, ΔW also has 1200 values, and every one of them is a free parameter learned by gradient descent.

Repeat this step billions of times across a massive general-purpose corpus, and you get a pretrained model: weights that encode broad patterns of language, code, and reasoning.

The same W + Δ W step can serve very different goals. When the dataset is huge and the goal is to absorb general patterns, we call it pretraining. When the dataset is small and the goal is to specialize an already-pretrained model, we call it fine-tuning. The next lesson puts those two side by side.

Pretrain vs Fine-Tune →

Try the interactive version ↗

As a paid member of the AI by Hand Academy, you now have full access to the Library’s interactive models, exercises, and calculations that bring AI concepts to life.

…

Read more

I just completed two new workbooks: Matrix Multiplication and Linear Layer, each has 25 exercises.

Linear Layer

👉 Check out these two workbooks

(Available as a limited-time free preview for newsletter subscribers.)

~ Prof. Tom Yeh

This is a proof-of-concept prototype with the first 15 exercises of Chapter 1: Dot Product. You fill in the blanks, pick from multiple choices, and check your answers — all in your browser.

👉 Give it a try

Let me know what you think.

I still believe pen and paper is the best way to learn math. There’s something about the focus and concentration that a blank page demands. But many of you have asked for an interactive option, and I think there’s real value in being able to test yourself and get immediate feedback.

~ Prof. Tom Yeh

The first collection covers attention with seven diagrams in one learning path:

1. QKV Projection: wher…

Read more

Self-attention is the core mechanism behind Transformers, yet it’s almost always explained through equations and code. What gets lost is the spatial relationship between matrices — how dimensions propagate, why certain shapes must match. You can’t see that Q and K must be the same size but V doesn’t have to be, or that the score matrix S and attention matrix A are always square, just by reading S = softmax(Kᵀ Q / √d).

Think you already know self-attention inside and out and don’t need this tool? Here are three questions to test your knowledge:

1. Does increasing the model size (X’s height) change the shape of the score matrix S?

2. Does increasing the sequence length (X’s width) change the size of Wq, Wk, or Wv?

3. Does increasing the key dimension (K’s height) change the size of the attention matrix S = QKᵀ?

Use the sliders to change these values and observe what happens before you read the answer key below.

[space intentionally left blank so you don’t read the answers right away]

Answers:

Model Size

Q: Does increasing the model size (X’s height) change the shape of the score matrix S?

A: No. Drag the Model Size slider and watch: X gets taller, the weight matrices Wq, Wk, Wv grow wider — but S stays exactly the same size. The score matrix S is always sequence length × sequence length, regardless of model size. Model size controls how many features each token carries, but the attention pattern — which tokens attend to which — is purely determined by sequence length and key dimension. You can see nothing in the right side is changing.

Sequence length

Q: Does increasing the sequence length (X’s width) change the size of Wq, Wk, or Wv?

A: No. Increasing sequence length does not change any weight matrix. The number of trainable parameters stays the same. That’s why when you hear a model increases its context length from 5K to 5M, the “size” of the model does not change. However, the time to compute the attention matrices (S and A) grows quadratically with sequence length.

Key Dimension

Q: Does increasing the key dimension (K’s height) change the size of the attention matrix S = QKᵀ?

A: No. As you increase the key dimension, query and key vectors grow longer, but the attention score matrix (S) and attention weight matrix (A) do not change size. The attention matrix is always sequence length x sequence length regardless of key dimension. However, each value in it — a dot product — takes more to compute, because the key and query vectors are longer.

Give Self Attention Flow a try. Resize any matrix in the diagram and watch the dimensions “flow” to other matrices in the self-attention mechanism.

Matmul’s Dimensions are Confusing

If you’ve worked with matrix multiplication or matmul, you know the rule: if W’s shape is (m, n) and X’s shape is (n, p), then F = WX’s shape must be (m, p). The inner dimensions must match, the outer dimensions give you the result. It’s simple enough to state. But in practice, I struggled as a student in the past and I’ve watched my students struggle stumbling over this again and again. Which one is the inner dimension? Is it the columns of the first or the rows of the second?

And when you chain multiple multiplications together, it gets worse.

Consider W₅(W₄(W₃(W₂(W₁X)))). What happen if we increase W₃’s width by 40?

Consider ((((WX₁)X₂)X₃)X₄)X₅. What happen if X₂’s height increases from 150 to 300?

This is when my brain starts to hurt. But remember these questions for now, because we will revisit them using MatmulFlow later.

The problem with the standard notation, I think, is it is too abstract. You’re staring at letters — W, X, F, m, n, p — trying to keep track of which belongs to which. There’s nothing to see.

Just Shift the Second Matrix Up

Several years ago, I came up with a visual method that makes the shapes of matrix multiplication obvious at a glance. This method is extensible to complex chains of multiplication. No memorization needed. Here’s how it works.

Take your two matrices, W and X, and represent them as rectangles whose width and height correspond to the number of columns and rows. Now take the second matrix X and shift it up, placing it above and to the right of W.

Look at what happens. The right edge of W lines up with the bottom edge of X. That alignment is the inner dimension — the one that has to match for the multiplication to be valid. And the result F = WX simply fills in the remaining rectangle below X and to the right of W. It takes its height from W and its width from X.

The constraint that felt abstract — “the inner dimensions must match” — becomes geometric and obvious. If W’s right edge doesn’t line up with X’s bottom edge, the multiplication can’t happen. You don’t need to remember the rule. You can see it.

Chain of Matmuls

This visualization method extends naturally to chains of matrix multiplications. Each layer multiplies the output F of the previous layer by a new weight matrix W.

In MatmulFlow, stacking layers vertically means multiplying on the left. The output F₁ of the first layer feeds down to become X₂ for the next layer, giving F₂ = W₂ (W₁X₁).

The same visual logic applies in the other direction. Stacking layers horizontally means multiplying on the right. The output F₁ feeds across to become W₂ for the next layer, giving F₂ = (W₁X₁)X₂.

Let’s revisit W₅(W₄(W₃(W₂(W₁X)))) and (((W₁X₁)X₂)X₃)X₄)X₅. If we apply the same logic to visualize them, we can immediately see how they are different. The first expression stacks vertically, while the second expression stacks horizontally.

Resizing a Matrix

Back to our questions earlier, what happen if we increase W₃’s height by 40?

What happen if X₂’s height increases from 150 to 300?

Give MatmulFlow a try. Resize any matrix in the chain and watch the dimensions “flow” to other matrices in the chain in both directions.

Mastering Matrix Multiplication

If you want to commit yourself to mastering matrix multiplication, the paper version of my Matrix Multiplication Workbook was recently released on Amazon. Click this link to take a look.

If you are a member of the AI by Hand Academy, just a reminder that you already have access to the electronic version of this workbook.

SwiGLU has become a standard activation in many frontier large language models because it increases expressive power without adding much structural complexity. It combines GLU-style input-dependent gating with Swish-style amplification: the gate allows features to be selectively passed or suppressed based on the current input, while the multiplicative Swish behavior lets strong signals be amplified rather than merely filtered.

In contrast, GLU uses the same dynamic gating mechanism to allow identical features to have different activation values, but it does not amplify them.

Furthermore, Swish uses a static gating mechanism; the activation values are identical because the features are identical, independent of the input.

Going even further back, ReLU represents the extreme case of static activation: a hard cutoff at zero with no gating signal at all. Every neuron follows the same fixed rule, which highlights why later designs moved toward input-dependent, learnable gating.

Calculation

SwiGLU begins with an input vector x that is sent through two learned linear transformations in parallel. The first transformation, W1, produces the feature vector W1x. The second transformation, W2, produces a gate logit W2x. This gate logit is passed through a sigmoid function to obtain a gate value between 0 and 1, which can be interpreted as the “open percentage” of the gate. Up to this point, the mechanism matches GLU: the feature vector is modulated by a learned gate derived from the same input.

The distinctive step in SwiGLU is that the gate does not stop at σ(W2x). Instead, it multiplies the sigmoid output by the original gate logit, forming (W2x) · σ(W2x). This creates an amplified, Swish-style gate that can smoothly scale features rather than only attenuate them. Finally, the output is computed as W1x ⊙ ((W2x) · σ(W2x)), meaning the feature vector is element-wise multiplied by this amplified gate. The result is a dynamic, input-dependent transformation that can both suppress and amplify features in a smooth and expressive way.

For comparison, below is the diagram showing the computation of GLU. In GLU, the gate logit is passed through a sigmoid and directly used to modulate the feature vector. There is no additional multiplication with the original gate logit—so no amplification step. The gate strictly scales features between 0 and 1, meaning it can only attenuate or pass them through, but not amplify them.

Batch

Let us scale the calculation to a batch of six examples, x1 to x6. Each input vector has dimension 3, and the output dimension is 2. The two linear transformations are applied to every example in the batch, producing a 2-dimensional feature vector and a 2-dimensional gate logit for each input. After applying the sigmoid (and the additional amplification step in SwiGLU), each of the two output features receives its own independently scaled gate value. In other words, gating happens per feature and per example, so every feature channel can be modulated differently across the six inputs.

Application

The most common application of SwiGLU is in the feed-forward network (FFN) of the Transformer architectures behind frontier models, where it replaces the traditional ReLU activation used in the original Transformer paper. The FFN typically consists of two linear layers: the first expands the hidden dimension into a higher-dimensional space (for example, from 2 to 4), and the second projects it back to the original model dimension (from 4 back to 2), as illustrated in this example. In this expanded space (dimension 4), the feature vector, the gate logit, and the resulting gated activation vector all share the same dimensionality, allowing element-wise modulation before the final projection back to the model size.

Below is the layout of the feed-forward layer in the original Transformer paper for comparison. Notice the differences. First, the activation function is ReLU, a fixed, predefined nonlinearity. Second, the first projection is a single linear transformation—there is no parallel split into separate feature and gate branches. In other words, the original FFN performs a simple Linear → ReLU → Linear sequence, without any dynamic gating mechanism.

Implementation: Standalone Layer

PyTorch

Excel Emulation

Implementation: Feed-Forward Layer in Frontier Models

Below is a practical implementation of SwiGLU inside a feed-forward network. Instead of using two separate linear layers for the feature and gate branches, we combine them into a single “up” projection that outputs twice the hidden dimension. his allows the model to perform one larger matrix multiplication instead of two smaller ones. In modern GPU kernels, a single larger matmul is typically more efficient After this combined projection, the tensor is split into feature and gate branches, the gating and amplification are applied, and the result is projected back to the model dimension. This structure reflects how SwiGLU is implemented in most frontier Transformer models for both performance and scalability.

PyTorch

Excel Emulation

Excel Blueprint

🔗 View the Excel Blueprint Online

(Limited time preview)

This is part of the Essential AI Math Blueprints series. Get the full series by joining the AI by Hand Academy.

What becomes possible when we stop comparing AI to ourselves? The answer just might be the difference between automation and transformation. ~ Jaime Teevan

Jaime Teevan is the Chief Scientist and Technical Fellow at Microsoft. Let me compare myself to her. I am always two steps behind. I was born two years later than her. I entered MIT two years later than her. I got my PhD two years later than her. I have two children, two fewer than her four. 😁

As I was comparing myself to Jaime, I was also reminded how similar we were back at MIT. We shared the same thesis committee member, Prof. Rob Miller. We shared the same application focus—search. Jaime studied personalized search, whereas I studied image-based search. We shared the same duty to cook lunch for more than a hundred PhD students.

Fast-forward to 2020, Jaime started the Future of Work initiative at Microsoft. I received a new grant from NSF to study AI’s impact on the Future of Work. In retrospect, we both underestimated the impact, another similarity.

In 2025, Jaime, along with her team of editors and contributors from inside and outside of Microsoft, published the fifth New Future of Work report. Our team also published the third paper from my Future of Work NSF project, once more two steps behind.

p.s., I highly recommend you all to read this report: https://aka.ms/nfw2025

When I compare myself to Jaime, my human friend, my connection with her grew stronger. When I compare myself to AI, AI just knows more, writes better, counts faster, drives safer, draws nicer. But there’s no soul behind it. It simply has access to a lot more data. I cannot feel connected with AI. AI isn’t human.

Indeed, AI isn’t human. Last week Jaime wrote an article about this point. She kindly agreed to let me share this article with you.

AI Isn’t Human

written by Jamie Teevan (Link to her original article on LinkedIn)

Early lessons from our first close encounter with a very different intelligence.

It’s tempting to think of AI as being “like us.” Generative AI models speak our language, respond fluidly, and feel conversational, so we naturally anthropomorphize them, going so far as to frame them as virtual colleagues or even friends. But leaning too hard on a human metaphor for AI minimizes what’s uniquely human and constrains our imagination as to how we can best use it. After all, we already know how to collaborate with people. The real opportunity ahead is to explore what’s possible when we work with something that is fundamentally different.

AI doesn’t just mimic human intelligence, it introduces new capabilities that can transform how we learn, create, and collaborate. Here are five ways AI is different from humans, and why those differences matter.

1. Scale of Intelligence

Humans excel at nuanced judgment, but we’re bounded by time and attention. Psychologist George A. Miller famously found that most people can actively hold only around five to nine independent numbers in their head at once. No person can read thousands of survey responses or synthesize millions of product reviews in an instant. AI models have no such limits. Consider restaurant reviews. Where you once had to read through each review one by one to get a feel for a restaurant, AI can now surface patterns across hundreds of reviews, amplifying insights or perspectives that you might otherwise have missed. This isn’t just about efficiency; it’s a shift toward collective intelligence.

Implication: Use AI to aggregate insights from employees, customers, and partners, turning fragmented feedback into actionable strategy.

2. Speed of Ideas

Human communication is often asynchronous and inherently slow. Even in fast-moving organizations, feedback loops take time. We wait for inspiration to strike, and then we wait again for others to weigh in. AI, on the other hand, responds instantly, offering ideas on demand and immediate feedback. You can brainstorm a dozen directions in seconds, and then immediately test and refine them. AI becomes both your creative partner and your first reviewer, helping you move from blank page to viable concept without waiting for the next meeting or email reply. It’s now possible to replace the “waiting game” of traditional collaboration with immediate, context-aware input. Late at night or across time zones, you can get suggestions, iterate, and improve, all without waking anyone up.

Implication: Redesign the cadence of work. Route first drafts and first feedback to AI by default, then bring people in where judgment is needed.

3. Continuity of Interaction

People carry context from one conversation to the next. That continuity is valuable. It’s how we build relationships and shared understanding. But continuity can also constrain us. We may hesitate to ask the same question twice or to explore radically different directions for fear of seeming inconsistent. If I make a bad argument in this article, you can’t just wipe it from your mind and let me give it another go. With AI, it’s possible to erase history and start over – there’s no penalty to being curious. Want to explore a different framing or simulate an alternative scenario? Reset and rerun. AI won’t judge, remember, or cling to the past. This ability to start an interaction afresh makes AI a powerful tool for creative exploration.

Implication: Use AI to model multiple futures. Instead of debating one strategy, generate and compare several, stress-testing assumptions before committing.

4. Externalization of Cognition

Human reasoning is opaque. We infer intentions and logic from what others say or do, but we can’t scroll through someone’s thought process. That opacity has social value. It fosters trust, privacy, and connection. But it also limits how we learn from one another. AI makes this invisible process visible. You can review chain-of-thought, tweak prompts, and trace how changes influence outcomes. Ask AI to provide feedback on an email from the perspective of an engineer, a marketer, or a customer, and watch how the responses shift. This transparency makes it easier to experiment, iterate, and learn. What’s more, AI’s thinking isn’t just visible, it’s portable. Prompts, responses, and model configurations can be saved, reused, and passed between people, turning not just outputs but the reasoning behind them into something others can pick up and use.

Implication: Make thinking shareable. Capture and circulate high-impact prompts, workflows, and model setups so your organization learns faster than any one person can.

5. Ownership of Outcomes

Humans operate inside systems of accountability. We learn through discomfort, where mistakes sting, reputations are at stake, and judgment is shaped by social and organizational norms. AI, on the other hand, can operate boldly and at scale, but it can’t be held responsible. It can’t lose its job, face consequences, or feel the weight of a bad decision. This difference matters. As AI takes on more execution, the human role shifts toward deciding when to trust its output, when to override it, and how to design the surrounding guardrails. The risk isn’t that AI will make “wrong” choices but that it has no skin in the game. That’s why human oversight matters. We need to design AI workflows that align AI outputs with organizational values, ensuring creativity doesn’t come at the cost of compliance or ethics. The opportunity is to pair AI’s fearless exploration with human oversight to open up previously unimaginable paths that people can turn into direction.

Implication: Treat AI as a bold contributor, but not a responsible one. Assign clear human ownership for decisions to ensure AI operates in alignment with your organization’s values.

The Leadership Opportunity

AI isn’t here to replace human intelligence; it’s here to expand it in ways we’ve only begun to imagine. AI will inevitably change your organization, and it’s your ability to capitalize on what makes it different that will determine how meaningful that change is.

So stop using AI to do the things humans already do well, and start using it to scale insight, accelerate iteration, explore without fear, and externalize expertise. More than a new tool, AI is a new layer of organizational cognition, one that can capture, share, and amplify how your teams think.

The next time you catch yourself saying, “AI is like a person,” pause. Then ask: What becomes possible when we stop comparing AI to ourselves? The answer just might be the difference between automation and transformation.

~ Jaime Teevan, Chief Scientist, Microsoft

New Future Of Work Report 2025

4.67MB ∙ PDF file

Download

Download