Mixture of Experts (MoE) by Hand ✍️
Calculating AI by Hand: 23 of 28
Library › Calculating AI by Hand ✍️
Mixture of Experts (MoE) by Hand ✍️
Reinforcement Learning with Human Feedback (RLHF) by Hand ✍️
Mixture of Experts (MoE) lets a model grow large while staying efficient. Instead of every parameter processing every token, a gate network routes each token to a small subset of expert sub-networks, so only a fraction of the parameters run per token. Mistral's Mixtral brought sparse MoE into mainstream open LLMs. Here is a hands-on exercise to build the intuition by hand.
Setup
Step 1 of 6: Inputs
The MoE block received two tokens (blue, orange).
Route Token X1
Step 2 of 6: Gate → Expert 2
Gate Network processed X1 (blue) and determined Expert 2 should be activated.
Step 3 of 6: Expert 2
Expert 2 processed X1 (blue).