"Expert Choice" Mixture of Experts (MoE)

Frontier by Hand ✍️

Thanks for becoming a paid subscriber. This is where you’ll get my newest AI by Hand ✍️ worksheets before anyone else. This first release takes you into Expert-Choice Mixture of Experts (MoE) and shows you how this method contrasts with traditional MoEs that use token choice.

Q: Why Expert Choice routing?
A: Because traditional MoE (Token Choice) suffers from load imbalance—some experts get overloaded with tokens while others stay idle—wasting capacity.

Q: How does Expert Choice fix this?
A: By letting experts select their top tokens, it prevents any expert from being overloaded with tokens and keeps computation balanced.

Q: Who invented Expert Choice routing?

Researchers at Google.

Worksheets

Download

These worksheets are available to AI by Hand Academy members. You can become a member either (1) directly through the Academy or (2) via a paid Substack subscription. Both options provide the same access.