MHA, MQA, GQA, MoE-A: More Attention!

Frontier AI Drawings: 2 of 13

Library › Frontier AI Drawings

1. "Expert Choice" Mixture of Experts (MoE)

2. MHA, MQA, GQA, MoE-A: More Attention!

3. New GPT-OSS Trick to Ignore Tokens

4. MXFP4, FP4, FP8

5. LoRA, Fine-Tune, Pre-Train

6. QLoRA, DoRA, BitFit, NF4 vs INT4

7. KV Cache, Prefill, Decode

8. EmbeddingGemma, MRL, InfoNCE, Embed vs. Decode

9. Inference Batching, Request-vs-Token Level

10. MLP Parallelism: Data, Context, Row, Column, Pipeline

11. RoPE vs PE in QKV Self-Attention

12. RMS, Group, Layer, Batch Norm, Tensor Parallelism

13. Qwen 3

More attention for better attention.

This week we zoom out to the bigger picture: how attention itself has evolved.

GQA showing up in GPT-OSS is yet another sign of its wide adoption, mirroring trends we’ve seen in PaLM, LLaMA, and other large-scale models. These are no longer niche optimizations; they’re becoming standard in high-performance architectures.

Drawings

I’ve created four new drawings breaking down the math behind:

1. Multi-Head Attention (MHA) – the original transformer workhorse.

2. Multi-Query Attention (MQA) – shares keys and values across heads for speed.

3. Grouped-Query Attention (GQA) – the middle ground between MHA and MQA, used in GPT-OSS and increasingly in other frontier models.

4. Mixture-of-Experts Attention (MoE-A) – routes attention to specialized experts for scalability.

---

Page 1 of 8

Become a member to access the rest of the drawings.