Grouped-Query Attention

Attention series: 11 of 11

Attention Series:

1. QKV Projection

2. Attention Computation

3. Self Attention

4. Cross Attention

5. Self Attention vs Cross Attention

6. Self Attention (Shared KV)

7. Multi-Head Attention

8. Fused QKV (Multi-Head)

9. Single vs Multi-Head Attention

10. Multi-Query Attention

11. Grouped-Query Attention

Grouped-Query Attention (Ainslie et al., 2023) is the middle ground between Multi-Head Attention and Multi-Query Attention. Instead of giving every head its own K and V (MHA) or sharing one K and V across all heads (MQA), GQA splits heads into groups. Each group shares one set of K and V projections.

Paid members: open the interactive diagram below ↓