Self Attention (Shared KV)

Attention series: 6 of 11

Attention Series:

1. QKV Projection

2. Attention Computation

3. Self Attention

4. Cross Attention

5. Self Attention vs Cross Attention

6. Self Attention (Shared KV)

7. Multi-Head Attention

8. Fused QKV (Multi-Head)

9. Single vs Multi-Head Attention

10. Multi-Query Attention

11. Grouped-Query Attention

This is a single attention head that receives shared Key and Value from another head — the building block of Multi-Query Attention. Only the Query projection (Wq) is unique to this head. K and V are computed once elsewhere and reused here, cutting the KV cache cost.

Paid members: open the interactive diagram below ↓