Cross Attention

Attention series: 4 of 11

Attention Series:

1. QKV Projection

2. Attention Computation

3. Self Attention

4. Cross Attention

5. Self Attention vs Cross Attention

6. Self Attention (Shared KV)

7. Multi-Head Attention

8. Fused QKV (Multi-Head)

9. Single vs Multi-Head Attention

10. Multi-Query Attention

11. Grouped-Query Attention

Cross attention is what happens when the queries and the keys/values come from different sources. In self-attention, every position in a sequence attends to every other position in the same sequence. In cross attention, one source (X) provides the queries and a second source (E) provides the keys and values.

Paid members: open the interactive diagram below ↓