Self Attention by Hand ✍️
Calculating AI by Hand: 10 of 28
Library › Calculating AI by Hand ✍️
Self Attention by Hand ✍️
Reinforcement Learning with Human Feedback (RLHF) by Hand ✍️
Self-attention is what enables LLMs to understand context.
How does it work?
This exercise demonstrates how to calculate a 6-3 attention head by hand. Note that if we have two instances of this, we get 6-6 attention (i.e., multi-head attention, n=2).
Goal:
Transform [6D Features 🟧] to [3D Attention Weighted Features 🟦]
Setup
Step 1 of 9: Given
A set of 4 feature vectors (6-D)
Step 2 of 9: Query, Key, Value
Multiply features x's with linear transformation matrices WQ, WK, and WV, to obtain query vectors (q1,q2,q3,q4), key vectors (k1,k2,k3,k4), and value vectors (v1,v2,v3,v4).
"Self" refers to the fact that both queries and keys are derived from the same set of features.
Attention Scores
Step 3 of 9: Prepare for MatMul
Copy query vectors
Copy the transpose of key vectors