Transformer by Hand ✍️
Calculating AI by Hand: 11 of 28
Library › Calculating AI by Hand ✍️
Transformer by Hand ✍️
Reinforcement Learning with Human Feedback (RLHF) by Hand ✍️
To study the transformer architecture, it is like opening up the hood of a car and seeing all sorts of engine parts: embeddings, positional encoding, feed-forward network, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking. This list of jargons feels overwhelming!
What are the key parts that really make the transformer (🚗) run?
In my opinion, the 🔑 key is the combination of: attention weighting and feed-forward network.
All the other parts are enhancements to make the transformer (🚗) run faster and longer, which is still important because those enhancements are what lead us to "large" language models. 🚗 -> 🚚
Setup
Step 1 of 6: Given
Input features from the previous block (5 positions)
Attention
Step 2 of 6: Attention Matrix
Feed all 5 features to a query-key attention module (QK) to obtain an attention weight matrix (A). I will skip the details of this module. In a follow-up post I will unpack this module.
Step 3 of 6: Attention Weighting
Multiply the input features with the attention weight matrix to obtain attention weighted features (Z). Note that there are still 5 positions.
The effect is to combine features across positions (horizontally), in this case, X1 := X1 + X2, X2 := X2 + X3....etc.