Recurrent Neural Network (RNN) by Hand ✍️
Calculating AI by Hand: 7 of 28
Library › Calculating AI by Hand ✍️
Recurrent Neural Network (RNN) by Hand ✍️
Reinforcement Learning with Human Feedback (RLHF) by Hand ✍️
A Recurrent Neural Network (RNN) processes a sequence one token at a time, carrying hidden states forward from step to step. It contrasts with the Transformer (which processes all tokens in parallel) and is a prerequisite for understanding Mamba.
How does an RNN work?
Setup
Step 1 of 9: Given
Initialize the hidden states to [0, 0].
Process x1 (t=1)
Step 2 of 9: Hidden States
Linearly combine the first input x1 and the hidden states [0, 0] using weights A and B, then apply the ReLU activation, to obtain the new hidden states [3, 6].
Step 3 of 9: Output
Linearly combine the hidden states [3, 6] using weights C to obtain the first output y1.