LSTM by Hand ✍️
Calculating AI by Hand: 8 of 28
Library › Calculating AI by Hand ✍️
LSTM by Hand ✍️
Reinforcement Learning with Human Feedback (RLHF) by Hand ✍️
Since their introduction by Hochreiter and Schmidhuber in 1997, LSTMs dominated as the most effective way to handle long sequential data, at least until the Transformer wave reshaped the landscape.
LSTMs belong to the broader family of recurrent neural network (RNNs) that process data sequentially in a recurrent manner.
Transformers, on the other hand, abandon recurrence and use self-attention instead to process data concurrently in parallel.
Recently, there is renewed interest in recurrence as people realized self-attention doesn’t scale to extremely long sequences, like hundreds of thousands of tokens. Mamba is a good example to bring back recurrence.
All of a sudden, it is cool to study LSTMs.
How do LSTMs work?
Setup
Step 1 of 15: Given
🟨 Input sequence X1, X2, X3 (d = 3)
🟩 Hidden state h (d = 2)
🟦 Memory C (d = 2)
Weight matrices Wf, Wc, Wi, Wo
Process t = 1
Step 2 of 15: Initialize
Randomly set the previous hidden state h0 to [1, 1] and memory cells C0 to [0.3, -0.5]
Step 3 of 15: Linear Transform
Multiply the four weight matrices with the concatenation of current input (X1) and the previous hidden state (h0).
The results are feature values, each is a linear combination of the current input and hidden state.