How does a network remember? A recurrent neural network, or RNN, reads a sequence one step at a time. At each step, it takes one input and updates a hidden state. The hidden state is what the network remembers. We build an RNN in Excel, one change at a time. I keep the weights fixed, so each example differs from the last in one way only.
d=1
Let's start with the smallest network there is. It reads one input, x, in the cream cell at the top. The weight Wx and the bias b, in light red on the left, turn x into F. The bias is the darker red. We write the bias as one more weight, and put a constant 1 under x for it to multiply. So [Wx b] · [x; 1] is the same as Wx · x + b. A tanh turns F into the hidden value h, in green. Then Wy and by turn h into the output y, in blue. There is no memory yet.
