Library › RNNs by Hand, in Excel
Small Language Model
This is the last article of the series. It has one example: the small language model from the start of the series, with every piece now in place.
words, d=6, h=6, layers=4, vocab=30, seq=30, softmax, sample, tied
It reads a word, looks up its embedding, and passes it through four stacked layers. Then it scores every word in the vocabulary, samples one, and reads that word next. Thirty steps later, it has written five sentences.
Now you are rewarded with an understanding of everything important to a language model, including:
recurrence: the hidden state, and the same weights reused at every step
hidden size, input size, and batches
several outputs per step, softmax, and many-to-one
stacked layers and bidirectional RNNs
the encoder, the decoder, and the bottleneck of the context between them
tokens, one-hot vectors, embeddings, and tied embeddings
argmax and sampling
autoregressive generation and prompts
loss and its gradients
I stop short of backpropagation through time. I am still in the middle of doing research on how to represent it in Excel.
The download below has the whole language model in Excel, with live calculations.