Backpropagation
AI by Hand ✍️ in Excel
Arguably the most mysterious, magical, and difficult-to-understand process, backpropagation enables a deep neural network to figure out how to update each layer with respect to a given cost function!
A few months ago, I shared an exercise to demonstrate it is possible to calculate backpropagation for a non-trivial, three-layer network by hand. I hope this exercise can help demystify and simplify the understanding of backpropagation.
I was surprised many people actually were able to follow the calculations! 🙌
Some even discovered a calculation error in the forward step. I definitely appreciated their pointing it out!
Since then, I have been asked by many people to create a spreadsheet version of this exercise.
My first attempt was to use Google Sheet. But it turns out it is very difficult to draw a network graph and insert equations to explain the calculations.
I decided to use Excel instead and got pretty good results.
Compared to the original exercise, this Excel version has several enhancements:
1. Extend from one input to a batch of two inputs.
2. Show equations
3. Show/Hide the forward and backward passes
4. Break calculations into individual row and column operations. You can press F2 in each cell to see which row and column of values are involved in the calculation of the value in that cell.
I hope many of you would find this exercise helpful in enhancing your understanding of the backpropagation process!
Download
🙌 Thanks a lot!
Dear Tom another time I found your illustration in excel very clear! I'm waiting for LSTM with multi LSTM units, many inputs and batch size.