Activation: the series
12 interactive diagrams
Activation functions are the small nonlinear hinges that let neural networks learn non-trivial patterns. Without them, no matter how deep the network, the whole thing collapses to a single linear map.
This series tells three stories. Part 1 follows you walking the boba block, scoring chewiness. Part 2 zooms out to the bankruptcy court reviewing the same shops' books. Part 3 changes scene to typhoons hitting the coast.
1. Softmax
2. Sigmoid
3. Tanh
4. ReLU
5. Leaky ReLU
6. ELU
7. SiLU
8. GELU
9. Log-Sum-Exp
10. Softplus
11. GLU
12. SwiGLU
