Activation
12 interactive lessons
Growing up in Taiwan, nothing can activate my neurons as much as a cup of boba tea. Artificial neurons need activating too. Modern deep learning offers a long menu of them: ReLU, Sigmoid, Tanh, Leaky ReLU, ELU, SiLU, GELU, Softmax, Softplus, Log-Sum-Exp, and more. A bit like walking into a Taiwanese boba shop and staring at the wall menu for ten minutes, trying to figure out what to order.
When I was a student, I just memorized the formulas and names: flashcards before every exam. Or I'd just go with ReLU, the default choice that works well enough in most cases. Like always ordering the taro milk tea with pearls, the default choice that works well enough in most cases. That got me through the projects.
But I never actually understood why each one existed. What problem did it solve? What did it fail at? Why did anyone bother inventing the next?
Stories instead of Equations
Instead of just memorizing equations, I like to explain things using stories. Math concepts are easier to feel than to derive, and for me, feeling a concept is what makes it stick. A story gives the concept a concrete role you can replay in your head.
For activation functions, the scenes come from the Taiwan I grew up in: blocks of boba shops on every corner, the famously chewy QQ pearls, the opening and closing of boba shops, and the typhoons! I hope when you see an activation function in PyTorch later, even if you don't remember the formula, you can still remember my boba story.
Tracing the Diagram
For each concept, I created a diagram to help you visualize and trace the computation. Inputs on the left, operations in the middle, output on the right, arrows carrying values from one column to the next and labeled with what they do (exp, sum, ×, ÷). I want you to trace it with your eyes before you read a word: pick an input cell, follow its arrows step by step, and watch how each operation transforms the value on its way to the output.
Walking through the Math
Then I walk you through the math step by step: a short numbered list, one bullet per operation in the diagram (Input, Gate, Output), each with a one-line explanation of what that step does. The list mirrors the diagram component by component, so you can read it in words or trace it visually and arrive at the same story.
Reading the Numbers
Then I read the numbers with you: a small table with a row for each interesting input, paired with the function's output and a plain-English verdict ("almost certain return", "shuttered", "still carrying -0.27K of debt"). The verdicts are the part I want you to remember when you spot the function in someone else's code later.
Diving into Equations
For some functions, I close with a dive into the equations, tracing where the canonical textbook form comes from: derivatives, identities, and the special-case relationships between members of the same family (sigmoid as softmax-of-two, softplus as LSE-of-two, the slope of softplus is sigmoid). This section is optional. Skip it if the story and the table already gave you what you need.
Three Arcs
This chapter walks through the major activation functions in three short story arcs.
Arc 1 — QQ Scale (3 stories). You walk a boba block in Taiwan, scoring each shop's chewiness on the famous QQ scale. Softmax, Sigmoid, and Tanh all turn those scores into different kinds of verdicts: a coupon split across many shops, a probability of returning to one, a word-of-mouth tilt about whether to recommend it.
Arc 2 — Boba Shops (5 stories). Same block, six months later. The bankruptcy court is reviewing each shop's monthly books, and the rule gets steadily more lenient with each new judge. ReLU shutters anything in the red. Leaky ReLU grants Chapter 11 protection. ELU adds a credit limit on the damage. SiLU brings in a hesitating sigmoid judge. GELU widens the grading scale further. Each judge extends the previous court's rule by one notch.
Arc 3 — Typhoons (4 stories). The setting shifts to the coast. Storms hit, the news rolls out school cancellations, and damage grows exponentially with category. Log-Sum-Exp combines several typhoons into a single equivalent rating. Softplus is the same recipe shrunk to one storm weighed against a fixed baseline of calm. Then the arc turns from measurement to response: GLU turns on the pump by a learned amount, and SwiGLU overrides the safety cap so the pump can push into overdrive when the rain keeps coming.
Let's get activated!
1. Softmax
2. Sigmoid
3. Tanh
4. ReLU
5. Leaky ReLU
6. ELU
7. SiLU
8. GELU
9. Log-Sum-Exp
10. Softplus
11. GLU
12. SwiGLU
