Tanh
Activation series · 3 of 12
Activation › Tanh
Tanh is sigmoid's signed cousin: same S-curve, stretched to span [-1, +1] instead of [0, 1]. The signed range is its whole point: a single output can carry both direction and magnitude (push or pull, gain or loss, yes or no with a sign).
QQ Activation (3 of 3)
The next day at lunch, a coworker asks: "How was that new place you tried?" What I say next is tanh.
Same shop, same chewiness score `x` from the QQ scale — but the downstream question has shifted. Sigmoid was private: what is the chance between 0 and 1 that I will come back? Tanh is public: what will I tell others? Good (+1)? Bad (-1)? Or say nothing (0)?
Tanh turns the same chewiness score from -3 to 3 into a signed tilt between -1 and 1: positive means rave, negative means "don't go there", zero means stay quiet.
So, in my mind, I gave a chewiness score of -1. Publicly, I said subtly, "the new shop has room for improvement."
Walking through the Math
1. Input: the new shop's chewiness `x` and the fixed baseline 0.
2. Scale: double the input, 2x. The "twice the score" trick that makes sigmoid produce the right shape for the tanh bridge. The baseline stays 0.
3. Exp: exponentiate both, e^(2x) and e^0 = 1.
4. Sum: Z = e^(2x) + 1.
5. Sigmoid: P = [e^(2x) / Z, 1 / Z], the same probability pair as sigmoid, but on the doubled input.
6. Center: shift down by 0.5, P − 0.5, recentering on zero (values in [-0.5, +0.5]).
7. Tanh: double, 2(P − 0.5). Top row is tanh(x), bottom row is −tanh(x).
Sigmoid's two outputs always sum to one; tanh's two outputs always sum to zero. Same machinery, recentered.
Reading the Numbers
What's tanh's word-of-mouth verdict for each chewiness?
Notice the symmetry around zero. Tanh treats positive and negative experiences as equally meaningful. The shop doesn't just lose your loyalty (sigmoid), it suffers active harm to its reputation. That's the cost of sharing a chewiness card with a wide social circle.
Diving into Equations
The walkthrough computed tanh as `2σ(2x) − 1`. Here's how that collapses into the canonical form you'll see in textbooks.
Start with the bridge:
Substitute the sigmoid definition σ(2x) = dfrac{1}{1 + e^{-2x}}:
Combine over the common denominator:
Multiply top and bottom by e^{x}:
That's the canonical form for tanh: two equivalent expressions, same curve.
← Previous:
Sigmoid
Next:
ReLU →