ELU
Activation series · 6 of 12
Activation › ELU
ELU (Exponential Linear Unit) is the smoother Leaky ReLU. It softens the corner at zero by letting the negative side curve downward exponentially toward a fixed floor at -α, instead of continuing as a flat-rate line that bleeds forever.
The Fate of Five Boba Shops (3 of 5)
Same boba block, new courtroom rule. Where Leaky ReLU bled an unlimited fraction of every shop's debt (a -100K loss still cost the system -10K), ELU caps the damage: losses decay exponentially toward a hard floor at -α (here, -0.9K).
A shop earning 3K profit keeps it all. A shop losing 1K only owes the system about 0.57K, most forgiven. A shop hemorrhaging 4K sits near the cap at -0.88K. And no matter how deep a shop sinks — 100K, 1M, anywhere — its damage can never breach -0.9K. That's the credit limit in action.
Walking through the Math
1. Profit & cap: each shop's profit x and the editable cap α (default 0.9).
2. Decay: eˣ − 1, the alternate path each shop would take if it went negative. For positive x this value runs large (and is ignored); for negative x it bottoms out at −1.
3. Output: apply the ELU rule per row. If x > 0, pass x through; if x ≤ 0, pick α times the decay value, bounded above the floor −α.
Element-wise, like its predecessors. Raise α for more lenient treatment, lower it for tighter exposure.
Reading the Numbers
What does the credit-limit court do at each profit level (with α = 0.9, cap = -0.9K)?
Notice how losses asymptote toward the cap at -0.9K. No matter how deep a shop goes (3K, 10K, 100K), its impact on the system can't breach that floor. That's the credit limit at work.
← Previous:
Leaky ReLU
Next:
SiLU →