Leaky ReLU
Activation series · 5 of 12
Activation › Leaky ReLU
Plain ReLU wipes negative values to zero. Clean, but a shop that shutters can never recover, since both its output and its gradient stay pinned at zero. This is the dying ReLU problem, and in deep networks it can quietly kill a meaningful fraction of the units.
The Fate of Five Boba Shops (2 of 5)
Same boba block. The bankruptcy court has a new rule: instead of shuttering every failing shop the moment it goes red (ReLU's verdict), the court grants Chapter 11 protection. The shop's debt is restructured down to a fraction α (typically 0.1) — most of it forgiven, the shop wounded but not killed. A small negative signal still flows through, so the gradient survives, and the shop can crawl back to life if a TikTok goes viral.
Walking through the Math
1. Input & leakage: each shop's profit x and the editable rate α (default 0.1).
2. Gate: 1 for shops in the black, α for those under bankruptcy protection.
3. Output: x multiplied by the gate. Profitable shops pass through untouched; struggling ones shrink to α·x but still carry a sign.
Element-wise like ReLU, but every neuron's losses leak through at rate α.
Reading the Numbers
What does the 10% restructuring deal look like across profit (loss) levels?
The leak is linear: doubling the debt doubles the residual loss. There's no ceiling on how much a single shop can drag the system down.
← Previous:
ReLU
Next:
ELU →