ELU

Activation series: 6 of 12

Activation Series:

1. Softmax

2. Sigmoid

3. Tanh

4. ReLU

5. Leaky ReLU

6. ELU

7. SiLU

8. GELU

9. Log-Sum-Exp

10. Softplus

11. GLU

12. SwiGLU

ELU (Exponential Linear Unit) is the smoother Leaky ReLU. It softens the corner at zero by letting the negative side curve downward exponentially toward a fixed floor at -α, instead of continuing as a flat-rate line that bleeds forever.

Paid members: open the interactive diagram below ↓