Softplus

Activation series: 10 of 12

Activation Series:

1. Softmax

2. Sigmoid

3. Tanh

4. ReLU

5. Leaky ReLU

6. ELU

7. SiLU

8. GELU

9. Log-Sum-Exp

10. Softplus

11. GLU

12. SwiGLU

Softplus is the simplest case of log-sum-exp: a single input weighed against a fixed baseline of zero. It's two-input LSE with the second input pinned to e⁰ = 1. Outside hidden layers, softplus is the natural form of binary cross-entropy loss: −log(σ(x)) = softplus(−x).

Paid members: open the interactive diagram below ↓