Log-Sum-Exp

Activation series: 9 of 12

Activation Series:

1. Softmax

2. Sigmoid

3. Tanh

4. ReLU

5. Leaky ReLU

6. ELU

7. SiLU

8. GELU

9. Log-Sum-Exp

10. Softplus

11. GLU

12. SwiGLU

Log-sum-exp (LSE) is the smooth maximum function: the smooth-and-differentiable cousin of max. It shows up at the heart of softmax, cross-entropy loss, and any computation that combines values on a log scale.

Paid members: open the interactive diagram below ↓