AI by Hand ✍️

AI by Hand ✍️

Log-Sum-Exp

Activation series: 9 of 12

Prof. Tom Yeh's avatar
Prof. Tom Yeh
May 11, 2026
∙ Paid

Library › Activation

  1. Softmax

  2. Sigmoid

  3. Tanh

  4. ReLU

  5. Leaky ReLU

  6. ELU

  7. SiLU

  8. GELU

  9. Log-Sum-Exp

  10. Softplus

  11. GLU

  12. SwiGLU

equation

Log-sum-exp (LSE) is the smooth maximum function: the smooth-and-differentiable cousin of max. It shows up at the heart of softmax, cross-entropy loss, and any computation that combines values on a log scale.

Members read on: the worked example and the full walkthrough.

This post is for paid subscribers

Already a paid subscriber? Sign in
© 2026 Tom Yeh · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for great culture