Log-Sum-Exp
Activation series · 9 of 12
Activation › Log-Sum-Exp
Log-sum-exp (LSE) is the smooth maximum function: the smooth-and-differentiable cousin of `max`. It shows up at the heart of softmax, cross-entropy loss, and any computation that combines values on a log scale.
Typhoon Aftermath (1 of 4)
Other than boba, another thing I loved growing up was typhoons. Why? Because I could get a day off from school! The TV would scroll the city closure notice across the bottom of the screen, the family would stock up on instant noodles, and we'd watch the rain hammer the windows knowing there'd be no class the next morning. The bigger the storm, the longer the holiday.
Typhoon categories scale exponentially in damage, roughly 4× per step in real life. For our LSE math we'll use e as the per-step multiplier, the function's natural base. So Cat 5 isn't 5× worse than Cat 1; it's e⁴ ≈ 55× worse. Damage scales like eˣ.
When several typhoons hit at once, what's the combined damage expressed back on the category scale? You can't just add categories; that's adding logs, not numbers. You have to exit log space, sum real damages, and re-enter. That's log-sum-exp: exp → sum → log. The result is the smooth maximum — as one storm pulls ahead, LSE approaches it; when storms are close, it sits above them by a consensus bonus.
Walking through the Math
1. Category: each storm's category x.
2. Damage: exponentiate to actual damage, eˣ.
3. Total: sum all five damages into a single total, Z = Σ eˣ.
4. Back to category: take the log to convert back to the category scale, LSE = ln(Z).
For these five categories (1 through 5), LSE ≈ 5.45, just above the worst single storm at Cat 5, because the smaller storms contribute a small bonus.
Reading the Numbers
How does LSE combine different typhoon mixes? Each row walks through the recipe: exp every category, sum the damages, take the log to land back on the category scale.
When one input dominates, LSE collapses to max. When inputs are tied, the bonus is largest (ln n above the shared value). Spread inputs sit somewhere in between.
← Previous:
GELU
Next:
Softplus →