AI by Hand ✍️

AI by Hand ✍️

GLU

Activation series: 11 of 12

Prof. Tom Yeh's avatar
Prof. Tom Yeh
May 11, 2026
∙ Paid

Library › Activation

  1. Softmax

  2. Sigmoid

  3. Tanh

  4. ReLU

  5. Leaky ReLU

  6. ELU

  7. SiLU

  8. GELU

  9. Log-Sum-Exp

  10. Softplus

  11. GLU

  12. SwiGLU

equation

GLU is the first activation in this chapter where the network decides about a value rather than only shaping it. The same input is run through two learned linear transforms (one produces a value, the other produces a 0..1 mask), then multiplied elementwise.

Members read on: the worked example and the full walkthrough.

This post is for paid subscribers

Already a paid subscriber? Sign in
© 2026 Tom Yeh · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for great culture