GLU

Activation series: 11 of 12

Activation Series:

1. Softmax

2. Sigmoid

3. Tanh

4. ReLU

5. Leaky ReLU

6. ELU

7. SiLU

8. GELU

9. Log-Sum-Exp

10. Softplus

11. GLU

12. SwiGLU

GLU is the first activation in this chapter where the network decides about a value rather than only shaping it. The same input is run through two learned linear transforms (one produces a value, the other produces a 0..1 mask), then multiplied elementwise.

Paid members: open the interactive diagram below ↓