SwiGLU is GLU with the sigmoid gate replaced by a SiLU gate. The two-branch shape is identical; the only change is that the gate is no longer capped at 1 — it can push past full rated capacity into overdrive.
Members read on: the worked example and the full walkthrough.
