SwiGLU

Definitions

1. A gated activation function variant used in some transformer feed-forward layers to improve model quality relative to simpler activations.

   In plain English: An internal neural-network function used in some modern AI models.

   Example: "The newer checkpoint switched to SwiGLU in the MLP blocks."

Related Terms

• Attention Mechanism
• Mixture of Experts
• scaffolding
• Hypernetwork
• Decoder-Only Transformer
• Gating Network
• Positional Encoding
• Neural Network Depth