Mixture of Experts

Definitions

1. A neural network architecture that routes different inputs to different specialized sub-networks (experts), activating only a subset for each input. Allows massive models to be computationally efficient — only a fraction of parameters are used per inference.

   In plain English: An AI design where different specialized mini-models handle different types of questions — like a hospital routing you to the right specialist instead of one doctor doing everything.

Related Terms

• Transformer
• Attention Mechanism
• attention head
• Decoder-Only Transformer
• Gating Network
• Positional Encoding
• SwiGLU
• Compute Optimal Training
• Emergent Behavior AI