Mixture of Depths

Definitions

1. An architectural idea where different tokens or inputs receive different amounts of computational depth, allowing the model to spend more effort on harder parts and less on easier ones. It aims to improve efficiency without applying the same amount of compute everywhere.

   In plain English: An architecture that uses more or less computation depending on how hard each part is.

   Example: "They experimented with mixture of depths to reduce inference cost on routine tokens while preserving quality on harder reasoning steps."

Related Terms

• AI Model Registry
• Attention Score
• Bidirectional Encoder
• Batch Inference
• Cognitive Architecture
• Linearized Attention
• Multi-Head
• Parameter Efficient
• Parameter Sharing
• Window Attention