Multi-Head

Definitions

1. Describing an architecture that uses multiple attention heads or output heads to process information in parallel or from different representational perspectives. The term commonly appears in transformer and multitask model discussions.

   In plain English: Using multiple parallel heads to process or represent information.

   Example: "The paper showed how multi-head attention let the model track different relationships in the same sentence at once."

Related Terms

• Transformer
• AI Model Registry
• Attention Score
• Bidirectional Encoder
• Cognitive Architecture
• Linearized Attention
• Mixture of Depths
• Window Attention