Grouped Query Attention

Definitions

1. An attention variant where multiple query heads share key and value heads, reducing memory bandwidth and improving inference efficiency.

   In plain English: A more efficient attention method that shares some internal parts to save compute.

   Example: "Grouped query attention helped the model run faster without a huge quality drop."

Related Terms

• BERT
• Cross-Attention
• KV Cache
• Bitnet
• Decoder-Only Transformer
• Positional Encoding
• RoPE
• Parallel Decoding