Attention Glossary
Browse 7 attention terms defined in plain English, from the cultural dictionary of computing.
7 Attention Terms
- Cross-Attention
- An attention mechanism where one sequence attends to representations computed from another sequence. It influences how models are trained, evaluated, or...
- Focus Mode
- A tool setting or personal working mode intended to reduce interruptions and help maintain concentration. In tech culture focus mode is part workflow feature,...
- Grouped Query Attention
- An attention variant where multiple query heads share key and value heads, reducing memory bandwidth and improving inference efficiency.
- Linearized Attention
- An approach to making attention mechanisms more computationally efficient by approximating or restructuring them so cost scales more favorably with sequence...
- Multi-Head
- Describing an architecture that uses multiple attention heads or output heads to process information in parallel or from different representational...
- Transformer
- A neural network architecture that uses self-attention mechanisms to process sequential data in parallel rather than sequentially. Introduced in the 2017 paper...
- Window Attention
- An attention pattern that restricts attention to local windows or neighborhoods instead of letting every token attend to every other token globally. Window...
Related Topics
- Ai (3 terms in common)
- Transformers (2 terms in common)
- Efficiency (2 terms in common)
- Architecture (2 terms in common)
- Inference (1 terms in common)
- Productivity (1 terms in common)
- Multimodal (1 terms in common)
- Foundational (1 terms in common)
- Tools (1 terms in common)