Attention

Definitions

1. The core mechanism of transformer models that allows each token to dynamically attend to (weigh the importance of) every other token in the sequence. Self-attention computes Query, Key, and Value matrices to determine relevance. Multi-head attention runs multiple attention patterns in parallel. 'Attention Is All You Need' (2017) revolutionized NLP.

   In plain English: The mechanism that lets AI models figure out which words in a sentence are most relevant to each other.

   Example: "In the sentence 'The cat sat on the mat because it was tired,' attention lets the model understand that 'it' refers to 'the cat.'"

Related Terms

• Large Language Model
• Attention Mechanism
• LLM
• Context Window
• Token
• Retrieval-Augmented Generation
• Semantic Search
• Embedding Model
• Beam Search
• BERT
• Byte Pair Encoding
• Differential Transformer
• Dual Encoder
• ELMo
• Graph Transformer
• Hypernetwork
• Slot Filling