Window Attention

Definitions

1. An attention pattern that restricts attention to local windows or neighborhoods instead of letting every token attend to every other token globally. Window attention improves efficiency, especially on long inputs, at the cost of more limited direct context mixing.

   In plain English: Attention limited to local windows to reduce computation.

   Example: "The long-document vision model used window attention to keep computation manageable on high-resolution inputs."

Related Terms

• Batch Inference
• Linearized Attention
• Mixture of Depths
• Multi-Head
• Parameter Efficient
• Parameter Sharing
• Prompt Compression
• Sparse Model
• Token Merging