Sparse Model

Definitions

1. A model in which only a subset of parameters or pathways are active for a given input, rather than using the full parameter set every time. Sparse models can improve efficiency if routing and serving are handled well.

   In plain English: A model that uses only part of its parameters for each input.

   Example: "They explored a sparse model architecture to reduce inference cost while keeping high-capacity behavior on difficult queries."

Related Terms

• Context Extension
• Data Mixture
• Dense Model
• Frontier Model
• Internal Representation
• Large Context Model
• Likelihood
• Linearized Attention
• Model Efficiency
• Parameter Efficient
• Parameter Sharing
• Prompt Compression
• Token Merging
• Window Attention