Context Distillation

Definitions

1. A technique for training or adapting models so they internalize behaviors that would otherwise require large context at inference time. The goal is to compress useful contextual guidance into model behavior more efficiently.

   In plain English: Teaching a model to absorb recurring context so less needs to be included at runtime.

   Example: "They explored context distillation to reduce how much instruction text had to be sent with every request."

Related Terms

• Model Context Protocol
• Chinchilla Scaling
• Constitutional AI Method
• Expert Iteration
• Frozen Embedding
• Goal Conditioned
• Gradient-Free Optimization
• Machine Unlearning
• Multi-Turn
• Process Reward Model
• Replay Buffer
• Sequence Length