Model Distillation Loss

Definitions

1. The training loss used when a student model learns to match outputs or distributions from a teacher model during distillation.

   In plain English: The error signal used when training a smaller model to imitate a larger one.

   Example: "They tuned the model distillation loss to preserve ranking behavior."

Related Terms

• distillation
• Active Learning
• Adaptive Learning Rate
• Adversarial Training
• Batch Size
• Curriculum Learning
• Deep Learning
• Deep Reinforcement Learning
• Neural Scaling Laws
• Teacher-Student Model
• Compute Optimal Training