distillation
/dis-tih-LAY-shun/ · noun · AI & Machine Learning · Origin: 2015
Definitions
Distillation (or knowledge distillation) is a model compression technique in machine learning where a smaller, more efficient student model is trained to replicate the behavior of a larger, more capable teacher model. Introduced by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean in 2015, the technique works by training the student not just on the original training data labels but on the soft probability distributions (soft targets) output by the teacher model, which contain richer information about relationships between classes. For example, a teacher model classifying animals might assign small but nonzero probabilities to similar animals, and these nuanced predictions help the student learn more effectively than hard labels alone. Distillation enables deploying sophisticated AI capabilities on resource-constrained devices like phones and edge hardware. Many commercial language models use distillation to create smaller, faster variants (like DistilBERT or various small chat models distilled from larger ones). The technique represents a practical solution to the tension between model capability and deployment efficiency.
In plain English: Shrinking a big, expensive AI into a smaller, cheaper one that works almost as well, like condensing a textbook into study notes.
Example: We distilled the 70B parameter model down to 7B and kept 90% of the performance at a fraction of the inference cost.
Etymology
- 2006
- Bucila, Caruana, and Niculescu-Mizil demonstrated 'model compression,' training a small neural network to mimic a large ensemble model's predictions.
- 2015
- Hinton, Vinyals, and Dean published 'Distilling the Knowledge in a Neural Network,' formalizing 'knowledge distillation.' The teacher-student framework and soft label (temperature) technique became standard.
- 2019
- DistilBERT demonstrated that distillation could compress Transformer language models to 60% of their size while retaining 97% of performance, making deployment on edge devices practical.
- 2023-Present
- Distillation became central to deploying large language models efficiently. Smaller models trained on larger model outputs (e.g., Alpaca from LLaMA) democratized access to capable AI systems.
Origin Story
Shrinking Giants: How Small Models Learn from Big Ones
Knowledge distillation is a model compression technique where a smaller 'student' model is trained to replicate the behavior of a larger, more capable 'teacher' model. The concept was formalized by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean in their influential 2015 paper 'Distilling the Knowledge in a Neural Network.' Hinton's key insight was that the teacher model's output probabilities (called 'soft labels') contain rich information beyond simple correct/incorrect classifications. For example, when a digit classifier processes an image of a '7,' the teacher might assign small but meaningful probabilities to '1' and '9,' encoding knowledge about visual similarity that hard labels (just '7') cannot convey. By training the student on these soft targets at a raised 'temperature' (which softens the probability distribution), the smaller model learns the teacher's nuanced understanding of the data. The technique proved transformative for deploying AI in resource-constrained environments: mobile devices, edge computing, and real-time applications that cannot afford the latency of running massive models. Distillation has become a standard tool in modern ML pipelines, used by companies like Google (DistilBERT), Apple (on-device models), and virtually every team working with large language models that need to ship efficient production systems.
Coined by: Geoffrey Hinton, Oriol Vinyals, Jeff Dean
Context: Formalized in a 2015 paper showing that smaller models could learn rich representations from larger models' soft probability outputs.
Fun fact: DistilBERT, created by Hugging Face in 2019, used distillation to produce a model 60% smaller and 60% faster than BERT while retaining 97% of its language understanding capabilities, demonstrating that most of a large model's knowledge can be compressed into a fraction of the parameters.