Quantization
Noun · AI & Machine Learning
Definitions
Reducing the precision of a model's numerical weights (e.g., from 32-bit to 4-bit) to decrease memory usage and increase inference speed, with minimal quality loss. The technique that lets you run a 70B-parameter model on a gaming laptop.
In plain English: Compressing an AI model by using less precise numbers for its internal calculations — makes it smaller and faster with only a small quality trade-off.
Related Terms
- Edge Computing
- Inference
- GPU
- Edge AI
- Weights
- Open Weight
- tokens per second
- distillation
- model serving
- Overclocking
- AutoML
- Backpropagation
- Bayesian Optimization
- Contrastive Loss
- Cross-Entropy Loss
- Direct Preference Optimization
- DPO
- Elastic Weight Consolidation
- Federated Averaging
- Floating Point Quantization
- Flow Matching
- INT4
- INT8
- Knowledge Distillation
- Learning to Rank
- Production Build
- Zip
- ONNX Runtime