model serving

Definitions

1. The infrastructure and process of deploying trained machine learning models to production so they can receive input data and return predictions in real time or in batches. Key concerns include latency, throughput, model versioning, A/B testing between model versions, canary deployments, and hardware optimization (GPU/CPU allocation).

   In plain English: Putting a trained AI model online so applications can send it data and get predictions back — like deploying a website, but for an AI brain.

   Example: They containerized the recommendation model and deployed it behind a model serving layer that handles 50,000 prediction requests per second with p99 latency under 20ms.

Related Terms

• CI/CD
• Inference
• Quantization
• DevOps
• tokens per second
• feature engineering
• feature store
• MLOps
• Immutable Deploy
• CUDA