Model Throughput

Definitions

1. The rate at which a model-serving system can process requests or generate tokens over time.

   In plain English: How much AI work a system can handle in a given amount of time.

   Example: "We improved model throughput by batching requests and optimizing attention kernels."

Related Terms

• Inference
• KV Cache
• vLLM
• AI Inference Cost
• Batch Inference
• Groq Chip
• Parallel Decoding
• Think Token
• Token Per Second
• Inference Speed