Inference Latency

Definitions

1. The delay between submitting an inference request and receiving the result. Inference latency is affected by model size, hardware, batching, network overhead, and any retrieval or tool steps around the model call.

   In plain English: How long it takes an AI model to return an answer.

   Example: "They reduced inference latency by moving reranking off the critical path for simple customer questions."

Related Terms

• AI Speed
• Inference Budget
• Inference Cost
• Inference Server
• Inference Speed
• Inference Time
• Model Latency
• On-Device Inference
• Response Time AI