GPU Utilization

Definitions

1. A measure of how fully a GPU is being used for useful computation over time. Low GPU utilization can indicate bottlenecks in data loading, batching, model parallelism, or orchestration rather than lack of raw hardware capacity.

   In plain English: How much of the GPU's capacity is actually being used.

   Example: "Profiling showed poor GPU utilization because requests were too small and arrived too unevenly to keep the inference workers busy."

Related Terms

• AI Cache
• AI Efficiency
• AI Optimization
• AI Performance
• AI Speed
• GPU Memory
• Inference Speed
• LLM Throughput
• Throughput AI