vLLM

Definitions

1. An open-source inference engine for large language models focused on high-throughput serving and efficient memory usage.

   In plain English: Open-source software for serving large language models efficiently.

   Example: "We switched to vLLM to improve concurrent request handling on the same GPUs."

Related Terms

• Floating Point Quantization
• Hugging Face
• INT4
• INT8
• KV Cache
• LLM Agent
• LLM Evaluation
• AI Inference Cost
• Model Throughput