Inference Glossary

Browse 52 inference terms defined in plain English, from the cultural dictionary of computing.

52 Inference Terms

Aggregation Attack
An attack where individually harmless pieces of information are combined to infer sensitive data or reveal protected patterns. Aggregation attacks matter in...
Batch Inference
Running model predictions on many inputs together, usually to improve throughput and hardware efficiency compared with one-at-a-time processing.
Bitnet
A family of neural-network approaches that push weights toward extremely low-bit representations, often to reduce memory use and speed up inference.
Floating Point Quantization
A compression technique that lowers numerical precision to reduce memory, bandwidth, and inference cost. It influences how models are trained, evaluated, or...
Groq
A company that designs custom AI inference chips (Language Processing Units, or LPUs) optimized for running large language models at extreme speed. Groq's...
Groq Chip
AI-serving hardware associated with Groq, often discussed for low-latency language-model inference.
Grouped Query Attention
An attention variant where multiple query heads share key and value heads, reducing memory bandwidth and improving inference efficiency.
Implicit Reasoning
Reasoning that appears in a model's behavior without being explicitly exposed step by step in the output. The term is used when a system seems to perform...
Inference Budget
The amount of compute, latency, tokens, or money available for running inference on a task or product. Inference budgets force tradeoffs among model quality,...
Inference Cost
The cost incurred when running a model to produce outputs, including compute usage, provider charges, and operational overhead. Inference cost is a central...
Inference Latency
The delay between submitting an inference request and receiving the result. Inference latency is affected by model size, hardware, batching, network overhead,...
Inference Optimization
The process of making model inference faster, cheaper, or more efficient through techniques such as quantization, batching, caching, compilation, or smarter...
Inference Server
A server or service process dedicated to hosting models and executing inference requests. Inference servers manage model loading, batching, scheduling, and...
Inference Speed
How quickly a model can process inputs and generate outputs during inference, often measured in latency or tokens per second. Inference speed influences user...
Inference Time
The time taken by a model or AI pipeline to compute an output for a specific input at runtime. The phrase is often used interchangeably with inference latency,...
INT4
A four-bit integer numerical format commonly used for aggressive model quantization during inference. It influences how models are trained, evaluated, or...
INT8
An eight-bit integer numerical format used to reduce model size and speed up inference workloads. It influences how models are trained, evaluated, or served,...
KV Cache
A transformer inference cache that stores previously computed key and value tensors for faster decoding. It influences how models are trained, evaluated, or...
LLM Agent
LLM Agent is an AI or ML concept used to represent, train, evaluate, or deploy learned systems. It is commonly used for building production models and research...
LLM Evaluation
LLM Evaluation is an evaluation concept used to measure model quality, robustness, or efficiency. It is commonly used for comparing systems before release or...
LLM Fine-Tuning
LLM Fine-Tuning is a stage of model optimization where weights or behaviors are adjusted from data or feedback. It is commonly used for foundation-model...
LLM Inference
LLM Inference is the phase where a trained model processes new inputs to produce predictions or generations. It is commonly used for production APIs, batch...
LLM Routing
LLM Routing is an AI or ML concept used to represent, train, evaluate, or deploy learned systems. It is commonly used for building production models and...
LLM Safety
LLM Safety is a safety or governance control for limiting harmful, noncompliant, or insecure model behavior. It is commonly used for production AI systems that...
Local LLM
Local LLM is an AI or ML concept used to represent, train, evaluate, or deploy learned systems. It is commonly used for building production models and research...
Long Context
Long Context is an AI or ML concept used to represent, train, evaluate, or deploy learned systems. It is commonly used for building production models and...
Model Throughput
The rate at which a model-serving system can process requests or generate tokens over time.
Multimodal Tokenizer
Multimodal Tokenizer is a unit or control marker used when text is segmented and generated by language models. It is commonly used for prompt assembly,...
On-Device Inference
Inference performed locally on a device rather than in the cloud. On-device inference is valuable when privacy, responsiveness, bandwidth, or offline operation...
ONNX Runtime
A runtime for executing models in the ONNX format across different hardware backends with an emphasis on portability and inference performance.
OpenAI
OpenAI is an AI company and research lab that develops foundation models, developer APIs, and products such as ChatGPT. It is commonly used for building...
Output Token
Output Token is a unit or control marker used when text is segmented and generated by language models. It is commonly used for prompt assembly, decoding...
Parallel Decoding
Methods that generate or verify multiple output tokens in parallel to reduce latency compared with strictly sequential decoding.
Prompt Caching
Prompt Caching is a prompting technique or prompt-side control used to shape large-model behavior at inference time. It is commonly used for chat systems,...
Prompt Chaining
Prompt Chaining is a prompting technique or prompt-side control used to shape large-model behavior at inference time. It is commonly used for chat systems,...
Prompt Engineering Detail
Prompt Engineering Detail is a prompting technique or prompt-side control used to shape large-model behavior at inference time. It is commonly used for chat...
Prompt Injection Detail
Prompt Injection Detail is a prompting technique or prompt-side control used to shape large-model behavior at inference time. It is commonly used for chat...
Prompt Template
Prompt Template is a prompting technique or prompt-side control used to shape large-model behavior at inference time. It is commonly used for chat systems,...
Prompt Tuning
Prompt Tuning is a prompting technique or prompt-side control used to shape large-model behavior at inference time. It is commonly used for chat systems,...
Replicate
A platform for running open-source machine learning models in the cloud via a simple API. Replicate lets developers run models without managing infrastructure:...
Stop Token
Stop Token is a unit or control marker used when text is segmented and generated by language models. It is commonly used for prompt assembly, decoding control,...
Subword Tokenization
Subword Tokenization is a unit or control marker used when text is segmented and generated by language models. It is commonly used for prompt assembly,...
System Prompt Detail
System Prompt Detail is a prompting technique or prompt-side control used to shape large-model behavior at inference time. It is commonly used for chat...
Think Token
A colloquial term for tokens consumed during hidden reasoning or intermediate internal processing in some model designs or product discussions.
Together AI
A cloud platform for running, fine-tuning, and training open-source AI models. Together AI provides an inference API for popular open models (LLaMA, Mistral,...
Token Budget
Token Budget is a unit or control marker used when text is segmented and generated by language models. It is commonly used for prompt assembly, decoding...
Tokenization Detail
Tokenization Detail is a unit or control marker used when text is segmented and generated by language models. It is commonly used for prompt assembly, decoding...
Token Limit
Token Limit is a unit or control marker used when text is segmented and generated by language models. It is commonly used for prompt assembly, decoding...
Token Per Second
A throughput measure showing how many tokens a model can generate each second during inference.
Tokens Per Second Detail
Tokens Per Second Detail is a unit or control marker used when text is segmented and generated by language models. It is commonly used for prompt assembly,...
vLLM
An open-source inference engine for large language models focused on high-throughput serving and efficient memory usage.
Zero-Shot Prompting
Zero-Shot Prompting is a prompting technique or prompt-side control used to shape large-model behavior at inference time. It is commonly used for chat systems,...

Related Topics