Training Glossary

Browse 117 training terms defined in plain English, from the cultural dictionary of computing.

117 Training Terms

Activation Function
A mathematical function applied to a neural unit's output to introduce nonlinearity into the model. It influences how models are trained, evaluated, or served,...
Active Learning
A machine learning training concept related to active learning and how model parameters are learned or stabilized. It influences how models are trained,...
Adaptive Learning Rate
A machine learning training concept related to adaptive learning rate and how model parameters are learned or stabilized. It influences how models are trained,...
Adversarial Training
A machine learning training concept related to adversarial training and how model parameters are learned or stabilized. It influences how models are trained,...
AI Training
The process of adjusting a model's parameters using data so it learns patterns or capabilities relevant to a task or domain. In product discussions, the term...
Batch Size
A machine learning training concept related to batch size and how model parameters are learned or stabilized. It influences how models are trained, evaluated,...
Chinchilla Scaling
A scaling-law result emphasizing that, for a given compute budget, model size and training data should be balanced rather than simply making models larger with...
Compute Optimal Training
A training regime chosen to get the best model quality for a fixed compute budget by balancing model size and data volume appropriately.
Constitutional AI Method
An alignment method in which an AI system is guided by an explicit set of principles or rules, often using self-critique and revision to improve responses...
Context Distillation
A technique for training or adapting models so they internalize behaviors that would otherwise require large context at inference time. The goal is to compress...
CTF
CTF is a capture-the-flag exercise where participants solve security challenges to practice exploitation, analysis, and defense. Security teams use it to...
Curriculum Learning
A machine learning training concept related to curriculum learning and how model parameters are learned or stabilized. It influences how models are trained,...
Cyber Range
Cyber Range is a controlled environment for realistic security training, attack simulation, and defensive exercises. Security teams use it to enforce trust,...
Deep Learning
A machine learning training concept related to deep learning and how model parameters are learned or stabilized. It influences how models are trained,...
Deep Reinforcement Learning
A machine learning training concept related to deep reinforcement learning and how model parameters are learned or stabilized. It influences how models are...
distillation
Distillation (or knowledge distillation) is a model compression technique in machine learning where a smaller, more efficient student model is trained to...
Distributed Training
A machine learning training concept related to distributed training and how model parameters are learned or stabilized. It influences how models are trained,...
Dropout Detail
A machine learning training concept related to dropout detail and how model parameters are learned or stabilized. It influences how models are trained,...
End-to-End Learning
A machine learning training concept related to end-to-end learning and how model parameters are learned or stabilized. It influences how models are trained,...
Epoch
One complete pass through the entire training dataset. Models are typically trained for multiple epochs — enough to learn patterns but not so many that they...
Expert Iteration
A training approach in which a stronger search or expert process generates improved targets or guidance that a model then learns to imitate, repeating this...
Exponential Moving Average
A smoothed running average that weights recent observations more heavily than older ones. It influences how models are trained, evaluated, or served, and it...
Fine-Tuning
The process of taking a pre-trained model and further training it on a smaller, domain-specific dataset to improve performance on a particular task. Like...
Frozen Embedding
An embedding layer or embedding representation whose parameters are kept fixed during some later stage of training or adaptation. Freezing embeddings can...
Full Fine-Tuning
A machine learning training concept related to full fine-tuning and how model parameters are learned or stabilized. It influences how models are trained,...
Goal Conditioned
Describing a model or policy that takes an explicit goal as part of its input so its behavior can adapt to the desired outcome. Goal-conditioned approaches are...
Gradient
A machine learning training concept related to gradient and how model parameters are learned or stabilized. It influences how models are trained, evaluated, or...
Gradient Accumulation
A machine learning training concept related to gradient accumulation and how model parameters are learned or stabilized. It influences how models are trained,...
Gradient Checkpointing
A machine learning training concept related to gradient checkpointing and how model parameters are learned or stabilized. It influences how models are trained,...
Gradient Clipping
A machine learning training concept related to gradient clipping and how model parameters are learned or stabilized. It influences how models are trained,...
Gradient Descent
A machine learning training concept related to gradient descent and how model parameters are learned or stabilized. It influences how models are trained,...
Gradient Explosion
A machine learning training concept related to gradient explosion and how model parameters are learned or stabilized. It influences how models are trained,...
Gradient-Free Optimization
Optimization methods that do not rely on computing gradients and instead search using alternatives such as evolutionary strategies, sampling, or black-box...
Gradient Vanishing
A machine learning training concept related to gradient vanishing and how model parameters are learned or stabilized. It influences how models are trained,...
Group Normalization
A normalization method that standardizes activations within channel groups instead of across the batch. It influences how models are trained, evaluated, or...
Hyperparameter Sweep
A systematic run of many training jobs with different hyperparameter settings, such as learning rate or batch size, to find stronger model performance.
Hyperparameter Tuning Detail
A machine learning training concept related to hyperparameter tuning detail and how model parameters are learned or stabilized. It influences how models are...
Imitation Learning
A machine learning training concept related to imitation learning and how model parameters are learned or stabilized. It influences how models are trained,...
Incremental Learning
A machine learning training concept related to incremental learning and how model parameters are learned or stabilized. It influences how models are trained,...
Instruction Tuning
A machine learning training concept related to instruction tuning and how model parameters are learned or stabilized. It influences how models are trained,...
Knowledge Cutoff
The latest point in time up to which a model's training data is known or assumed to reflect information. Knowledge cutoff matters because models without...
Layer Normalization
A normalization method that standardizes activations across features within each example. It influences how models are trained, evaluated, or served, and it...
Leaky ReLU
An activation function that keeps a small nonzero slope for negative inputs. It influences how models are trained, evaluated, or served, and it can materially...
Learning Curriculum
A machine learning training concept related to learning curriculum and how model parameters are learned or stabilized. It influences how models are trained,...
Learning Rate
A machine learning training concept related to learning rate and how model parameters are learned or stabilized. It influences how models are trained,...
Learning Rate Schedule
A machine learning training concept related to learning rate schedule and how model parameters are learned or stabilized. It influences how models are trained,...
Learning Rate Warmup
A machine learning training concept related to learning rate warmup and how model parameters are learned or stabilized. It influences how models are trained,...
Loss Function
Loss Function is a training-time optimization concept that governs how model parameters are updated. It is commonly used for iterative learning loops for...
Loss Landscape
Loss Landscape is a training-time optimization concept that governs how model parameters are updated. It is commonly used for iterative learning loops for...
Machine Unlearning
Techniques aimed at removing the influence of specific data from a trained model without fully retraining it from scratch. Machine unlearning is discussed in...
Model Collapse
A phenomenon where AI models trained on AI-generated data progressively degrade, losing diversity and accuracy over generations. The ouroboros of AI — models...
Model Distillation Loss
The training loss used when a student model learns to match outputs or distributions from a teacher model during distillation.
Model Fine-Tuning Cost
The total expense of adapting a base model to a new task, including compute, data preparation, experimentation, and engineering time.
Neural Network Depth
The number of layers in a neural network, which affects representational capacity, compute cost, and optimization behavior.
Neural Scaling Law
Neural Scaling Law is an AI or ML concept used to represent, train, evaluate, or deploy learned systems. It is commonly used for building production models and...
Neural Scaling Laws
Empirical relationships showing how model performance changes as compute, data, or parameter count increases.
Next Token Prediction
The task of predicting the next token in a sequence given the preceding context, which is a core training objective for many language models. Despite its...
Noise Injection
The deliberate addition of noise to inputs, activations, gradients, or parameters during training or testing to improve robustness, exploration, or...
NVIDIA H100
A high-end NVIDIA data-center GPU widely used for training and serving large AI models.
Objective Function
The function a model or training process is trying to optimize, such as minimizing error or maximizing reward. The objective function strongly shapes what the...
Optimizer
Optimizer is a training-time optimization concept that governs how model parameters are updated. It is commonly used for iterative learning loops for neural...
Overfitting
When a model learns the training data too well — including its noise and random fluctuations — and fails to generalize to new, unseen data. The model has...
Pentest Lab
A controlled environment built for practicing, validating, or teaching penetration testing without risking production systems. Pentest labs often include...
Performance Benchmark
Performance Benchmark is an evaluation concept used to measure model quality, robustness, or efficiency. It is commonly used for comparing systems before...
Perplexity Detail
Perplexity Detail is an evaluation concept used to measure model quality, robustness, or efficiency. It is commonly used for comparing systems before release...
Phishing Simulation
A controlled internal exercise that sends realistic but benign phishing-style messages to measure and improve user awareness, reporting, and response habits....
Pipeline Parallelism
Pipeline Parallelism is a systems strategy for splitting model computation or parameters across devices. It is commonly used for training and serving workloads...
Precision Detail
Precision Detail is an evaluation concept used to measure model quality, robustness, or efficiency. It is commonly used for comparing systems before release or...
Preference Data
Data capturing which outputs humans prefer among alternatives, often used to train ranking models, reward models, or alignment systems. Preference data is...
Process Reward Model
A reward model that evaluates the quality of intermediate reasoning steps or process traces rather than scoring only the final answer. Process reward models...
Ranking Loss
Ranking Loss is a training-time optimization concept that governs how model parameters are updated. It is commonly used for iterative learning loops for neural...
Recall Detail
Recall Detail is an evaluation concept used to measure model quality, robustness, or efficiency. It is commonly used for comparing systems before release or...
Refusal Training
Training or tuning methods intended to improve when and how an AI system refuses unsafe, disallowed, or unsupported requests. Refusal training is part of...
Regularization
Regularization is a training-time optimization concept that governs how model parameters are updated. It is commonly used for iterative learning loops for...
Replay Buffer
A stored collection of past experiences or transitions used in reinforcement learning so training can reuse prior data instead of relying only on the newest...
Reward Signal
The signal used to tell a learning system how good or bad an action, output, or trajectory was. Reward signals drive optimization, so poorly specified signals...
RLHF
Reinforcement Learning from Human Feedback — a training technique where human evaluators rank model outputs by quality, and the model is trained to produce...
ROC Curve
ROC Curve is an evaluation concept used to measure model quality, robustness, or efficiency. It is commonly used for comparing systems before release or model...
Safety Training
Training or tuning intended to improve safe behavior, refusal quality, and compliance with policies or human preferences. Safety training is one layer among...
Sales Enablement
The systems, content, training, and process support that help sales teams perform more effectively and consistently. Good sales enablement shortens ramp time...
Scaling Law
Scaling Law is an AI or ML concept used to represent, train, evaluate, or deploy learned systems. It is commonly used for building production models and...
Self-Supervised
Describing learning methods that use structure already present in unlabeled data to create training signals, rather than relying on externally labeled...
Stochastic Gradient Descent
Stochastic Gradient Descent is a training-time optimization concept that governs how model parameters are updated. It is commonly used for iterative learning...
Streaming Inference
Streaming Inference is the phase where a trained model processes new inputs to produce predictions or generations. It is commonly used for production APIs,...
Supervised
Describing learning that uses labeled examples or explicit targets to teach a model the desired mapping from inputs to outputs. Supervised methods remain...
Synthetic Data
Artificially generated data that mimics the statistical properties of real data without containing actual personal or sensitive information. Used to augment...
Synthetic Data
Artificially generated data that mimics the statistical properties of real data, used when real data is scarce, expensive, or privacy-sensitive. The model...
synthetic training data
Training data generated by AI models rather than collected from humans, used to train or fine-tune other models. While dramatically cheaper to produce,...
Teacher-Student Model
A setup where a larger or stronger teacher model guides a smaller student model during training, often to transfer capability more efficiently.
Tensor
Tensor is an AI or ML concept used to represent, train, evaluate, or deploy learned systems. It is commonly used for building production models and research...
Tensor Core
Tensor Core is an AI or ML concept used to represent, train, evaluate, or deploy learned systems. It is commonly used for building production models and...
TensorFlow
TensorFlow is a machine learning framework for defining, training, and deploying numerical computation graphs. It is commonly used for deep learning research,...
Tensor Parallelism
Tensor Parallelism is a systems strategy for splitting model computation or parameters across devices. It is commonly used for training and serving workloads...
TensorRT
TensorRT is an NVIDIA inference optimization runtime for compiled neural networks. It is commonly used for serving GPU models with lower latency and higher...
Test Set
Test Set is an evaluation concept used to measure model quality, robustness, or efficiency. It is commonly used for comparing systems before release or model...
TPU
TPU is a specialized accelerator designed for large-scale tensor operations. It is commonly used for training and inference workloads with dense linear...
Training Budget
The amount of money, compute, time, or engineering effort available for training or fine-tuning a model. Training budgets constrain how much experimentation...
Training Config
The set of configuration values used for a training run, such as learning rate, batch size, optimizer choice, schedule, and data settings. Training config...
Training Cost
The total cost of training a model, including compute usage, storage, engineering time, data preparation, and supporting infrastructure. Training cost is a...
Training Curriculum
The planned ordering or progression of training examples, tasks, or difficulty levels over the course of learning. Training curricula are used when the...
Training Distribution
The distribution of examples, patterns, and conditions present in the data a model was trained on. A mismatch between training distribution and real-world use...
Training Dynamics
The behavior of the training process over time, including how loss, gradients, representations, and performance evolve during learning. Studying training...
Training Efficiency
How effectively a training setup converts compute, data, and time into improved model performance. Better training efficiency means reaching useful quality...
Training Infrastructure
The hardware, software, orchestration, storage, and operational systems used to run model training. Training infrastructure becomes a major engineering...
Training Loss
The loss measured on training data during model learning, used to indicate how well the model is fitting the current examples according to the objective....
Training Loss Curve
A graph showing how training loss changes over time, used to diagnose convergence, instability, or undertraining.
Training Objective
The objective function or goal that training is trying to optimize, such as next-token prediction, classification accuracy, or reward maximization. The...
Training Pipeline
The full pipeline that prepares data, configures jobs, runs training, evaluates checkpoints, and promotes successful artifacts. Training pipelines help make...
Training Recipe
A practical specification for how to train a model, including data setup, optimizer, schedules, regularization, and engineering tricks that together produce a...
Training Schedule
The plan governing how training progresses over time, such as learning rate schedules, phase transitions, checkpoint timing, or staged dataset exposure....
Training Strategy
The overall approach a team takes to training a model, including data choices, objectives, scaling decisions, adaptation methods, and evaluation criteria....
Triplet Loss
Triplet Loss is a training-time optimization concept that governs how model parameters are updated. It is commonly used for iterative learning loops for neural...
Underfit
To train a model that is too simple or insufficiently learned to capture the underlying pattern in the data.
Upskilling
The process of deliberately building new skills so someone can handle more advanced tools, roles, or responsibilities.
Vanishing Gradient
Vanishing Gradient is a training-time optimization concept that governs how model parameters are updated. It is commonly used for iterative learning loops for...
Warm Start
Starting training or optimization from an existing model, checkpoint, or good prior state instead of beginning from random initialization. Warm starts are...
Weights
The learned numerical parameters of a neural network, adjusted during training to minimize error. A model's weights are its accumulated knowledge — when...

Related Topics