Training Dynamics

Definitions

1. The behavior of the training process over time, including how loss, gradients, representations, and performance evolve during learning. Studying training dynamics helps teams diagnose instability, overfitting, or stalled progress.

   In plain English: How the training process changes over time.

   Example: "The researchers tracked training dynamics to understand why the model improved quickly early on and then plateaued."

Related Terms

• distillation
• Expert Iteration
• Gradient-Free Optimization
• Inference Optimization
• Neural Network Pruning
• Next Token Prediction
• Noise Injection
• Objective Function
• Score Function