Neural Scaling Laws

Definitions

1. Empirical relationships showing how model performance changes as compute, data, or parameter count increases.

   In plain English: Rules of thumb about how AI improves when you scale model size, data, and compute.

   Example: "Neural scaling laws shaped the argument for training larger models on more tokens."

Related Terms

• RLHF
• Model Collapse
• synthetic training data
• distillation
• Synthetic Data
• Activation Function
• Compute Optimal Training
• Emergent Behavior AI
• Model Distillation Loss