Overfitting

Definitions

1. When a model learns the training data too well, including its noise and accidental quirks, and so performs worse on new data than its training score suggests. The telltale sign is a widening gap between training and validation performance: training error keeps falling while validation error flattens or rises. Overfitting is the high-variance side of the bias-variance trade-off and becomes more likely as model capacity grows relative to the amount of data. Standard countermeasures are more or better data, regularisation such as L1 and L2 penalties, dropout, early stopping, data augmentation, and simpler architectures, with cross-validation used to detect the problem honestly. Its opposite, underfitting, is a model too simple to capture real structure. A subtler cousin is overfitting to a benchmark, where repeated tuning against the same test set leaks it into model selection and inflates reported results.

   In plain English: When an AI gets perfect scores on its practice tests but fails the real exam — it memorized the specific examples instead of learning the underlying patterns.

Related Terms

• Fine-Tuning
• RLHF
• Training Data
• Epoch
• Underfit
• Model Fine-Tuning Cost
• NVIDIA H100
• Out-of-Distribution

Collections Including This Term

• The AI Glossary
• Data Science Decoded