Benchmark Contamination

Definitions

1. When benchmark test data leaks into a model's training set, inflating evaluation scores without genuine capability improvement. A pervasive problem in LLM evaluation — models may 'memorize' benchmark answers rather than demonstrating true reasoning. Mitigated by held-out test sets, live evaluations, and contamination detection.

   In plain English: When an AI gets high test scores because it accidentally saw the test answers during training, not because it's actually smarter.

   Example: "The model scores 95% on MMLU but struggles with rephrased versions of the same questions — likely benchmark contamination."

Related Terms

• AI Benchmark
• AI Fairness
• AI Hallucination
• AI Model Card
• Anomaly Detection ML
• Binary Classification
• Classification Threshold
• Conditional Generation
• Conformal Prediction
• LLM-as-Judge
• Needle in a Haystack Test
• Perplexity Score
• MMLU Benchmark