benchmark contamination

Definitions

1. The phenomenon where a model's training data inadvertently (or deliberately) includes examples from evaluation benchmarks, inflating its apparent performance without genuine capability improvement. A persistent methodological challenge that undermines the credibility of leaderboard comparisons.

   In plain English: When an AI gets a high test score because it accidentally saw the answers during training, not because it actually learned the material.

   Example: Their SOTA claim fell apart when researchers discovered benchmark contamination — the model had memorized 40% of the test set verbatim.

Related Terms

• Hallucination
• Perplexity
• Benchmark
• grounding
• guardrails
• F1 score
• GLUE Benchmark
• AI Safety Benchmark
• Eval Suite
• MMLU Benchmark
• Benchmark Saturation