Benchmark Contamination: Why LLM Scores Can Lie

Benchmark contamination is one of the most consequential and least discussed problems in LLM evaluation. When a model scores 90% on a test it may have partially memorised during pretraining, that number tells you far less than it appears to.

What contamination is and how it happens

Large language models are trained on web-scale corpora scraped from the open internet. Many popular benchmarks — MMLU, HumanEval, GPQA — have their questions, answers, and answer explanations publicly available. If any of that material appears in training data, the model can reproduce correct answers through memorisation rather than reasoning. This is contamination: the test set leaks into training.

Contamination does not require deliberate cheating. A pretraining crawl from 2024 will naturally include Stack Overflow posts that discuss HumanEval solutions, Reddit threads that debate GPQA answers, and academic papers that reprint MMLU questions verbatim. The model never "studied for the test" in any intentional sense, but the effect on scores is the same.

How memorisation inflates scores

The inflation mechanism is subtle. A contaminated model does not simply output a memorised answer string — it learns the statistical association between a question pattern and a correct answer. On questions it has seen, it will confidently pick the right option even when the underlying reasoning chain is missing. The score goes up. Generalisation does not.

Researchers have quantified this. Studies on GPT-class models found accuracy jumps of 5-15 percentage points on benchmarks with high web presence compared to equivalent benchmarks constructed from private or paywalled sources. That gap is the contamination premium — it looks like capability but is actually recall. See our guide to reading benchmark scores for the full checklist of red flags.

How labs detect and mitigate contamination

Detection approaches fall into two categories. The first is n-gram overlap: check whether benchmark questions appear verbatim or near-verbatim in training data. This catches the most obvious cases but misses paraphrase. The second is behavioural: compare model accuracy on publicly available benchmark questions versus equivalent private questions that test the same skill. A large accuracy gap is a contamination signal.

Mitigation strategies include: holding out benchmarks entirely from training data scrapes, using private or dynamically generated question sets, and performing regular contamination audits before publishing scores. Some labs now publish contamination analyses alongside results — treat the absence of such analysis as a caveat, not a clean bill of health.

Why fresh and private benchmarks matter

The cleanest solution is a benchmark that did not exist when training data was collected. Humanity's Last Exam crowdsourced questions from domain experts specifically to create novel, hard-to-memorise items. SWE-bench Pro draws from repositories and issues filed after major model training cutoffs, reducing the chance that solutions appear in any pretraining corpus. BrowseComp evaluates live web search, so static memorisation provides little advantage.

Private benchmarks — where the test questions are never released — go further. If you cannot scrape the questions, you cannot train on them. The tradeoff is reduced reproducibility: independent researchers cannot verify the score. The field is still negotiating the right balance. For your own evaluation needs, see our guide to evaluating an LLM for your use case, which covers building private eval sets. The broader context is laid out in the complete guide to LLM benchmarks.

Key takeaways

• Contamination inflates scores by letting models recall rather than reason.
• It happens passively through web scraping, not only deliberate data inclusion.
• N-gram overlap checks and private-set comparisons are the main detection tools.
• Fresh, post-cutoff, or private benchmarks are the most contamination-resistant.
• When a lab publishes no contamination analysis, treat its scores with extra scepticism.