Benchmark Contamination: Why Your Eval Scores Lie

What you need to know

Two teams are choosing a model. A Bengaluru fintech needs to extract obligations from vendor contracts and answer customer queries in Hindi and Tamil as well as English. A Manchester health-tech company needs to triage clinical correspondence under UK information-governance constraints. Both start the same way: open the two candidate vendors' release posts, compare the benchmark tables, pick the higher number.

That process is more fragile than it looks, and one of the biggest reasons is data contamination — test items leaking into the training data. When an evaluation item has already appeared somewhere in a model's pre-training corpus, the model can answer it partly from memory rather than by genuinely generalising, and the reported score goes up without the underlying capability going up with it. The contamination literature is explicit about this mechanism: leakage inflates reported benchmark performance by inducing memorisation rather than genuine generalisation.

This is not a niche worry about obscure test sets. The benchmarks named repeatedly in the contamination literature are the ones everybody quotes: MMLU, GSM8K, MATH and HumanEval. One Johns Hopkins-led analysis reported that 29.1% of MMLU test items showed signs of contamination. Treat that as a single study's finding rather than a settled consensus figure — methodologies for detecting contamination differ substantially and produce different numbers — but even taken conservatively it says the problem is material rather than marginal. An arXiv line of work separately reports contamination of major mathematics benchmarks across 31 models, which tells you this is a property of the corpus ecosystem rather than of any one training run.

Three things follow, and the rest of this guide unpacks them.

• A public leaderboard number is not evidence about your task. It is evidence about a benchmark, under unknown contamination conditions, on a distribution that is almost certainly not yours.
• You can detect contamination approximately, without training-data access. Clean-mirror testing, overlap checks and perplexity-gap probes will not give you certainty, but they will tell you whether a score deserves your trust.
• The durable fix is a private eval set you never publish. Everything else is mitigation; this is the actual solution, and it is well within the reach of a five-person team.

One framing point before we go further, and it matters. Contamination is overwhelmingly structural and unintentional. Web-scale pre-training corpora sweep up forums, tutorial blogs, GitHub issues, Kaggle notebooks, Stack Exchange threads and dataset mirrors. Benchmark items circulate through all of those channels. No deliberate misconduct is required for a test item to end up in a training set, and nothing in this guide should be read as an allegation that any particular lab has contaminated any particular benchmark. The interesting question is not who is to blame; it is what you should do about it.

What contamination actually is

"Contamination" gets used as a single word for at least four distinct problems, and conflating them is why teams reach for the wrong remedy. Separating them is the first useful step.

Verbatim test-item leakage

The clean case: the exact question, and often the exact answer, appears somewhere in the pre-training corpus. A GSM8K word problem posted on a tutorial site with its worked solution; an MMLU question quoted in a revision guide; a HumanEval function with its canonical implementation sitting in a public repository. The model does not need to reason; it needs to recall. This is the form most people picture, and paradoxically it is the easiest to detect and the least common in absolute terms.

Near-duplicate and paraphrase leakage

The same problem in different clothes. The numbers are changed but the structure is identical; the wording is paraphrased but the reasoning chain is the same; the item has been translated and re-posted. Exact-match decontamination filters — the n-gram overlap checks that dataset builders typically run — miss this entirely, which is precisely why a benchmark can be described as decontaminated and still be inflated. This is the form that quietly does the most damage.

Task-format leakage

Nobody has seen your specific item, but the model has seen many thousands of items in exactly that shape: four-option multiple choice with a particular preamble, grade-school arithmetic word problems with a two-step structure, docstring-to-function coding prompts. The model has learned the format's regularities — including which answer positions are statistically favoured and which distractor styles are usually wrong. That is real skill of a sort, but it is skill at the benchmark, not at your task, and it does not transfer to a differently shaped production workload.

Post-hoc leakage

The slowest and most inevitable form. A benchmark is released clean. Researchers discuss individual items in papers. Practitioners paste failing cases into blog posts, GitHub issues and forum threads to ask why a model got them wrong. Evaluation harnesses log examples into public CI output. Two years later, a crawler sweeps all of it into the next pre-training corpus. Every widely used public benchmark is on a one-way trip from clean to contaminated, and the more useful a benchmark is, the faster it makes that trip.

Leakage type	How it inflates the score	How detectable it is
Verbatim test-item leakage	Direct recall of question and answer; no reasoning required	Highest — n-gram and substring overlap checks catch it if you can reach the corpus
Near-duplicate / paraphrase	Recall of the reasoning template with surface variation; looks like generalisation	Moderate — needs embedding-similarity or clean-mirror testing; exact-match filters miss it
Task-format leakage	Learned priors about the format, including answer-position and distractor regularities	Low — shows up as order and format sensitivity rather than as overlap
Post-hoc leakage	Items enter later corpora via blogs, issues and forum threads discussing them	Very low from outside — depends on crawl timing you cannot observe

The research community has responded on several fronts, and it is worth knowing the names because they turn up in release posts. On the mitigation side, inference-time decontamination tries to recover an honest score from a contaminated benchmark without retraining: the literature names TED (Dong et al., 2024), which calibrates outputs via repeated sampling, and ITD (Zhu et al., 2024), which rewrites prompts using auxiliary models. The paper "When Benchmarks Leak: Inference-Time Decontamination for LLMs" sits in this line of work. On the detection side, CoDeC is a contamination-detection method accepted at ICLR 2026; membership inference attacks (MIA) originally developed for privacy research have been adapted to detect pre-training data membership; and kernel-divergence methods are proposed for quantifying leakage, as in "How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergence". A third strand argues the fix belongs upstream in benchmark design itself — see "LLM Benchmark Datasets Should Be Contamination-Resistant" — while "NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark" makes the case that contamination measurement should be a standard, per-benchmark reporting obligation rather than an occasional research exercise.

Why this is your problem, not just a research problem

It is tempting to file contamination under "interesting but academic". Three very ordinary situations say otherwise.

Model selection. You are choosing between two models whose published scores differ by a few points on a shared benchmark. If contamination affects the two models differently — and there is no reason it would affect them equally, given different corpora, different cutoffs and different filtering — then that gap carries no information about which will serve your users better. You have made a real decision, with real switching costs, on a difference that may be entirely artefactual.

Vendor and procurement comparisons. Benchmark tables travel into procurement documents, board decks and, increasingly, contractual annexes. A UK health-tech team justifying a model choice to an information-governance committee, or an Indian lender documenting model selection for an internal risk review, needs the justification to hold up under scrutiny. "It scored higher on MMLU" is a weak foundation when a single study reports 29.1% of MMLU test items showing signs of contamination.

The upgrade that made things worse. This is the incident pattern that sends teams looking for this article. A newer model with better published numbers goes into production and quality drops — on the long-tail intents, the code-switched Hindi-English queries, the region-specific regulatory language. Public benchmarks do not measure any of that. Their headline numbers may additionally be inflated by leakage. The only instrument that would have caught the regression before users did is an evaluation set built on your own traffic, ideally run behind a staged rollout; our guide to shadow and canary deploys for model upgrades covers the rollout half of that pairing.

There is a second, independent reason the public numbers cannot answer your question, and it applies even to a perfectly clean benchmark. The widely quoted suites are overwhelmingly English-centric and general-domain. They contain essentially nothing about Indian-language performance in Hindi, Tamil, Bengali or Marathi, nothing about code-switched input, nothing about UK regulatory phrasing, NHS correspondence conventions or FCA-adjacent language. A Bengaluru fintech and a Manchester clinic are both operating in distributions that the public leaderboard has never sampled. Contamination is a reason to distrust the number; distributional mismatch is a reason the number was never about you in the first place.

Detection you can actually run

Almost no one outside a frontier lab can inspect a pre-training corpus. Everything below is designed for teams without that access. None of it yields proof; all of it yields evidence, and evidence is enough to decide whether a published number should influence your decision.

Clean-mirror testing — the highest-value technique

This is the one to run first. Take a sample of benchmark items and rewrite each into a mirror: an item that preserves the reasoning difficulty exactly while changing every surface feature a memoriser could latch onto. Then compare scores on originals against mirrors. Genuine capability transfers; memorisation does not.

The technique has real precedent. Reporting on this line of work describes one case where contaminated items were swapped for clean mirrors and re-solved, with a Mistral model dropping by as much as 13 percentage points on a clean GSM8K test. That is a single reported case rather than a general law — the specific model version is not something we can state here, and the size of any such gap will vary by model, benchmark and mirroring method — but it illustrates the scale of correction that mirroring can surface.

Concrete perturbation recipes, in rough order of how much they cost you:

• Rename entities. Change every proper noun: people, companies, cities, products. "Priya" becomes "Aisha", "Manchester" becomes "Coimbatore", "Acme Ltd" becomes something else again. Cheap, and it breaks verbatim matching immediately.
• Change the numbers. For quantitative items, alter every figure while keeping the arithmetic structure identical, and recompute the answer. Critically, keep the difficulty band the same — do not turn a three-digit multiplication into a two-digit one, or you have measured your own perturbation rather than the model.
• Reorder options. For multiple-choice items, permute the answer options and re-index the key. This directly probes the answer-position priors that task-format leakage produces.
• Translate and back-translate. Round-trip the item through another language to force a paraphrase that preserves meaning while destroying surface form. Have a human check that the difficulty survived; back-translation sometimes simplifies phrasing in ways that flatter the model.
• Restructure the frame. Turn a multiple-choice item into a free-response one, or invert the question so the answer becomes a given and one of the givens becomes the unknown. The most powerful perturbation and the most labour-intensive.

Two disciplines make or break this method. First, a human must verify that difficulty is preserved — an automated rewrite that quietly makes items easier or harder produces a score gap that says nothing about contamination. Second, run enough items and report an interval. A five-point difference on forty items is noise. The labelling discipline this requires is the same one described in our guide to building a human annotation pipeline for evals and golden sets, and it is worth borrowing wholesale.

Canary strings and never-published held-out sets

A canary is a rare, unique string embedded in a document you control — a random 32-character token, say — which no natural corpus would contain. If a model can reproduce or complete that string, the document reached its training data. Canaries are how dataset publishers mark their own material, and they are worth planting in anything of yours that goes public and might later be crawled.

The stronger version is simply not publishing at all. A held-out set that has never existed in any public location cannot have leaked, and that guarantee is stronger than any detection method. This is why the private eval set in the next section is the main recommendation of this guide rather than an afterthought.

N-gram and near-duplicate overlap checks

You cannot search a frontier lab's corpus, but you can check your own eval items against whatever public corpora you can reach: open pre-training datasets, documentation you have crawled, your own public repositories, publicly archived support content. Any hit is a straightforward disqualification for that item.

# Illustrative only. n-gram overlap between eval items and a reachable corpus.
from collections import defaultdict
import re

def ngrams(text: str, n: int = 13) -> set[str]:
    toks = re.findall(r"[a-z0-9]+", text.lower())
    return {" ".join(toks[i:i + n]) for i in range(len(toks) - n + 1)}

def build_index(corpus_docs, n=13):
    """corpus_docs: iterable of (doc_id, text) for any corpus you can reach."""
    index = defaultdict(set)
    for doc_id, text in corpus_docs:
        for g in ngrams(text, n):
            index[g].add(doc_id)
    return index

def contamination_report(eval_items, index, n=13, threshold=0.10):
    """Flags an eval item when >= threshold of its n-grams appear in the corpus."""
    flagged = []
    for item in eval_items:
        grams = ngrams(item["prompt"] + " " + item["answer"], n)
        if not grams:
            continue
        hits = {g for g in grams if g in index}
        ratio = len(hits) / len(grams)
        if ratio >= threshold:
            sources = set().union(*(index[g] for g in hits))
            flagged.append({"id": item["id"], "overlap": round(ratio, 3),
                            "sources": sorted(sources)[:5]})
    return sorted(flagged, key=lambda r: -r["overlap"])

Be honest about the limits. A 13-gram exact-match check catches verbatim leakage and nothing else. Paraphrases sail straight through, which is why an embedding-similarity pass over the same pair of sets is a useful complement — and why clean-mirror testing remains the higher-signal method.

A perturbation harness

Mirroring becomes tractable when you wire it as a harness rather than doing it by hand each time. The sketch below runs originals and mirrors through the same scorer and reports the gap.

# Illustrative only. Perturbation harness: original vs clean-mirror scoring.
import statistics
from dataclasses import dataclass

@dataclass
class Item:
    id: str
    prompt: str
    answer: str
    mirror_prompt: str | None = None   # human-verified rewrite
    mirror_answer: str | None = None

def score_set(model, items, use_mirror: bool) -> list[int]:
    results = []
    for it in items:
        prompt = it.mirror_prompt if use_mirror else it.prompt
        gold   = it.mirror_answer if use_mirror else it.answer
        if prompt is None:
            continue
        pred = model.complete(prompt)
        results.append(int(normalise(pred) == normalise(gold)))
    return results

def memorisation_gap(model, items, runs: int = 5):
    """Repeat both arms; a stable gap is the signal, one run is not."""
    orig, mirr = [], []
    for _ in range(runs):
        orig.append(statistics.mean(score_set(model, items, use_mirror=False)))
        mirr.append(statistics.mean(score_set(model, items, use_mirror=True)))
    gap = statistics.mean(orig) - statistics.mean(mirr)
    spread = statistics.pstdev(orig) + statistics.pstdev(mirr)
    return {"original": round(statistics.mean(orig), 4),
            "mirror":   round(statistics.mean(mirr), 4),
            "gap":      round(gap, 4),
            "run_spread": round(spread, 4),
            "verdict": "investigate" if gap > 2 * spread else "inconclusive"}

Note the deliberately cautious verdict rule: a gap only counts when it exceeds the run-to-run spread by a comfortable margin. Sampling variance across runs is large enough to manufacture convincing gaps out of nothing, and a contamination finding is exactly the kind of result people quote without checking how it was produced.

Membership-inference-style perplexity probes

The intuition behind adapting membership inference attacks to contamination detection is simple: a model tends to assign lower loss — higher confidence — to text it has seen during training than to comparable text it has not. So if your suspected-seen benchmark items score systematically lower perplexity than a control set of known-unseen items matched for domain, length and difficulty, that asymmetry is consistent with membership.

# Illustrative only. Perplexity-gap probe (needs logprob access from the model API).
import math, statistics

def sequence_logprob(model, text: str) -> tuple[float, int]:
    """Returns (sum of token logprobs, token count)."""
    r = model.score(text)           # provider-specific logprob endpoint
    return sum(r.token_logprobs), len(r.token_logprobs)

def perplexity(model, text: str) -> float:
    total, n = sequence_logprob(model, text)
    return math.exp(-total / max(n, 1))

def membership_probe(model, suspected_seen: list[str], known_unseen: list[str]):
    """known_unseen MUST be matched on domain, length and difficulty, and MUST
    post-date the model's training cutoff. Otherwise this measures nothing."""
    seen   = [perplexity(model, t) for t in suspected_seen]
    unseen = [perplexity(model, t) for t in known_unseen]
    return {
        "median_ppl_suspected": round(statistics.median(seen), 3),
        "median_ppl_control":   round(statistics.median(unseen), 3),
        "ratio": round(statistics.median(seen) / statistics.median(unseen), 3),
        "note": "ratio well below 1.0 is suggestive, not probative",
    }

Now the honest caveats, because this probe is easy to over-read. Membership inference is weak against very large models trained for few epochs on enormous corpora — the memorisation signal per item is faint. It requires logprob access, which many hosted APIs do not expose. It is exquisitely sensitive to the quality of the control set: if your controls are even slightly simpler, shorter or more idiosyncratic than the suspected items, you will measure that difference and call it contamination. And the whole approach is confounded by ordinary distributional familiarity — a model may assign low perplexity to well-written textbook prose because it has read a great deal of well-written textbook prose, not because it read that item. Use it as a corroborating signal behind clean-mirror testing, never as a headline claim.

Order and format sensitivity as a memorisation smell

The cheapest diagnostic in the toolkit. Permute the answer options in a multiple-choice set and re-score. Change the answer labels from A/B/C/D to 1/2/3/4. Move the question above the passage instead of below it. Genuine understanding is largely invariant to these; benchmark-shaped pattern-matching is not. Large swings under trivial reformatting tell you the score is riding on format priors — which is task-format leakage doing exactly what the table above predicts. This check costs one afternoon and belongs in every model evaluation you run.

Building a contamination-resistant private eval set

Everything above is diagnosis. This section is the cure, and it is the single highest-return investment an applied team can make in evaluation. A private eval set built from your own production data cannot have leaked into a model trained before you built it, because it did not exist anywhere a crawler could reach.

The research community reaches for the same principle when it needs a genuinely clean measurement. One zero-leakage design in the literature is elegant in its simplicity: collect the 2026 Korean CSAT mathematics examination immediately upon public release and evaluate models straight away, so that no model could have encountered the items during training. Your private eval set is the same idea, applied continuously — you are manufacturing a fresh, unleaked exam out of your own traffic.

Source items from your production logs

Real user queries beat synthetic ones on every axis: they carry the actual distribution, the actual ambiguity, the actual messiness. A Bengaluru fintech's set will contain code-switched Hindi-English queries, abbreviations specific to Indian banking, and PAN and GST formats. A Manchester health-tech set will contain NHS terminology, UK date conventions and referral phrasing. Neither appears in any public benchmark, and both are exactly what determines whether the deployment works. Stratify deliberately: sample across intents, languages, input lengths and — importantly — across the failures you already know about. Our guide to error analysis and building evals from production logs covers that sampling discipline in detail, and it is the natural companion to this section.

Redact before anything else. Production logs contain personal data, and an eval set is a long-lived artefact that gets copied into notebooks, CI runners and analytics tools. Strip or pseudonymise identifiers at ingestion, under DPDP obligations in India and UK GDPR in Britain, and keep the redaction step inside the pipeline rather than as a manual pre-processing habit that someone will eventually skip.

Never publish it

This is the rule the whole thing rests on, and it is the one most often broken with the best intentions. Publishing your eval set to demonstrate rigour destroys it: the moment it is crawled it becomes a future contaminated public benchmark, and you cannot un-publish from a training corpus. Publish the methodology — sourcing, labelling protocol, rubric, harness, sample size, intervals — and keep the items private. If an external party genuinely needs verification, run the evaluation on their behalf under an agreement, or release a small, deliberately sacrificial subset while keeping the scoring set intact.

The same discipline extends to daily habits, and this is where sets die quietly. Do not paste eval items into any chat interface. Not into a consumer assistant to "check what the right answer is", not into a code assistant to "help me write the grader", not into a shared document that syncs to a public workspace. Consumer chat products may retain and train on conversations under some settings, and an eval item pasted into one is an eval item you can no longer trust. Write this into your team norms explicitly, because otherwise the highest-value engineers are exactly the ones most likely to break it out of curiosity.

Dated collection windows and a rotating holdout

Stamp every item with the date it entered the set. Dated windows let you make an argument you otherwise cannot: "these 200 items were collected in the quarter beginning April 2026, after the training cutoff claimed for both candidate models". That is a real, checkable cleanliness claim, and it is why collection metadata is not bureaucracy.

Then rotate. Split the set into an active portion you evaluate against routinely and a sealed holdout that is opened rarely — before a major model migration, say. Frequent evaluation against a fixed set produces a subtler contamination of its own: you gradually tune prompts, retrieval and post-processing to that specific set until the score reflects your fitting rather than your capability. The sealed holdout is what tells you whether the active-set gains were real. Refresh the active portion on a schedule — a fixed share replaced each quarter — so it tracks a drifting user population.

Version it, control access to it, and wire it into CI

An eval set is a production artefact and deserves production treatment: semantic versioning, a changelog explaining why each item was added, retired or relabelled, and immutable historical versions so a score from six months ago remains interpretable. Store it in a private repository or an access-controlled bucket with a small named group of people who can read it. Log who ran what against which version.

Once versioned, wire it into your pipeline so it runs automatically on prompt changes, retrieval changes and model upgrades rather than only when someone remembers; our guide to running evals in CI for prompt and agent regression testing covers the mechanics. If the set includes open-ended outputs graded by a model, calibrate the grader before you trust it, because a miscalibrated judge will happily undo all the care you took over contamination. For the broader assembly job, our guide to building an LLM evaluation suite with golden sets and judges is the umbrella piece this one sits under. Teams evaluating coding assistants have an unusually good version of this available to them, since their own repository is a private benchmark nobody else can have trained on, and that approach generalises well beyond code.

Dimension	Public benchmark	Private golden set
Cost to obtain	Near zero — download and run	Real — sourcing, redaction, labelling, ongoing maintenance
Contamination risk	High and rising; every published benchmark decays towards leakage over time	Near zero while it stays unpublished and access-controlled
Comparability across vendors	Strong — everyone quotes the same suites, which is their genuine value	Only within your own runs; you cannot cite it in a public comparison
Usefulness for a shipping decision	Weak — wrong distribution, unknown leakage, no coverage of your languages or domain	High — measures the task you actually ship, in the languages you actually serve
Shelf life	Degrades as the benchmark saturates and leaks	Improves with age as coverage and labelling quality accumulate

Read the table as complementary rather than competitive. Public benchmarks answer "is this model roughly in the right class?" — a coarse screen worth keeping. Private sets answer "should we ship this?" Use the first to shortlist and the second to decide, and never let the first override the second.

Reading other people's benchmark claims

You will still read release posts, and you should — you simply need to read them the way an auditor reads a set of accounts. Six questions, in the order worth asking them.

• Is the benchmark version stated? Benchmarks are revised, subsetted and re-annotated. "MMLU" without a variant, or a coding benchmark without saying which split, is not a reproducible claim. Our news analysis of the gap between SWE-bench Verified and SWE-bench Pro is a good illustration of how much the variant matters.
• Was decontamination described, and how? The word "decontaminated" is nearly meaningless on its own. What was the method — exact match, n-gram overlap at what value of n, embedding similarity at what threshold? What was removed, and how many items? A vendor who reports the number of items dropped is telling you something; one who uses the word and moves on is not.
• Is this the base model or a fine-tuned variant? Instruction-tuning and post-training introduce their own contamination surface, and a score from a heavily tuned variant tells you little about the checkpoint you will actually be served through the API.
• Is the evaluation harness public? Prompt formatting, few-shot count, answer extraction and scoring rules move numbers by several points on identical models. Without the harness, you cannot compare two vendors' figures on the same benchmark at all, because you do not know they measured the same thing.
• Are error bars or confidence intervals given? A point estimate with no interval, on a benchmark of a few hundred items, is a number that cannot support a comparison. Sampling variance alone spans several points at those sizes.
• Is there any contamination analysis at all? The argument in "NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark" is that per-benchmark contamination measurement should be routine reporting. Publications that include such an analysis, or that report clean-mirror results alongside originals, have earned more trust than those that do not — reward them accordingly.

The same checklist applies to your own reporting. If you publish evaluation results — in a case study, a conference talk, a customer-facing document — state the harness, the sample size, the intervals and the collection dates. Frameworks such as "ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems" exist partly to make this kind of reporting reproducible, and adopting a named framework makes your numbers easier for someone else to trust.

Common pitfalls, with symptom and fix

Seven failure patterns that recur, each with the symptom that gives it away.

• Publishing your eval set to demonstrate rigour. Symptom: a repository named evals-public, or a blog post listing example items. Fix: publish the methodology, the rubric and the aggregate results; keep the items private. If you have already published, treat that set as compromised for any model trained afterwards and rebuild the scoring set from fresh production data.
• Assuming a post-cutoff benchmark is automatically clean. Symptom: "this benchmark was released after the model's cutoff, so contamination is impossible." Fix: treat cutoffs as claims about a corpus snapshot, not guarantees. Models get updated and fine-tuned; retrieval and tool use reach live content; new benchmarks frequently recycle items from older sources. A post-cutoff release lowers verbatim-leakage risk and does nothing about near-duplicate or task-format leakage.
• Treating a single perturbed run as significant. Symptom: a Slack message reading "mirrored the items, score dropped six points, it is contaminated." Fix: repeat both arms, report the interval, and require the gap to exceed run-to-run spread by a clear margin — the rule encoded in the harness above. Have a second person confirm the mirrors preserved difficulty before anyone quotes the number.
• Over-correcting into an eval set that no longer resembles your users. Symptom: a set full of adversarial, deliberately weird items that nobody in production would ever send, on which every model scores badly and the ranking never changes. Fix: keep the set anchored to sampled production traffic. Adversarial items belong in a separate, clearly labelled robustness suite that is scored and reported separately, never blended into the headline metric.
• Relying on exact-match decontamination alone. Symptom: an n-gram filter run once, no paraphrase check, and a confident claim of cleanliness. Fix: layer the methods — exact overlap, then embedding similarity, then clean-mirror testing on a sample. Each catches what the previous one misses, and only the last catches paraphrase leakage.
• Optimising against the same eval set indefinitely. Symptom: the active-set score climbs steadily for months while user-reported quality stays flat. Fix: the sealed rotating holdout. If active-set gains do not reproduce on the holdout, you have been fitting the set rather than improving the system.
• Leaking your own set through everyday tooling. Symptom: eval items appearing in CI logs, error-tracking payloads, analytics events or a screenshot in a status update. Fix: redact eval content from logs, keep the runner's output to identifiers and scores rather than item text, and audit which third-party services see evaluation payloads. This is the most common way a private set stops being private.

Where to start

If you take one action from this guide, make it the private set: 200 items from your own production traffic, redacted, labelled, dated, versioned, never published. It will tell you more about a model migration than every public benchmark combined, and unlike a leaderboard number it gets more valuable each quarter you maintain it.

If you have a fortnight, run the sequence in order. Week one: sample and label the private set, and run the order-and-format sensitivity check against your current model as a free baseline. Week two: mirror fifty items from whichever public benchmark your vendor comparison leans on, run both arms five times, and see whether the gap survives the interval. Then wire the private set into CI so it runs on every prompt and model change, and add the reading checklist to whatever document your team uses to evaluate vendor claims.

As of July 2026, contamination remains an open research problem with active work on all fronts — inference-time decontamination via TED and ITD, detection methods such as CoDeC accepted at ICLR 2026, kernel-divergence quantification, membership-inference adaptations, and the argument that benchmarks should be contamination-resistant by design. That work matters and is worth following. But none of it changes what an applied team should do, and the applied answer has been stable for a while: stop asking whether the leaderboard is honest, and start measuring the thing you actually ship. For reasoning-heavy workloads where the output is a process rather than an answer, our guide to evaluating reasoning models on process, outcome and chain-of-thought faithfulness is the natural next read.