Research checked October 31, 2026. This is a technical analysis of published evaluation methods, not a new audit of any commercial model. Numerical examples and proposed experiments are hypothetical. Dates refer to the identified papers rather than an invented breaking-news event.
The test has a history
A language model receives a mathematics question and returns the correct answer. The evaluator records a success. On the surface, the measurement is simple. The difficulty begins before the question reaches the model: did some version of this problem already appear in its training material, together with its answer, solution, or surrounding discussion? If so, what does the success demonstrate about the model's ability to solve a different problem tomorrow?
The issue is particularly visible in public benchmarks, whose questions must be accessible enough for researchers to inspect and reproduce results. Publication makes scrutiny possible, but it also makes the material available to future data-collection pipelines. An evaluation dataset can become training material without anyone deliberately deciding to cheat. A benchmark question may be quoted in a tutorial, translated into another language, embedded in a repository, or reproduced in generated practice material.
The revised LiveBench paper provides a useful dated starting point. Initially submitted in June 2024 and revised in April 2025, it describes frequently refreshed questions from recent sources and uses the title “Contamination-Limited.” That wording is more informative than an unconditional promise that any public benchmark can remain permanently untouched by training. The paper's design is evidence of a mitigation strategy, not proof that every later model has never encountered any related material. LiveBench paper.
The public-interest question is not whether benchmarks should be abandoned. It is what a particular score can support. A benchmark may still reveal useful differences even when exposure is uncertain, but the uncertainty belongs beside the result. An evaluation that claims generalization needs stronger evidence than an evaluation that merely reports performance on a familiar public collection. Those are different scientific questions, and they should not be compressed into the same promotional sentence.
Define the exposure before debating its effect
Contamination is often discussed as a yes-or-no property of an entire model. A more useful analysis identifies the relationship between specific evaluation items and specific training material. Exact question-and-answer duplication is one relationship. A paraphrased question with the same solution is another. A general textbook explanation of the underlying principle is different again. Learning a principle is normally the point of training; memorizing the test's answer key threatens a claim that the test measures performance on unseen questions.
Consider a hypothetical arithmetic item asking how many seats remain after several groups board a bus. Training on arithmetic and word problems is legitimate preparation. Training on that exact bus problem and its answer creates a different evidentiary condition. Changing the passengers' names may not remove the relevant exposure. Changing the relationships between quantities may create a genuinely new problem, but it may also alter difficulty. An evaluator must say which kind of novelty is required.
Exposure also has timing. Material could enter pretraining, supervised fine-tuning, preference optimization, a retrieval database, or a test-time tool result. The consequences for interpretation differ. A model that retrieves a public answer during an explicitly open-book evaluation may be following the rules. The same retrieval during a claimed closed-book reasoning test would undermine the stated protocol. The word contamination is less helpful than a description of the permitted and prohibited information paths.
A precise audit therefore starts with the claim: unseen exact items, unseen item families, unseen solutions, or unseen subject matter? No single test can establish all four automatically. The evaluator should define an exposure taxonomy before examining results, because redefining the category after seeing an inconvenient overlap invites selective reporting. The taxonomy need not be elaborate. It must be explicit enough that two reviewers can explain why a matching document counts, does not count, or remains ambiguous.
A match is evidence with a denominator
Suppose a hypothetical audit searches a training corpus and finds exact text from fifty of a benchmark's one thousand questions. The immediate conclusion is that fifty questions have identified overlaps under the search procedure. It does not follow that the other nine hundred fifty were never encountered. The corpus supplied to the auditor might be incomplete, or a transformed version might evade the matching rule. Nor does it follow that all fifty matches affected the model's answer.
The denominator needs several labels. How many benchmark items were searchable? How much of the training corpus was available? Were documents decoded successfully? Did the search include fine-tuning data and synthetic material? Were source code, tables, scanned documents, and translated text represented? A claim of “five percent overlap” is difficult to interpret without knowing whether it means five percent of all items, all processed items, or all items with accessible text.
Matching rules matter too. An exact match after punctuation normalization is not identical to a match after removing numbers, names, and answer choices. The latter may detect more related problems but also identify commonplace templates. A report should preserve examples of the transformation so readers can understand what was treated as equivalent. It should not expose private training records unnecessarily, but it should provide enough evidence to assess the categorization.
The recommended unit of reporting is an item-level record with a match type, source identifier, transformation method, and review outcome. Aggregate statistics can then be reconstructed rather than accepted as a mysterious summary. Where access restrictions prevent public release, an independent reviewer may inspect the underlying evidence under appropriate controls. That arrangement is weaker than unrestricted reproducibility in some respects, but it is stronger than a percentage with no described inspection route.
Why strings do not capture every repeated problem
Yang and colleagues' November 2023 study examines contamination through rephrased samples. It reports that paraphrasing and translation can bypass string-based decontamination, and it identifies risks involving synthetic datasets as well. These are findings within the paper's experimental settings; they do not establish a universal contamination percentage across current models. Rephrased-sample study.
The underlying measurement challenge can be understood with a hypothetical geometry question. One version asks for the area of a rectangle with given dimensions. Another describes a rectangular garden using the same dimensions and requests the amount of turf. A literal search may find little overlap. Yet the mathematical operation and answer are identical. Whether this constitutes problematic contamination depends on the benchmark's intended novelty and the specificity of the match.
At the opposite extreme, two unrelated questions may share long phrases because both use a standard exam template. A string detector can mark them as duplicates even though the quantities and reasoning differ. Increasing the matching threshold reduces some false positives but may miss meaningful paraphrases. Lowering it increases sensitivity while requiring more review. There is no threshold that resolves every semantic ambiguity merely by being more conservative.
A useful audit separates candidate generation from adjudication. Broad automated matching can surface possible relationships. A second stage can examine whether the matched material contains the same answer-bearing structure. That review should record uncertainty rather than force every candidate into a binary category. Where semantic similarity tools are used, the report should identify their limitations and validate a sample of both accepted and rejected candidates.
The important lesson is not that exact matching is worthless. It is a reproducible and often valuable first layer. The mistake is treating the absence of literal overlap as proof that no relevant information crossed the training-test boundary. A detector's output describes what that detector found under its assumptions, not the entire history of the model.
The counterfactual hidden inside an accusation
To say that contamination inflated a score is to make a causal claim. The claim compares the observed model with a counterfactual version trained without the relevant exposure. That counterfactual is usually unavailable for a commercial model. Finding overlap supports concern, but it does not by itself measure how many answers would have changed if the overlapping documents had been removed.
A controlled research experiment can approximate the question by training comparable models with different exposure conditions. One group receives ordinary training data; another also receives a specified set of benchmark items. The experiment should vary exposure dose, formatting, and whether answers are included. It should evaluate both the exposed items and genuinely separate material. If performance rises only on the exposed set, the interpretation differs from an improvement that transfers broadly.
Even this design has complications. Training randomness, data ordering, optimization settings, and compute can affect outcomes. Removing documents changes the data mixture and sometimes the number of training steps. Replacing removed material with unrelated text introduces its own treatment. The experiment should explain what was held constant and what necessarily changed. Multiple seeds can help characterize variability, although they do not erase systematic differences between conditions.
For reporting, distinguish three claims: exposure was identified; exposure produced an effect in a controlled experiment; exposure explains a particular deployed model's score. Evidence for the first two does not automatically prove the third. The final claim may remain plausible but unmeasured when the actual training process is inaccessible.
This distinction also prevents an unfair inference about intent. A contaminated model may result from negligent filtering, accidental ingestion, or deliberate optimization against a test. Data overlap alone does not identify which occurred. A technically serious article can describe a measurement problem without alleging deception unless there is separate evidence about decisions and conduct.
Black-box tests ask narrower questions
Oren and colleagues' October 2023 paper proposes a test based on the likelihood of a benchmark's canonical ordering compared with shuffled orderings, under an exchangeability assumption. It offers statistical guarantees within that framework and reports limited evidence of pervasive contamination in the models it audited. The result should not be expanded into a guarantee that every unflagged model is uncontaminated. Black-box contamination study.
The distinction between detecting a signature and proving an absence is central. A positive result can be informative when the assumptions fit the data. A negative result may reflect genuinely absent exposure, insufficient statistical power, a different form of memorization, or a benchmark whose structure does not suit the test. The evaluator needs to explain what patterns the method could detect at the available sample size.
The phrase black box also covers different access levels. Some interfaces return only a generated answer. Others provide token probabilities or allow controlled scoring of supplied text. A method requiring likelihoods cannot simply be applied to a service that exposes only conversational output. Workarounds may change the statistical question. A report should describe the actual interface rather than invoking black-box testing as if it were a single capability.
Repeated queries introduce another practical issue. If the service changes model versions during the audit, observations may no longer come from one stable system. If requests are routed differently by region or account, results may mix configurations. These are not reasons to reject external auditing; they are reasons to retain timestamps, endpoint identifiers, settings, and any available version information.
A careful conclusion states the tested model, benchmark, access mode, assumptions, and detection result. It avoids converting “no statistically significant signal under this procedure” into “the training data was clean.” That wording protects both the reader and the researcher from a stronger claim than the experiment can support.
Guessing the odd detail
Deng and colleagues' NAACL 2024 paper presents a retrieval-based overlap investigation and a method called Testset Slot Guessing. The latter includes asking a model to reconstruct a masked incorrect answer choice or an unlikely word. The idea is to probe details that may be difficult to infer solely from solving the question. These methods provide evidence to examine, not an infallible detector of every possible exposure. NAACL paper.
A hypothetical example clarifies the appeal. A multiple-choice question may have one correct answer and several arbitrary distractors. Knowing the subject can identify the correct answer, but it may not determine the exact phrasing of a particular wrong option. Reconstructing that phrasing can therefore be more suggestive of exposure than simply answering correctly. The strength of the evidence depends on how predictable the missing phrase was in the first place.
A common distractor may be easy to guess because many teachers use it. An unusual proper name may be recoverable from public context outside the benchmark. A model may generate several plausible alternatives, one of which happens to match. The audit should specify whether it counts the first answer, any answer across repeated attempts, or a ranked candidate list. Each procedure has a different chance baseline.
Matched controls can help. Construct comparable items with similarly predictable language but no known exposure, and compare reconstruction behavior. Avoid selecting only the most impressive matches after examining a large pool. If many probes are tried, the analysis should account for the selection process and report how many opportunities there were to obtain a striking result.
The method is most useful as part of converging evidence: a source overlap, an unusual reconstruction, a temporal pattern, and a controlled comparison may together support a stronger interpretation than any one signal. They still require care about what is established. A memorable demonstration should not replace a systematic accounting of all tested cases.
A detector can be passed for the wrong reason
Dekoninck and colleagues' February 2024 paper studies deliberate contamination designed to evade detection. Its experiments show that detection methods can be vulnerable under an adversarial model-provider setting. This is research on the limits of particular methods, not evidence that a named current supplier is using those techniques. Detection-evasion study.
The general evaluation lesson is that a screening procedure becomes less informative when the party being screened can repeatedly optimize against it. If a training pipeline is adjusted until a contamination test stops flagging, the resulting negative test may no longer carry the interpretation it had before that adaptive process. The pipeline may genuinely improve, but the test has also become part of development.
A reasonable response is to separate development checks from independent audit checks. Developers should use contamination screening while building datasets. Evaluators should retain additional methods or held-out probes that were not repeatedly exposed to optimization. The goal is not secrecy for its own sake. It is to preserve some evidence that was not selected precisely because the system already performed well on it.
This resembles an ordinary scientific distinction between exploratory analysis and confirmatory testing. Exploratory work helps discover patterns and improve methods. Confirmatory work asks whether a specified claim survives on evidence not used to choose it. Both are valuable, but presenting the former as the latter can make a result look more decisive than it is.
An audit should therefore report prior access. Who knew the test items? Who saw intermediate results? How many configurations were evaluated before the reported model was selected? These facts are part of the experiment, not administrative trivia. They help readers distinguish a system that generalizes to an independent test from one that was gradually tuned to a familiar measurement environment.
Freshness changes the experiment
LiveCodeBench, first submitted in March 2024, describes collecting new programming problems over time from competition platforms and evaluating several code-related capabilities. Its temporal approach makes it possible to examine performance on problems published in different periods. The original paper's problem count and evaluated models describe that historical release, not the current size of the project. LiveCodeBench paper.
Freshness is valuable because a problem published after a model's genuinely fixed training cutoff cannot have entered that earlier training run in its published form. But the inference depends on the cutoff being real, complete, and relevant to all stages of the model's development. A base model's cutoff may differ from the date of later fine-tuning. A service may use retrieval or updated components. The evaluator needs the system's information-access history, not just a date attached to one training stage.
New questions can also differ in difficulty. A competition may change its style, participant population, or problem distribution over time. A sharp performance drop after a cutoff can suggest exposure effects, but it can also reflect a shift in the tasks. To strengthen the interpretation, compare difficulty indicators, problem categories, and performance by independent reference systems where appropriate.
There is a second trade-off. A constantly refreshed benchmark improves temporal separation but complicates comparison across releases. A model scored in January and another scored in June may have answered different questions. Their headline percentages should not be treated as a direct race unless a linking design or shared evaluation window supports that comparison.
A useful presentation retains both a fixed historical set and a fresh set, labeled clearly. The historical set supports continuity; the fresh set probes new exposure conditions. Neither should borrow the other's strengths without acknowledging its weaknesses. Reporting both is more informative than trying to make one number serve incompatible purposes.
Rewriting a question can change the skill
A natural response to contamination is to paraphrase existing questions. That can be useful, but a rewritten item is not automatically equivalent to the original. Small wording changes can remove a clue, add ambiguity, change a cultural assumption, or alter the number of reasoning steps. If performance falls, the cause may be reduced memorization, increased difficulty, or both.
Imagine a hypothetical question about the average speed of a journey. One version states the distance and total time directly. Another describes two stages and expects the reader to infer the total. Both concern average speed, but the second includes extra arithmetic and comprehension. A difference in accuracy cannot isolate contamination unless the change in task demands is addressed.
The article's proposed procedure is to document the intended invariant before rewriting: mathematical relation, factual target, required evidence, or decision rule. Then have qualified reviewers check whether the transformed item preserves that invariant. Where feasible, compare human solve rates and error patterns. Human performance is not a perfect measure of model difficulty, but it can reveal obvious changes in clarity or complexity.
Preserve the original answer key and the reason a new answer key is valid. A paraphrase that accidentally changes the correct answer can turn a capable response into a scored failure. Automated generation can help produce candidates, but candidate generation is not equivalent to validation. A small, carefully reviewed paired set may support a clearer causal interpretation than a large collection of unchecked rewrites.
Finally, analyze item pairs rather than only averages. If a model improves on some paraphrases and worsens on others, the pattern may indicate sensitivity to wording rather than straightforward recall. The full paired result allows readers to examine that possibility. A single aggregate drop can conceal several different mechanisms that deserve different scientific explanations.
A new exam needs calibration
The GSM1k study, first submitted in May 2024 and revised in November, created a new arithmetic benchmark designed to resemble GSM8k in style and complexity. It compared properties such as human solve rates and solution steps. The paper reports evidence of overfitting for some model families while also finding generalization to novel problems. That mixed result is important: contamination concerns do not imply that every success is memorization. GSM1k study.
The broader design challenge is to create an exam that is new without being accidentally easier or harder. A fresh dataset written by a different team may contain shorter sentences, more familiar contexts, or fewer distracting quantities. A model can improve or decline for those reasons even if neither dataset was present in training. Calibration should therefore examine the properties relevant to the claimed capability.
For an arithmetic test, useful descriptors might include operations required, number of intermediate quantities, answer magnitude, and whether irrelevant information appears. For a coding test, descriptors might include algorithmic complexity, input constraints, language requirements, and hidden-test coverage. The appropriate descriptors depend on the task; a generic similarity score cannot replace domain understanding.
A calibrated replacement need not reproduce every feature of the old benchmark. In fact, it may deliberately correct weaknesses. But then the report should distinguish an exposure-control experiment from a redesigned capability test. Both can be valuable, yet they answer different questions. A redesigned test may be more useful operationally while being less suitable for estimating the old benchmark's contamination effect.
The strongest reporting shows the construction process, review criteria, rejected items, and known limitations. That transparency allows future researchers to challenge the calibration rather than treating “same difficulty” as an unsupported adjective. Newness is a property of history; comparability is a property that must be argued with evidence.
Training access is broader than a web crawl
A training-data audit that examines only a large pretraining crawl can miss later routes by which evaluation material enters a model. Instruction datasets may contain benchmark-like tasks. Feedback examples may reproduce public questions. Synthetic data may be generated by a model that has encountered the benchmark. Internal debugging conversations may include the exact failure cases that developers want the next version to solve.
These routes should not automatically be condemned. Learning from observed weaknesses is normal development. The evaluation problem arises when the same items are later represented as independent evidence of generalization. Once a test case becomes a training example, it can remain a valuable regression test, but its role has changed. The label should change with it.
A useful data lineage record distinguishes original source, transformation, inclusion stage, and use. An item might begin as a public question, be paraphrased into an instruction example, and then be filtered into a fine-tuning set. A search only for the original text may miss the final version. A record of the transformations can reveal the relationship even when literal similarity is weak.
The practical burden is substantial for large datasets. That does not justify pretending the burden has disappeared. Reports can state the portion of the pipeline covered and the uncertainty left by inaccessible stages. A partial audit with a clear scope is scientifically preferable to an absolute claim unsupported by the available records.
For readers comparing models, the question is whether the disclosed cutoff and decontamination process cover the evaluated system's whole development path. If the answer is unclear, the score can still be reported, but the interpretation should remain conditional. A date attached to a base model is not a complete biography of every model, adapter, retrieval index, and tool used in the final evaluation.
Test-time search changes what success means
An evaluation may legitimately allow browsing, code execution, calculators, or retrieval. These tools can make a system more useful, and testing them can be appropriate. But the system being evaluated is then larger than a model responding from its weights alone. The protocol should specify which external information sources are permitted and how the answers are scored.
A hypothetical coding agent might search the web and find a discussion of the exact competition problem. If the evaluation is intended to measure practical problem-solving with open resources, that behavior may be within scope. If it is intended to measure novel algorithmic reasoning, it may compromise the claim. Neither interpretation can be settled after the fact by whether the score looks impressive.
The audit should record relevant tool traces and distinguish retrieval of general documentation from retrieval of the exact solution. It should also consider whether public benchmark identifiers make exact-item search unusually easy. Removing an identifier may reduce that route, but it can also alter the normal task context. Again, the design depends on the capability being measured.
There is an additional fairness issue when different systems receive different information budgets. One model may be allowed many searches and retries while another receives a single prompt. The comparison may be useful as a comparison of complete products, but it should not be described as a pure comparison of model reasoning ability. Cost, latency, and tool access belong in the result's conditions.
A transparent report can present several modes: closed-book, restricted tools, and open-resource assistance. The point is not to declare one mode morally superior. It is to avoid attributing the benefits of an external information source to a capability the experiment did not isolate. Good evaluation measures the system actually used and names that system accurately.
Score inflation and ranking are separate questions
A benchmark can overestimate absolute performance without changing the order of two models. Conversely, a small amount of differential exposure can matter greatly when models have similar genuine ability. These possibilities are easy to confuse because leaderboards display both a score and a rank in the same row.
Consider a hypothetical pair of systems that would score seventy and sixty percent on a clean set. If exposure adds five percentage points to both, the ranking stays the same while both absolute scores become misleading for clean-task prediction. Now consider systems that would score seventy and sixty-nine percent, but only the second receives a substantial exposure advantage. Their order may reverse. The causal question depends on who was exposed, to what, and how that exposure affected performance.
This is why a global claim that “the benchmark is contaminated” does not fully describe its comparative value. It may still distinguish systems for some purposes, but the evidence must support that use. A stable rank does not validate an inflated absolute success estimate. An inflated score does not prove the entire ordering is meaningless.
For deployment decisions, absolute performance can be more important than rank. A service that requires reliable completion may not benefit from being the best among systems that all fail too often. Conversely, a research comparison may focus on relative improvement under a fixed, acknowledged benchmark protocol. The report should connect the metric to the decision rather than assume that rank alone answers it.
The proposed presentation includes raw scores, paired clean or fresh comparisons where available, and uncertainty intervals appropriate to the sampling design. It should also disclose whether rank differences are small enough that ordinary sampling variation could change the order. A precise-looking ordinal position can obscure a very imprecise difference between neighboring systems.
Sampling uncertainty does not repair selection bias
If a model answers a random sample of independent items, statistical uncertainty can describe how much the observed proportion may vary across similar samples. But a confidence interval does not automatically account for contamination, benchmark selection, prompt tuning, or unreported failed experiments. Those are different sources of uncertainty and bias.
A hypothetical developer may test twenty prompt templates and report only the best one on the benchmark. The resulting score is selected partly for favorable performance on that specific set. Calculating a narrow interval around it does not undo the selection. An independent evaluation set or a clearly separated development-and-test procedure is needed to assess how well the selected configuration generalizes.
Items may also be dependent. Several questions can derive from the same passage, codebase, problem template, or underlying fact. Treating them as fully independent may understate uncertainty. The appropriate analysis might group related items or report performance by source family. The choice should be explained in relation to the data's construction rather than applied mechanically.
Repeated attempts complicate the unit of analysis. A model allowed several generations per question may succeed at least once even when its first attempt fails. That can be useful, but “at least one success among many” differs from single-attempt reliability. The number of attempts, selection rule, and verification mechanism all belong beside the score.
An honest evaluation report does not need to solve every statistical problem perfectly. It needs to identify the principal assumptions and avoid claiming more certainty than the design supports. The most useful intervals are attached to a clearly defined quantity. If the quantity itself mixes exposure, adaptation, and inconsistent access conditions, precise arithmetic can make an unclear result look authoritative without making it more meaningful.
Preserve the experiment, not only the final table
Reproducibility requires more than a benchmark name. The evaluator should preserve the dataset version, item identifiers, exclusions, prompt templates, answer extraction rules, decoding settings, tool configuration, and scoring code. A small change in any of these can affect the measured result. Without them, a later researcher may repeat the label while running a different experiment.
For contamination analysis, retain the search procedure as well. That includes normalization rules, candidate thresholds, corpus coverage, duplicate handling, and adjudication criteria. If the audit used semantic matching, record the relevant model or embedding version and how candidate matches were reviewed. A list of removed documents is useful only if readers can understand why they were removed and what was never searched.
Versioning should distinguish corrections from new experiments. If an answer key is repaired, the report should identify affected items and recompute results where possible. If a model endpoint changes, the new run should receive a new record rather than silently overwriting the old score. Historical results can remain valuable when their conditions are preserved.
Public release has limits. Training data may contain private or licensed material, and evaluation items may need temporary protection. An artifact package can therefore separate public methods from controlled evidence. What matters is that the restriction is explicit and that an appropriate verification route exists. “Trust us” is not equivalent to a documented access arrangement.
The reader should be able to trace a headline result to the actual observations that produced it. That does not mean every reader will reproduce the experiment. It means reproduction and criticism are possible for someone with the necessary expertise and authorized access. A benchmark becomes more scientifically useful when disagreement can be resolved through inspectable records rather than competing assurances.
Keep a quarantine between development and confirmation
A practical laboratory can maintain two clearly labeled collections. The development collection contains known failures, public examples, and cases used to improve the system. The confirmation collection is reserved for a specified evaluation and has an access history. The separation is procedural as well as technical: a researcher who has repeatedly inspected confirmation answers can influence development even without copying the files into training.
A hypothetical team might release an old confirmation set once it becomes development material, then create a new independently reviewed set for the next claim. This rotation preserves the value of public scrutiny while acknowledging that a test cannot remain unseen after widespread release. The dates of access, release, and model selection should accompany the scores.
Access restrictions should be proportionate. A private set can contain mistakes that public review would catch, so secrecy is not itself evidence of quality. An independent reviewer can examine the questions and scoring before results are announced, with an agreed disclosure process afterward. The goal is a temporary separation of roles, not an opaque evaluation that no outsider can challenge. A well-managed quarantine preserves both independence and eventual inspectability, rather than sacrificing one permanently to obtain the other.
What a careful contamination claim should say
A strong claim identifies the model, the benchmark, the relevant development stage, the evidence of exposure, and the method used to estimate its effect. If some elements are unknown, they should remain unknown in the wording. “We found matching questions in the disclosed fine-tuning data” is different from “the model cheated.” “Performance declined on a calibrated fresh set” is different from “the old score was entirely memorized.”
The article's recommended reporting structure begins with the observation, then the interpretation, then the limitation. For example, a hypothetical audit could report exact overlaps for a specified subset, explain why those overlaps threaten an unseen-item claim, and note that the unavailable pretraining corpus prevents a complete exposure history. This is more informative than a dramatic binary verdict because it tells the reader which next evidence would matter.
A supplier's response should be evaluated with the same discipline. A filtering description supports a claim about process. A clean independent test supports a claim about behavior under that test. Neither automatically proves the other. Good documentation connects them without pretending that a well-intentioned procedure guarantees a perfect outcome.
The public stakes are substantial because benchmark numbers influence purchasing, research priorities, and beliefs about technological progress. Overstating contamination can be as misleading as ignoring it. If genuine generalization and exposure effects coexist, the task is to measure their relationship, not force the evidence into a story that every model is either a reasoning machine or a memorized answer sheet.
The useful question is specific: what does this score tell us about performance on the next relevant task? Answering it requires a history of the data, a clear account of permitted information, and an experiment that can separate plausible explanations. A benchmark remains valuable when those conditions are visible. Its number becomes suspect when the conditions disappear while the confidence remains.