Evidence checked October 31, 2026. This article analyzes dated primary research and proposes an original dataset-audit framework. It reports no new training experiment. All illustrative datasets, counts, and organizational scenarios are hypothetical.
The label conceals the pipeline
A dataset is described as synthetic, and the word appears to settle the important question. It does not. The examples might be generated from scratch, rewritten from human documents, labeled by a model, produced by a simulator, or selected from many candidate solutions after an external check. Each pipeline introduces different information and different errors. Treating them as one category makes both enthusiasm and criticism too easy.
Synthetic data can be useful because it allows deliberate construction of examples that are scarce, expensive, or difficult to collect. It can also reproduce the generator's limitations, flatten uncommon patterns, and create a false appearance of independent evidence. The relevant question is how the examples were made and what the resulting training process learned from them.
Self-Instruct, first submitted in December 2022 and revised in May 2023, provides a concrete historical example. Its pipeline generates instructions, inputs, and outputs, filters invalid or similar examples, and uses the retained material for instruction tuning. The paper reports improvements in its evaluated settings. It does not imply that any unfiltered collection of generated text will improve any model. Self-Instruct paper.
A useful public discussion therefore begins with a ledger rather than a verdict. Which generator produced the data? What source material or seeds did it receive? Which candidates were discarded? How were answers checked? How much human-origin information remained in the training mixture? Which tasks improved, and which became less reliable? Those questions turn the word synthetic from a marketing category into an inspectable experimental condition.
Different kinds of generation introduce different evidence
Suppose a hypothetical team creates ten thousand arithmetic exercises by sampling numbers from a known formula. The answers are computed independently. Another team asks a language model to invent ten thousand historical questions and answers from memory. Both datasets are synthetic in some sense, but the source of correctness is different. The first can have a mechanically checkable target; the second may contain fluent unsupported claims.
A third team begins with human-written documents and asks a model to produce summaries. Here the synthetic output depends on an external source, but it can omit qualifications or introduce errors. A fourth team uses a simulator to create sensor readings under specified physical assumptions. Its examples may be internally consistent while failing to represent real-world conditions outside the simulator's model.
The article's proposed taxonomy records the generation mechanism, conditioning information, validation method, and intended use. It distinguishes synthetic inputs from synthetic labels. A real photograph with an automatically generated label is not the same as a generated photograph with a verified label. A rewritten document is not independent of its source merely because the wording changed.
This distinction matters when counting dataset size. Ten paraphrases of one example may offer useful variation, but they do not necessarily provide ten independent facts. A million generated records can still derive from a narrow set of seeds. The dataset's apparent scale should be accompanied by information about its underlying diversity and dependencies.
The goal is not to privilege human material automatically. Human data can be wrong, biased, repetitive, or poorly documented. The point is to identify where evidence enters the pipeline and where assumptions are introduced. That makes it possible to evaluate quality without treating either human origin or machine generation as a blanket certificate.
Collapse research concerns a feedback process
Shumailov and colleagues' preprint, first submitted in May 2023 and revised in April 2024, studies recursive training on generated data and describes a loss of information in the tails of distributions. It examines several model families and frames the phenomenon as model collapse. The claim concerns specified recursive settings; it should not be translated into a statement that every use of synthetic data immediately makes a model worse. Curse of Recursion preprint.
A simple hypothetical sampling exercise illustrates the intuition without reproducing the paper's experiments. Imagine a source distribution with a rare category appearing in one of every thousand observations on average. A finite generated sample may omit it. If the next model is trained only on that sample, the missing category receives no direct support from the original observations. Repeating the replacement process can make the omission persistent.
The important word is replacement. The result depends on what information remains available across generations, how samples are drawn, and what the model can represent. A pipeline that retains original data and adds verified generated examples is different from one that repeatedly discards the original source and learns only from its own output. The label synthetic does not identify which process occurred.
Collapse should also be distinguished from ordinary errors, hallucinations, or a stylistically bland response. Those can occur for many reasons. Demonstrating a recursive degradation mechanism requires a sequence of training conditions and measurements, not merely showing one disappointing answer from a chatbot.
For reporting, identify the generation number, training mixture, sampling method, and evaluation distribution. Without those, a dramatic claim about models consuming their own output may obscure the exact mechanism that researchers measured and the design choices that could change it.
Accumulation changes the mathematical question
Gerstgrasser and colleagues' April 2024 paper explicitly compares replacement with accumulation. In its experiments and theoretical setting, retaining original data while accumulating successive synthetic generations avoids the collapse observed under replacement. The finding is a counterpoint to an unconditional inevitability story, not a guarantee that any accumulating dataset remains accurate or representative forever. Accumulation study.
The practical implication is that a data ledger needs mixture history. A statement such as “half the dataset is synthetic” is incomplete if it omits which half, how often examples are sampled, and whether original observations remain influential during training. A small preserved archive may have little effect if its sampling weight becomes negligible. Conversely, deliberate replay can maintain exposure to important real examples even as the stored collection grows.
Consider a hypothetical pipeline with one thousand original records and one thousand new generated records at each round. Storage accumulates, but training might sample uniformly from all records, oversample the originals, or use a fixed-size window that eventually drops them. These are different algorithms despite the same file directory containing every generation.
The evaluation should therefore distinguish storage retention from training exposure. Record the actual sampling schedule, weights, and number of passes. If rare original cases are preserved but never selected during a particular run, their existence on disk does not establish that they influenced the model.
This is a recurring theme in data science: a dataset description is not a training-process description. Both are needed. The accumulation result is valuable because it shows how a seemingly small change in the information flow can alter the conclusion. It encourages specific experiments rather than a universal slogan that synthetic data is either a cure for scarcity or an unavoidable route to failure.
A statistical model has assumptions, not a universal mixing ratio
Seddik and colleagues' April 2024 paper analyzes recursive synthetic-data training through a statistical model and examines conditions involving mixtures of real and synthetic data. It reports theoretical conclusions supported by empirical validation. Its thresholds and conclusions belong to the model and assumptions studied; they should not be converted into a universal safe percentage for every large-model training pipeline. Statistical analysis paper.
A hypothetical procurement presentation might claim that keeping twenty percent human data prevents collapse. That number would require evidence tied to the actual generation process, task distribution, and training method. A ratio alone ignores whether the human material covers the relevant cases, whether it is duplicated, and whether the synthetic material is independently checked or recursively copied.
The unit of the ratio also matters. Percent of documents, percent of tokens, percent of training steps, and percent of unique underlying examples can differ sharply. A few long generated documents may dominate token exposure while accounting for a small fraction of files. Repeatedly sampled examples may dominate training even when they are a small fraction of the stored dataset.
The article recommends reporting several complementary quantities: stored volume, unique-source counts where meaningful, sampling weights, and effective exposure during training. These measures do not solve every statistical problem, but they prevent a single percentage from concealing the pipeline's structure.
A theoretical result can guide a practical experiment by identifying variables that matter. It does not remove the need to measure the actual system. The scientifically useful response to a proposed mixing rule is to ask which assumptions hold, which do not, and what local evidence tests the difference. That is more constructive than treating theory as either an unconditional guarantee or irrelevant abstraction.
Useful synthetic data can have a curriculum
The June 2023 “Textbooks Are All You Need” paper describes a code-model training mixture using selected web material and generated textbook-style content and exercises. It reports performance for its specific model and benchmarks. The study supports investigating carefully curated training material; it does not prove that synthetic textbooks are sufficient for every domain or that source quality can be assessed by polished prose alone. Textbooks paper.
The curriculum idea is attractive because examples can be designed to isolate concepts, increase difficulty gradually, and expose common mistakes. A hypothetical programming dataset could include simple loops, boundary cases, and progressively more complex compositions. That structure may teach more effectively than a random pile of code snippets. But the curriculum also expresses choices about what counts as important.
If the generator repeatedly produces familiar textbook patterns, the student may perform well on those patterns while struggling with messy real tasks. Real inputs can contain incomplete requirements, unusual formatting, conflicting constraints, and errors. A curriculum should therefore be evaluated on independent tasks that include the conditions the model will encounter outside the training collection.
Difficulty labels need validation. A generator's declaration that an exercise is advanced does not establish that it requires advanced reasoning. Long wording can make an easy problem look difficult, while a short edge case can be genuinely demanding. Domain review and measured performance can help distinguish appearance from task structure.
The provenance ledger should record the intended curriculum and the evidence that it was achieved. Which concepts were targeted? How were examples distributed? What checks verified solutions? Which cases were rejected? This makes the curriculum a testable design rather than an aesthetic judgment that the dataset looks educational.
A correct final answer does not validate every explanation
STaR, introduced in March 2022, describes an iterative process that generates rationales, uses correct answers to guide further generation when needed, and trains on rationales that lead to correct answers. The paper reports improvements on its evaluated tasks. Its method illustrates how external answer information can enter a synthetic-data loop rather than the model learning only from unconstrained self-generated text. STaR paper.
For dataset auditing, the distinction between final-answer correctness and explanation correctness remains important. A hypothetical arithmetic solution may reach the right number through an invalid intermediate step that happens to cancel an earlier error. A code solution may pass a limited test suite while relying on an assumption not guaranteed by the specification. Filtering only on the final result can retain misleading reasoning.
The validation method should match the purpose of the data. If the goal is to teach a final mapping, a verified answer may be the main target. If the goal is to teach reliable explanations or procedures, intermediate claims need additional scrutiny. A model-generated rationale should not be treated as a faithful record of an internal reasoning process merely because it reads coherently.
Independent checks can include executable tests, symbolic verification where appropriate, or expert review of a sample. Each has limits. Tests can miss edge cases; formal checks depend on the specification; reviewers can disagree. The ledger should record which checks were applied and what they establish.
A useful error analysis examines accepted examples, not only rejected ones. The rejection count tells the team how much material was filtered, but the remaining error rate determines what the student actually learns. A small, carefully reviewed sample of accepted material can reveal systematic weaknesses that a large pass count conceals.
The generator has a version and a context
A synthetic dataset should identify the generator configuration used to create it. Model name alone may be insufficient if a hosted service changes over time or if prompts, tools, and sampling settings alter output. A reproducible record includes version identifiers where available, generation dates, prompt templates, relevant context, and decoding parameters.
A hypothetical team may generate examples in several batches over a month. If the provider updates the model midway, the collection could contain different styles or error patterns. That does not necessarily invalidate the data, but it creates a version boundary worth recording. Without timestamps and batch identifiers, later analysis may incorrectly attribute the difference to a filtering change or a random fluctuation.
Context matters as much as the generator. A model prompted with source documents, a glossary, and worked examples is not operating under the same conditions as one asked to invent content unaided. If those conditioning materials are unavailable, another team cannot reproduce the pipeline even if it has access to the same model.
The record should also include tool use. Generated code that was executed and repaired has a different history from code emitted once without feedback. A factual answer checked against a source has a different evidentiary basis from an unsupported answer. These histories can guide both training weights and later error investigation.
The article's proposed ledger does not require publishing private prompts or protected source material indiscriminately. It requires preserving enough authorized metadata and references to understand the process. Public documentation can summarize restricted components while identifying the limitation. The aim is traceability, not maximal disclosure at the expense of privacy or source obligations.
Seeds can become a hidden bottleneck in diversity
Many generation pipelines begin with a small set of examples. Those seeds can help establish format and task style, but they can also constrain what the generator produces. Thousands of outputs may remain close to a few underlying templates. Surface variation can create an impression of breadth while the underlying decision patterns remain narrow.
A hypothetical customer-service dataset might contain many names, products, and dates but only two complaint structures. A model trained on it may learn the phrasing of those structures well while failing on disputes involving multiple issues or incomplete evidence. Counting unique sentences would overstate the diversity relevant to the task.
The audit should identify dimensions of variation that matter: topic, reasoning operation, document structure, language, ambiguity, error type, and required response. These dimensions should be defined with domain expertise rather than inferred solely from embeddings or vocabulary counts. A dataset can be lexically diverse while conceptually repetitive.
Measure seed influence by grouping descendants where possible. If one seed produces a large family of examples, keep that relationship in the record. Train-test splits should avoid placing near-identical descendants on opposite sides and then claiming independent generalization. Family-level separation can provide a more demanding test.
The goal is not to eliminate all repetition. Repeated practice can be useful, and common tasks deserve representation. The question is whether repetition is deliberate and measured. A well-designed curriculum may oversample important patterns while preserving distinct evaluation cases. An undocumented pipeline may oversample them accidentally and mistake the resulting benchmark familiarity for broad capability.
Filtering is an intervention with side effects
Filtering removes material according to a rule, and every rule changes the training distribution. A quality filter may improve average correctness while disproportionately discarding unusual language, rare topics, or complex examples. A safety filter may remove harmful material but also reject benign discussion that shares vocabulary. A similarity filter may reduce duplicates while eliminating useful variants.
A hypothetical pipeline discards every answer longer than a threshold because long outputs are more expensive to review. The resulting dataset becomes shorter and perhaps simpler. If the student later struggles with long-form tasks, the failure may reflect the selection rule rather than the generator's intrinsic ability. The filter should be treated as part of the training design.
Record candidate counts before and after each stage. If the pipeline begins with one million examples and retains one hundred thousand, the ninety percent rejection rate is not automatically evidence of rigor. It could reflect effective screening, poor generation, or a rule that narrows the dataset excessively. The distribution of rejected examples matters.
The proposed audit samples both sides of the decision boundary. Review accepted examples for residual errors and rejected examples for false exclusions. Where practical, examine performance with and without particular filters. An ablation can reveal whether a filter improves the intended capability or merely changes the test distribution in a favorable way.
Filters should also be versioned. A dataset assembled under several evolving rules can contain inconsistent standards. That may be acceptable during experimentation, but the final release should identify the mixture or rebuild it consistently. Otherwise, downstream users inherit unexplained variation and may be unable to reproduce a reported result even when they follow the published description.
Model-based review can share the generator's blind spots
A generator and a reviewer may make correlated mistakes, especially when they are similar systems or share training influences. If both accept the same plausible false claim, the review stage can add confidence without adding independent evidence. The important question is what new information the reviewer contributes.
A hypothetical model invents a software API method, and another model judges the answer as clear and complete. That judgment may assess style without verifying that the method exists. An executable check against the actual library would answer a different question. Likewise, a factual claim can require a source lookup rather than another model's impression of plausibility.
The article recommends dividing validation into explicit categories: format, internal consistency, external correctness, task relevance, and suitability for the intended use. A model can assist with some categories while an independent mechanism handles others. The report should not call the whole example verified when only formatting was checked.
Disagreement is useful evidence. If reviewers disagree, retain the reason and investigate a sample rather than automatically selecting the most favorable verdict. A disagreement cluster may identify ambiguous instructions, missing context, or a domain where automated review is unreliable. These findings can improve the generation prompts and the evaluation design.
Human review is not automatically independent either. Reviewers can be influenced by the generated explanation or lack expertise in a specialized topic. Clear rubrics, appropriate expertise, and access to sources matter. The objective is not to replace one unquestioned authority with another, but to identify which checks actually reduce uncertainty about the training example's correctness.
Rare cases need an explicit place in the ledger
A dataset designed for average performance can underrepresent uncommon but consequential cases. Synthetic generation offers an opportunity to construct such cases deliberately, but only if the team knows what is missing. Asking for “diverse examples” is not a substitute for a coverage model grounded in the intended application.
Consider a hypothetical form-processing system. Most records may use standard formatting, while a small fraction contain mixed languages, missing fields, unusual date conventions, or conflicting identifiers. A generator may produce clean, complete forms unless instructed otherwise. The resulting training data can be large and polished yet poorly suited to the cases where users need the most help.
The proposed coverage plan lists important variation dimensions and identifies combinations that should be tested. It should distinguish realistic rare cases from impossible combinations. Domain experts can help establish plausibility, while actual authorized error reports can reveal patterns that designers did not anticipate.
Oversampling rare cases can be useful for training, but evaluation should preserve or explicitly model the intended deployment distribution. A model's overall score on a deliberately balanced challenge set does not directly estimate its average real-world error rate. Both views can be reported: performance under realistic prevalence and performance on stress cases.
The ledger should record why each rare-case family was created and how its realism was checked. That prevents synthetic edge cases from becoming arbitrary puzzles detached from the service. It also makes future updates easier: when a new failure pattern appears, the team can add a documented family rather than quietly changing the dataset and losing the history of what the model was trained to handle.
Synthetic does not mean privacy-free
Generated data can still depend on sensitive source material. A summary of a private record may preserve identifying details. A paraphrase can retain the same confidential facts. A generator may reproduce material from its context or training history. Calling the output synthetic does not by itself establish that it is safe to share.
A hypothetical team creates fictional customer records using real tickets as prompts. If the outputs retain unusual incidents, names, or combinations of attributes, the fiction label may be misleading. The audit needs to examine the transformation and the residual information, not merely the intended use of the word fictional.
Privacy review should be separated from correctness review. An accurate generated example may be inappropriate to publish, while a privacy-preserving example may be factually wrong or unrealistic. These are different criteria and should have separate outcomes in the ledger. A single quality score can conceal which requirement was actually assessed.
The article does not provide a legal test for anonymization or data rights. Those depend on context and applicable obligations. Its technical recommendation is to document source access, transformation methods, permitted uses, and the evidence supporting any privacy claim. Where uncertainty remains, restrict sharing rather than assume that generation erased the connection to the source.
Synthetic data can be valuable for reducing direct use of sensitive records in development, but the benefit should be demonstrated. Independent evaluation can examine whether important statistical properties are retained while identifying information is reduced under a specified threat model. Neither a visual inspection of a few rows nor a generator's assurance establishes that result for the entire dataset.
Provenance also supports corrections and withdrawal
A provenance ledger becomes especially useful when an error is discovered after training. If a generated family contains a wrong rule, the team needs to identify related examples, affected dataset releases, and models trained on them. Without lineage, correction can become a search for similar wording that misses paraphrases and transformed descendants.
A hypothetical arithmetic generator uses a flawed template for a particular percentage calculation. Thousands of examples inherit the error with different numbers. Fixing one visible example does not repair the family. The template identifier and generation batch can reveal the full scope far more efficiently than manual inspection of the final files.
The same principle applies to source changes. A document used for grounded generation may be corrected or withdrawn. The team should know which examples relied on it and whether the correction affects their labels. A source link alone may be insufficient if the page changes; a version or dated snapshot reference can help establish what the generator actually saw.
The proposed correction record includes the issue, affected lineage, review decision, replacement data, and downstream model versions. It should distinguish corrected dataset availability from completed model retraining. Publishing a fixed dataset does not automatically update models already trained on the old release.
This is ordinary maintenance made harder by scale, not an exotic property unique to AI. Synthetic generation can multiply both useful examples and systematic mistakes quickly. The ledger provides a way to contain that multiplication. Its value is measured not only when the dataset is released, but when someone later needs to explain and repair a result that looked convincing at the time.
Benchmark independence must survive generation
If a generator has encountered an evaluation benchmark, it may produce related examples even when the pipeline never explicitly requests them. A synthetic training set can therefore inherit benchmark exposure indirectly. That possibility does not prove contamination in any particular dataset, but it means the generation step should not be treated as a clean break from the teacher's history.
A hypothetical coding generator produces exercises resembling common public benchmark tasks. Some resemblance is expected because programming concepts recur. The audit should distinguish generic skill overlap from near-duplicate problem-and-solution pairs. Exact matching can identify some cases, while semantic review may be needed for transformed versions.
Evaluation material should be kept separate from prompts used to generate training data. If a team uses benchmark failures to request targeted examples, the resulting dataset may be useful for development, but that benchmark is no longer an untouched test of the improved system. A fresh or independently held-out evaluation is needed for a stronger generalization claim.
The ledger should record whether evaluation results influenced generation prompts, filtering thresholds, or checkpoint selection. These are legitimate development activities when disclosed. The problem is presenting the same evidence as independent confirmation after it has shaped the model.
This article focuses on synthetic-data provenance rather than the full contamination-audit literature. The practical point is narrow: generated examples have ancestors, and those ancestors can include the very tests used to advertise the student model. Recording that possibility allows the team to design an evaluation that measures useful transfer rather than repeated exposure to familiar material under new wording.
Measure diversity at the level of the intended skill
A dataset can contain many unique strings while covering few distinct tasks. Conversely, a compact dataset can contain structurally varied problems with similar vocabulary. Diversity metrics should therefore be chosen in relation to the capability being trained, not simply because they are easy to compute.
For a hypothetical extraction model, relevant variation might include document layout, entity type, ambiguity, and missing information. For code generation, it might include algorithmic pattern, input constraints, library use, and failure cases. For multilingual assistance, it might include language variety, register, and task context. One embedding-distance average cannot capture all of these dimensions adequately.
The article's proposed evaluation combines descriptive statistics with task-family analysis and independent performance tests. Descriptive statistics reveal obvious concentration. Task-family analysis checks whether the planned coverage exists. Performance tests determine whether the student can use that coverage on new examples. These stages answer different questions and should remain separate.
Avoid maximizing diversity blindly. A dataset containing many irrelevant or incoherent examples can score highly on some measures while becoming less useful. The goal is relevant coverage with correct labels and realistic structure. A rare task outside the intended application may deserve less weight than a common task with subtle but important variations.
Document the trade-off. If the team deliberately narrows the dataset to improve a specialized capability, say so. Specialization can be a strength when the deployment matches it. It becomes misleading only when the resulting model is presented as broadly improved without evidence. A provenance ledger should preserve the intended scope so that downstream users can judge whether the training data fits their own task.
Compare pipelines, not just dataset volumes
A claim that adding synthetic examples improved a model needs a comparison that isolates the relevant change. If the new run also uses more compute, a different optimizer, or a stronger base model, the improvement cannot automatically be attributed to synthetic data. The experiment should identify which variables changed and why.
A useful hypothetical design compares several conditions: original data alone, original plus generated data, original plus repeated original examples, and original plus an alternative augmentation. Matching training budget where appropriate helps distinguish the benefit of new content from the benefit of more optimization. No single design is perfect, but explicit controls make the interpretation clearer.
The amount of generated data should be varied rather than chosen once. A dose-response curve can reveal diminishing returns or deterioration. It can also show whether filtering improves quality enough to justify its cost. Reporting only the best mixture after extensive searching can overstate how reliably the recipe transfers to another setting.
Measure more than the headline benchmark. Include the intended task, relevant out-of-distribution cases, and retained capabilities that could be harmed. A pipeline may improve instruction following while reducing factual precision, or improve common cases while narrowing rare-case coverage. These trade-offs are part of the result.
The final report should connect performance to the exact dataset release and generation recipe. A statement that “synthetic data helped” is too broad to reproduce. A statement that a specified mixture, produced under documented conditions, improved defined outcomes at a stated training budget gives other researchers something they can test, challenge, and adapt.
The dataset needs documentation that outlives its creators
“Datasheets for Datasets,” first submitted in March 2018 and published in revised form in 2021, proposes documenting dataset motivation, composition, collection, recommended uses, and related context. It is a documentation framework rather than a guarantee of data quality. Its central idea applies naturally to synthetic collections, whose generation history can otherwise disappear behind a final download. Datasheets paper.
For the ledger proposed here, the essential record includes source and seed references, generator configuration, generation batches, validation rules, filter versions, lineage relationships, training mixture, known errors, and intended uses. Each field should serve a practical question. Documentation should be concise enough to maintain while detailed enough to support reproduction and correction.
A release should identify what is missing. If the generator is a proprietary service without a stable version identifier, record the available details and the resulting reproducibility limit. If source material cannot be redistributed, explain how authorized reviewers can inspect it or what claims remain unverifiable. Omitting uncertainty does not make the dataset more reproducible.
Assign maintenance responsibility. Someone needs to receive error reports, decide whether they affect the release, and communicate corrections. A static file with no owner may remain widely used after its creators discover problems. Version numbers and change logs make it possible for downstream users to know whether they are training on the corrected material.
The documentation should also separate permissions from technical quality. A dataset can be accurate but restricted in use, or permissively distributed but poorly suited to a task. Users need both kinds of information. A polished model card cannot substitute for the underlying data history when the scientific claim depends on how the examples were generated and selected.
A small ledger can expose a large claim
Consider a wholly hypothetical dataset release containing one hundred thousand retained examples. Its headline describes broad synthetic training data. The ledger reveals that eighty thousand derive from one seed family, fifteen thousand from a second, and five thousand from independent source-grounded tasks. This does not prove the dataset is bad. It changes the questions a reviewer should ask about coverage and generalization.
Suppose further that the generator created five hundred thousand candidates, with most rejected because they failed a formatting rule. The rejection rate tells little about factual correctness. If only a small sample received external verification, the report should not call every retained answer verified. The ledger makes this distinction visible without requiring a reader to inspect every record.
Now imagine that a later error report identifies a flawed assumption in the dominant seed family. The same lineage information that exposed concentration also makes repair possible. The team can identify affected examples, revise the generation procedure, and evaluate models trained on the old release. Without lineage, the apparent scale becomes a maintenance liability.
This example shows why provenance is more than a compliance ornament. It connects scientific interpretation to operational repair. It helps answer whether the model improved because of genuinely broader information, repeated practice on a narrow pattern, or an evaluation that resembled the generated material.
The ledger does not replace experiments. It makes experiments interpretable. A hundred thousand examples can be valuable, redundant, or harmful depending on their content and use. Counting them is the beginning of a description, not the conclusion of an argument about what the resulting model has learned.
The useful distinction is controlled generation versus unexplained recursion
The research record supports neither a blanket ban on synthetic data nor an unconditional celebration of endless generation. Carefully constructed examples, external checks, retained source information, and well-designed mixtures can support useful learning. Recursive replacement can lose information under studied conditions. The details of the loop determine which result is relevant.
For an institution evaluating a model trained with synthetic material, the first request should be the generation and selection history appropriate to the claim. Ask how correctness entered the pipeline, how diversity was measured, how original information was retained, and how the evaluation remained independent. A supplier may have legitimate reasons to restrict some details, but the restriction should be visible in the strength of the conclusion.
The most useful experiments compare documented pipelines at controlled budgets and test both improvements and losses. The most useful datasets preserve enough lineage to support correction. The most useful public reporting distinguishes a research result from an extrapolation about every future model or the entire internet.
Synthetic data is not a single substance with a fixed effect. It is the output of a process. That process can turn verified information into effective practice, or turn uncertain guesses into an apparently massive body of evidence. A provenance ledger helps readers tell the difference by keeping the ancestors, decisions, and checks attached to the examples.
The final question is therefore concrete: what new, reliable information or useful variation did this generation step add, and how was that demonstrated? A dataset that can answer that question has a stronger claim than one that offers only a large number and a reassuring label. Its value comes from a traceable learning process, not from being human or synthetic in the abstract.