Two editions of a fictional community workshop handout describe the same ordinary supply-return desk. One says, “Leave the card in the wooden tray.” The later edition says, “Bring the card to the welcome table.” A model returns a quotation that neither repeated exactly and labels it as the latest version. The facility name and general meaning sound convincing. That is precisely why the false quote is likely to survive a quick edit.
Quotes make two promises at once. Quotation marks commit the writer to a wording relationship, while the attribution commits them to a particular source. A paraphrase can preserve meaning without copying exact words. A quote can match the wording of one edition and still be wrong if attributed exclusively to another. An audit should decide which kind of statement was requested, which source was in scope, and whether the output actually provided either.
This is a safe, synthetic exercise using invented handouts, not authorization to transmit private material or quote outside a real source's reuse rules. The example sentence fragments are original teaching fixtures. For a real publication, follow its ordinary rights, editorial and disclosure process.
AI disclosure: Drafted with AI assistance. Handout editions, examples and test results are fictional, not claims about real sources or model evaluations.
Decide what the answer is allowed to call exact
A direct quote should represent the actual source wording according to your intended editorial conventions. A paraphrase should not wear quotation marks. A translation should be labeled as such, including which wording was translated. An abbreviation or other editorial convention can be legitimate where permitted and clearly represented, but it isn't a license to write a better sentence inside unchanged quote marks. Define the requested output before asking which model performed better.
In a synthetic prompt, ask for the exact line about returning a card from Edition 2, with a separate plain-language explanation if needed. Quote and explanation should occupy different fields. That prevents a useful summary from passing as a direct quotation. It also lets the person reading identify where interpretation begins without consulting the raw prompt.
A reference answer can name the precise edition, page or section, the permitted fragment, and material adjacent text affecting interpretation. Avoid specifying the model must reproduce cosmetic layout you don't care about. One workflow may ignore soft line wraps and repeated indentation; another examining a poem may need line integrity. Specify what normalization is permitted in your fixture. Don't retroactively change that policy when a favored output is close.
The goal is not to produce a perfect machine-readable definition of all quoting. It is to state what exactness means for the intended test. Don't require one true wording for paraphrase when several natural alternatives preserve the supported claim. Do require a quote to be grounded in the specified text, or explicitly label it as not available. A test valuable enough for release should identify the difference.
Give each edition a receipt that survives a download
Start with two authorized, identified source snapshots. Use a stable source record, title as printed, edition label, date if explicitly stated, and local reference or approved link. Keep page labels separate from PDF page positions when they don't line up. Document whether a record represents the entire handout or only an excerpt. If a later model lacks a page, reviewers need to know that before calling its abstention unnecessary.
A fictional source set can contain Community Studio Card Guide v1 and v2. Label the example text as synthetic. A review register may identify v1 by an asserted edition label from the cover and v2 by a controlled release ID. If the filename is guide-latest.pdf for both downloads, it is not the identifier. A source retrieval date records when you obtained the copy, not the date the handbook's policy became applicable. Quote fidelity and policy effectiveness are separate questions.
A source can be accurately identified even where private details are unnecessary. Retain only what your verification needs under authorized access, and prefer synthetic excerpts for method testing. If a provider or tool should not receive a live handout, don't send it to see whether it gets the quote right. Use the same exercise with original invented text. Reader receipts are useful when they don't require distributing more than the person needs.
The W3C Web Annotation Data Model describes ways to associate an annotation with part of a resource, including selections based on a quoted text segment and nearby context. A small operation can use the principle without a formal annotation framework. When a fragment occurs in more than one place, add a section or brief surrounding cue so a second reader has a reliable way to find it.
Make identical sentences a deliberate fixture
A pair of documents often shares most of its wording. An exact sentence found in both snapshots doesn't establish that a model uniquely retrieved the intended version. It does establish that the sentence occurs in the named edition if verified there. This is not a failure when the task just asked for what v2 says. It matters if someone claims the phrase was introduced in v2, proves its date based on the wording alone, or requests a sentence that distinguishes the two documents.
Consider the invented shared phrase “Keep the counter clear.” Both editions use the line. An answer that quotes it and cites v2 accurately can be valid. An answer saying “v2 added this instruction” is unsupported in this fixture. A third answer quotes it, cites v1 and then says v2 has no counterpart. Again the exact text is right and the proposition about difference is wrong. Grade these as separate properties rather than giving the most fluent answer a high overall impression.
Use a reference row noting duplicate text and where it occurs. Look for differences around it, such as a header change or a section that applies to a different activity. A shared sentence can be associated with a new decision without changing its characters. If a summary says “leave all items at the counter” from a line only instructing staff to keep it clear, it has added a meaning about placement even though it quoted the shared fragment elsewhere correctly.
An evaluation can choose a combination of unique phrases and intentionally shared ones. Unique phrases test wrong-version import. Shared ones expose exaggerated novelty or unsupported certainty about origin. Include one pair with comparable headings so using a visually similar source isn't enough. The tester should discover which evidence supports each expected answer, not write a score key after reading one system's result.
Work through a basic exactness and attribution matrix
Use a simple invented record:
| Edition | Section | Available exact phrase |
|---|---|---|
| V1 | Returning the card | “Leave the card in the wooden tray.” |
| V2 | Returning the card | “Bring the card to the welcome table.” |
| V1 and V2 | Keeping the area open | “Keep the counter clear.” |
Now imagine these response types to the prompt “Give v2's exact instruction for returning the card.” Answer A: “Bring the card to the welcome table,” attributed to V2. Depending on the test's stated punctuation convention, removal of the sentence-ending period may be allowed or need correction. Answer B: “Leave the card in the wooden tray,” attributed to V2. It is an exact match to V1's wording under the same convention and a wrong attribution. Answer C: “Leave the card at the welcome table,” attributed to V2. This may be a believable paraphrase in some contexts, but it is not the exact line. Answer D: “V2 is not available in my supplied context, so I can't quote it.” That may be appropriate if V2 was actually withheld; it is incomplete if the full relevant text was plainly supplied.
A strong grader doesn't ask for the best sounding sentence. It marks exactness under the declared policy, edition support, request fulfillment, and absence of unsupported statements. If some outputs provide both a quote and correct explanation, grade those independently. A model can get the quote right and claim it means a customer should leave without speaking to staff. If the supplied line doesn't establish that interpretation, the explanation introduced an unsupported consequence.
Change the prompt to “Summarize how the return process differs without direct quotes.” A faithful answer shouldn't be penalized for not reproducing exact text. Check temporal orientation and scope: V1 used a tray, V2 directs the card to the welcome table. An answer claiming the whole counter is now closed supplies more than the handout difference. The scorecard should stay consistent with the task rather than reward memorizing one quote format.
Stop a visually plausible combination from becoming a quote
A common error blends parts of two source sentences into a coherent third one. The model knows each half sounds familiar. The source has no such sentence. Depending on the requested task the content might be an accurate synthesis with proper attribution, but it cannot be a continuous verbatim quotation. If context is incomplete, do not join two good fragments with an invisible transition.
Consider V1 and V2 of a synthetic art-room cleanup sheet. Both include “Put unused paper in the rack,” but V1 says the label faces the aisle and V2 instead says the marked side faces up. The output “Put unused paper in the rack, label facing up” may be useful as a paraphrase if the actual V2 wording and object support it. It is not an exact V2 quote unless the source contains precisely that line under the stated conditions. Even a minor change can matter when the goal is proving the author wrote a specific phrase.
Splitting material over paragraph boundaries produces its own risk. A short target could be the first line of a paragraph and its limitation in the second. If the excerpt omits the limitation, a real reader may infer a broader claim even when the quote is exact. Use the relevant context in the output or withhold the fragment as misleading under the intended release rules. Quotation fidelity doesn't absolve the author from selecting an honest representation.
Watch for unattributed substitutions like “later,” “earlier,” or “actually” inserted near the quotation. They can make an exact fragment support a change narrative the evidence doesn't establish. Suppose V2 introduces a new reader instruction but the effective date is unknown. “The new instruction now applies” goes beyond matching the V2 wording. Say what the identified edition contains, not what you believe started happening outside it.
Define a useful response when the requested edition isn't available
An evaluation should include more than normal complete inputs. Give a fixture where only V1 is authorized but the prompt asks for a V2 fragment. The correct answer does not quote the older version under the newer label. It can state that the supplied record is V1 and V2's line cannot be verified. If the requested task allows showing the older comparison, label it explicitly. Do not silently present it as a satisfactory answer to the exact V2 request.
Now give V2 without the relevant section. A model can state that it didn't locate the line in the supplied excerpt. That is different from “V2 contains no instruction about returning cards.” Scope matters to absence claims just as much as positive claims. The latter needs complete appropriate evidence and a review capable of finding it. Be specific: “not in the excerpt I received” preserves the distinction.
A harder synthetic case gives the page plus a visibly unreadable character or line ending. An OCR output supplies a plausible phrase. An answer that labels its reading as uncertain can meet a cautious assistance goal but not a verbatim assurance. Provide a way to request a clearer authorized copy or mark the quote unverified. Do not reward a model for rewriting the missing text into something that sounds like the rest of the handout.
Abstention is not the universal best answer. A complete, unambiguous V2 case should reward finding and precisely attributing the requested line. An assistant that claims no information for everything avoids false quotes but doesn't perform the task. Keep missed supported answers separate from invented quotations. Useful behavior adjusts with the evidence; your tests should show both branches.
Check what was captured, not what should have been captured
Two software systems might report different answers because one was given a narrower excerpt. Record the actual fixture, its edition scope and permissions, and any system or retrieval differences. If the change is explicitly a retrieval change, those differences may be the experiment under test. If you are comparing writing approaches on the same source, retrieving different pages makes the test less diagnostic. Name the intervention before collecting examples.
A compact receipt can identify input snapshot, prompt revision, answer ID, observed response and limits. The W3C provenance data model distinguishes records, activities and agents. Thinking in those terms keeps an original edition, the process extracting it and the signed reviewer decision from becoming one untraceable paragraph. You do not need to implement a semantic data system to preserve those boundaries.
Do not repair output silently before grading. A reviewer noticing that it used the wrong quote marks or sentence can correct the eventual article according to editorial policy, while the experiment retains the original observation. Note whether the repair was a typographic convention allowed by the prior scoring policy or a real change of quoted wording. Otherwise future readers won't know whether the model or a human achieved the observed final result.
Keep blind review simple and honest. Hide system labels and recommenders where practical, but don't rewrite substantive artifacts to remove recognizable style. A writing style may reveal which tool likely produced an answer. That possible limitation can be recorded without invalidating every factual check. Edition support, exactness and unsupported claims can still be tested against the same source in either case.
Choose when whitespace is cosmetic and when it carries meaning
An exported PDF can insert a newline before a word that didn't start a new sentence. A document reader may normalize a repeated space or remove a discretionary hyphen. A quote audit ought to treat these changes according to the task. A short prose quotation review could ignore wrapping while preserving sentence punctuation and wording. A textual study could need the original break. Get that distinction into the fixture before grading.
For a synthetic card guide, imagine the visible copy wraps “welcome table” across two lines because a narrow patch of text is beside a photograph. A source-grounded answer can preserve the two words and produce them on one line if the declared convention allows it. Now imagine a bullet starter that says “optional” on one line and the action beneath. Removing the layout break may be harmless; removing the label's scope is not. A figure capture or copied text can hide whether a word introduced one bullet or the whole list. Compare the original when hierarchy has meaning.
Keep a minimal allowed normalization list. You might normalize line endings, spaces between words and a clearly documented typographic quote style. You should not silently expand abbreviations, correct spelling, change capitalization when it distinguishes identifiers, discard minus signs, or reinterpret punctuation around a negated phrase. Some real citation formats have controlled ways to indicate alterations. Follow the relevant editorial convention instead of inventing a new one to upgrade a near-match to exact.
Build two test patches. In the first a source wraps automatically in a plain prose paragraph. In the second the punctuation changes the reader's understanding. The grader should treat them differently even if an exact byte diff flags both. This is precision in policy rather than new technology. Leave original text available under authorized access so a qualified reviewer can recognize the difference instead of depending on a tool that already normalized it.
Compare an internal quote with an outward statement
An editor may do a source check for a colleague who is allowed to see the document, then publish an article to readers who are not. That transition deserves its own check. Authorization to read or quote in one context doesn't grant right to distribute the full exhibit or paste a restricted excerpt elsewhere. This exercise uses original short synthetic passages so the verification method doesn't require a private attachment. In practical work, use the actual rights and access process.
Keep the test output proportional to the question. If the requested evidence is an exact sentence, don't paste the whole section in a reusable prompt or outward report. A short phrase plus a stable anchor might be sufficient for the authorized reviewer. In public, an approved paraphrase or source reference may serve readers without reproducing a needed private document. Do not claim it as exact if asked to describe it without sharing its words.
A question about attribution can also reveal unnecessary people data. A document title and revision can establish where the passage occurs without naming everyone who edited or reviewed the draft. A reference file might contain notes that the source owner hasn't made public. Use fixtures stripped of real details when evaluating a general quoting technique. A faithful quotation doesn't make unrelated metadata suitable to transmit.
Reviewers can document “wording verified in restricted source, not approved for outward quoting” without recording the wording where it doesn't belong. That result deserves a different state from “can't inspect the source” and from “wording received is wrong.” One is a release boundary, one an evidence boundary and one a correction. A single unapproved flag covers all three and helps nobody choose the next permissible action.
Work an example where the document contains both passages
A fictional V2 booklet has a short current procedure and an appendix explaining how things used to work. Its current text says “Bring the card to the welcome table.” The appendix quotes the V1 instruction about leaving the card in a tray. Searching V2 finds both. An answer saying “V2 contains the words ‘Leave the card in the wooden tray’” can be true with the appendix reference. An answer claiming those are V2's current return instructions is false under this fixture. Text presence and source-role attribution need different checks.
Identify whether an excerpt is the document's own voice, a clearly marked historical extract, a communication quoted from another author, a sidebar, or an example. Citation to a page may be necessary but not sufficient if the roles are mixed there. A reviewer reading only a cropped search hit can miss the section heading introducing the quote. Preserve enough context to see how the source uses the text and name the passage's role in the receipt.
Practice by asking two different questions of the same synthetic fixture. First ask which wording appears in the appendix; then ask which instruction the current-procedure section gives. The first should return the earlier wording as embedded historical material. The second should return the new line. If a system always picks whichever quote comes first in the document, the pair reveals the wrong heuristic with very little input. If both are visible and the author's intents are clear, the assistant need not abstain merely because two statements occur in one file.
The same concept applies to footnotes and comparisons. A handout might cite an old welcome desk arrangement as a contrast, not ask the reader to follow it. A title like “Changes since last season” can carry important temporal context. Do not fold its old and new columns into one continuous instruction. When you cannot determine which source voice owns the statement, report that the wording is found and the interpretation remains open.
Audit a difference claim as carefully as a quotation
A user often wants to know what changed, not merely whether a line can be found. A model could correctly quote V2 yet claim that V1 never addressed the topic. The negative comparison needs a thorough enough review of the relevant V1 scope. If that earlier copy has only a few pages available, the approved answer may be conditional rather than absolute. This does not diminish the verified V2 quote; it narrows the change claim.
Work a synthetic update that moves a section without changing the wording. V1 has a supply return note under Workshop Exit. V2 puts the identical note under Desk Map. A generated change report says a new rule was added. What changed may be location in the handout rather than substance. Compare both the exact fragment and the associated scope. If the audience or conditions changed, the same quoted statement could be operating differently under the new structure. Don't declare no change based only on string identity.
Try the opposite: V2 keeps section title, formatting and opening clause but changes a condition from before noon to after lunch. If the model quotes only the identical beginning, it misses the part relevant to the question while technically using real words. An audit should check coverage for the material difference, not simply the presence of some textual overlap. An answer can report the unchanged portion and say that the edition added or replaced a time term if the full source supports that description.
Save both observed excerpts and the comparison conclusion. “Exact in V2” is a verification result; “new in V2 for this audience” is a comparative claim. Neither inherits approval from the other. In a real workflow, qualification about effective dates or governing policy may require an authorized decision outside the writing test. Respect that boundary. The utility of the quoted-version audit is that the reviewer or decision-maker starts with a clear source relationship rather than a confident blended line.
Make disagreement resolvable without a vote on style
Two reviewers may mark a fixture differently because they consulted different document versions. They may disagree about allowed punctuation treatment or whether a nearby phrase forms part of the quote. Treat their disagreement as diagnostic. Ask each to point to the original, quote the permitted fixture language, and identify which scoring clause they used. If there is still a material gap, document unresolved instead of forcing a count. Put preference about phrasing aside until the exactness and attribution checks are established.
Separate three cases. A missing document means this fixture may be unscorable for the requested claim. A rubric ambiguity may need a new documented version applied to both outputs. A genuine output defect should be logged where it occurred. Unmasking system identity before solving these issues can make a team tempted to excuse one kind of change differently for each answer. Apply the same policy to both.
A short audit table might have exactness, correct edition, relevant section, faithful meaning, source scope and missing information handling. An answer that says “not in the admitted excerpt” can pass cautious handling while not supplying the requested quote. A direct quote with a wrong edition fails attribution even if the words appear elsewhere. This gives a release owner more actionable information than calling every nonpassing answer hallucination.
The NIST Generative AI Profile reports confidently erroneous outputs and human overreliance as risks for generative systems. A plausible but invented quotation is a particularly clear example: styling can make a generated sentence look more authoritative than an openly tentative paraphrase. A structured independent reread is a practical control for this workflow, not a claim that a small scorecard neutralizes every form of risk.
Confirm the observed environmental differences
A quote test depends on what each system actually saw. One model might get the entire source through a permitted file reader while another receives text copied without section labels. If the task is measuring downstream usability of those actual configurations, that discrepancy is part of the test. If it is measuring how two writing prompts handle the same evidence, the gap makes the comparison invalid. Neither choice is inherently better; the key is agreeing which question the data can answer.
When comparing the same context, use a simple manifest listing identical sources and scope, prompt version, allowed transformations and exact answer objects. Don't give one system a follow-up request to correct an obvious miss while scoring the other's first output. If revision is part of the workflow, allow the same scripted correction opportunity and score first and later results separately. You can learn a lot from a model that identifies a correction honestly, but that shouldn't become evidence the initial quote was faithful.
Include only systems that can appropriately handle the material. A test restricted to synthetic handouts removes a needless reason to move private documents. If you cannot verify whether a tool retained or retrieved an older edition, state what the observed response suggests and name the limitation. A wrong answer can be diagnosed as incorrect without pretending you know the exact retrieval path or why it happened. Preserve enough metadata to check known differences rather than estimating them in hindsight.
Check for accidentally shared clipboard or cached fixture text. One repeated quote across two supposedly separate runs might come from the same copied answer rather than independent responses. Confirm capture IDs and whether both artifacts were recorded as intended before using their agreement as evidence. Applying simple document rigor to the testing workflow itself prevents a second layer of plausible but ungrounded attribution.
Recheck after a reputable source updates in place
Suppose the community studio issued both of the sample handouts from the same public download address. One day the link returns the first edition; the next it returns the second. A reviewer clicks the link in a six-month-old note and sees an apparently wrong quote. The old quote may have been accurate for what was available when cited. A mutable destination undermines provenance if the receipt did not identify the copy, but it doesn't prove the earlier model invented the sentence.
A narrow maintenance process keeps the earlier observed edition identity, controlled snapshot when permitted, current publication link, latest retrieval observation, and the relation between them if known. If you can't legally or practically retain the whole old source, preserve an authorized anchor and the minimal evidence suitable for your needs. State that an old result can't be fully reconstructed if that is the case. Silence about a missing snapshot won't restore it.
Write a synthetic test where both links visibly say Current Guide while one is a local archival copy. Ask reviewers to identify which source is present, not which label looks more official. The current website banner might establish what it offers today. The archive's cover can document what textual edition it is. A future editor can then choose the question: reproduce the old quoted claim or find the instruction applicable now. These outcomes can differ even when both records are authentic.
Test a correction notice as a separate object. A source owner can post “a paragraph in edition 2 should refer to the lobby, not the counter.” That notice doesn't rewrite the bytes already captured. A report may need to explain that Edition 2 as distributed contained one line and a later authorized correction changed guidance. Do not create a continuous quote from the original plus the notice. Attribute each on its own and let the approved editorial process decide how to present the corrected understanding.
Check a paraphrase with the same care as a direct quote
A good paraphrase is not just an exact quote with punctuation removed. It restates the supported meaning without changing actors, conditions or timing. Suppose the synthetic guide says “After the host confirms the card, return it to the welcome table.” A paraphrase that says participants can return the card to the table at any time loses the condition. Another reading “take the confirmed card to the welcome table” probably preserves it in the intended exercise. Have reviewers name the controlling event rather than compare sentence style.
Cases involving two editions can require a clearly labeled comparative paraphrase. V1 and V2 may both use a desk, but the first asks to leave the card while the second requires showing it to a particular role. “Both ask visitors to stop by the desk” is vague enough to conceal a practical change. If the user asks what changed, include the distinction plainly. If they only want shared facts and the exact behavior isn't relevant, the narrower common summary may be fine with its scope marked.
Make this part of the verification score. A transformed statement can contain a correct source quote and still misrepresent the source through the surrounding interpretation. Conversely, a good summary may obey a no-quotation request without copying the source's most recognizable sentence. Score by intended task. When adding rubrics, give one example of a faithful paraphrase and one plausible failure. That keeps reviews from drifting into preferences about elegance at the expense of important qualifiers.
Treat a translation constraint separately. If asked to quote a guide in its original language and explain it in another, preserve the original in a recognizable quote field and present the target-language explanation as a translation or interpretation, whichever is intended. A translated rendering can be faithful without being an exact quote of the original string. Don't call it verbatim from the source as if the source contained the new words. The same formatting choices can otherwise confuse both human readers and automated exactness comparisons.
Finally, let the audience reconstruct which kind of source use it is reading. A short edition note can distinguish current text, archived text, and editor paraphrase without cluttering the main page. If the source location or scope could change what a reader believes, preserve it. The thing to verify isn't whether the explanation sounds like the authors. It is whether the output faithfully says what the reviewed edition supports.
End with a receiving editor's proof, not a polished average
An editorial result can be concise: “Of eight synthetic cases, exact fragments matched in six under the declared whitespace rule. One matched the other edition, one required abstention because the requested section was unavailable. Three explanations offered new unsupported implications and were separately corrected.” Confirm the categories and their denominators under your test. The attempt does not estimate a general quote accuracy rate, and the results shouldn't be advertised as the measured accuracy of real systems if the cases and counts were examples.
For practical use the main handoff is not a percentage. It is the reviewed passage, exact edition, nearby scope, whether the text is quotation or paraphrase, and what still needs a decision. Save the approval state for publication separately from correctness. A reviewer who tested V2's wording on a sample document didn't verify human names, policy efficacy, or all later copies. The exact result is valuable without pretending otherwise.
A narrow first pilot can test one direct quote, one paraphrase, one exact wrong-edition quote, one unavailable source, and one shared line that is incorrectly claimed to be new. Make a small answer key before reviewing any generated artifacts. Change one condition at a time and observe which outcomes move. An assistant that retrieves the right line should receive credit for that limited fact, while leaving the associated editorial or policy question to authorized review.
The important habit is simple enough to do by hand: open the relevant edition, find the exact text and its role, and verify what you say it establishes. If the document is unavailable, say so. If you paraphrase, remove the quotation marks and retain support. If your conclusion is about change, check the other edition too. Those steps give a reader an answer worth trusting even when a model can produce a more confident sentence in less time.