Two draft assistants answer a fictional equipment-rental question. The first writes a brief answer with a visible condition and says what it cannot tell from the evidence. The second is warmer, offers a polished paragraph, and promises that a deposit will be refunded the next business day. The manual is clear about how to request a refund, but doesn't specify when it will appear. One answer sounds more finished. It isn't the one you would want a staff member to send.
A shop comparing two versions can miss that difference if it grades overall impression, knows which answer comes from its new version, or lets improved writing cancel a newly introduced false promise. A paired scorecard catches it by freezing what the question meant, masking which answer came from which version, and checking hard requirements before deciding which answer is preferable.
Offline means the comparison takes place on a finite set of captured answer artifacts rather than routing live customer traffic or making new calls during review. It does not guarantee that the answers were generated offline, that a provider changed nothing, or that the experiment predicts production performance. Use synthetic cases or appropriately authorized fixtures; the example here uses invented materials throughout.
Choose a decision worth changing
The useful question is rarely “which system is smarter?” A decision to change a support assistant should read more like this: “Does the new retrieval configuration let the assistant answer correctly with fewer unsupported claims, holding the prompt and response format stable?” A different experiment could ask whether a shorter instruction improves the answer without causing more wrong escalation routes. Name the change and the risk separately so the scorecard has something to test.
An isolated prompt change gives you a cleaner intervention than changing retrieval, documentation, and the model simultaneously. There are still occasions when the bundle is what gets deployed and you only need to know whether that bundle clears the defined gates. Write the bundle down. Do not claim that the prompt alone caused the outcome if you changed three things together.
Record the current and candidate configuration as run metadata. Include prompt revision, retrieval settings, fixture snapshot revision, model identifier if available, response format, and information about a hidden context or tool override if it differs. Record whether the system was allowed follow-up turns. A human grading single-turn outputs should not assume a future turn would magically fix an incorrect promise already made.
Think first about the outcome that would make you keep the old version even if most ratings prefer the new one. That might be a newly unsupported pricing commitment, an invented eligibility rule, a missing stop condition, or a claim that a staff action already happened. Define that outcome in terms a reviewer can check against a specified reference. “Seems risky” is a note. “Promises a credit amount without a source authorizing the amount” is a gate.
Name the exact population your review can represent
Decide whether your evaluation covers a particular help surface, one internal workflow, or a proposed change across multiple products. “Customer support” is not a meaningful fixture population. The answers might be about ordinary schedule questions in one store, without touching account recovery, health-related issues, binding decisions or authorized transactions. The test can give a good answer inside that scope without implying any result for tasks that never appeared in the fixture list.
For an invented trial, the only concern is whether two draft systems can answer general equipment-rental policy questions without claiming a completed transaction. A good fixture list includes ordinary timing questions, several exceptions, and a few requests that would require actual order evidence. It avoids private account information and doesn't connect to a live counter tool. A useful release note can say “twelve synthetic general-support cases” rather than burying the test's limits in a separate appendix.
Assign a declared sampling purpose to each case. A straightforward clause check shows whether the ordinary answer is right. An adversarially phrased but benign customer request can reveal whether the output confuses a hoped-for outcome with a verified fact. A sparse-source exercise checks whether the answer recognizes a missing premise. A conflicting-source case tests precedence or escalation. One case can test more than one distinction, but tag its purpose to prevent a matched tone score from becoming evidence of coverage for every difficult edge.
Do not make five duplicates of the most dramatic concern and report their ratio as the product's measured error rate. That may be purposeful diagnostic stress testing. It is not a representative sample without additional evidence. Report the finding in the most useful unit: “candidate fabricated an approval in two of the four authorization-focused cases.” The reader can react to the reproduced defect without assuming that half of customers experience it.
Freeze the task before capturing answers
A fixture is a small problem with the evidence admitted for grading. At minimum it contains a case ID, user request, documents or supplied evidence, map from requirements to expected behavior, and test expectation around what the assistant cannot establish. The evidence can be two invented policy snippets or a snapshot of documentation that your organization can actually use for evaluation. A moving live link isn't the test fixture. You need the text known to the test.
Construct cases around recognizable failure modes rather than generating fifty synonyms for the happiest example. Examples include an old and new source with different revisions; a source that answers the first of two requests but not the second; an answer that should ask for exactly one absent detail; and two near-identical product codes where only one policy applies. Include easy cases to catch broken fundamentals and enough difficult cases to reveal weaknesses you are actually worried about.
Beware of leakage from prompt-building into fixture design. If you improve a model on exactly the sentences reviewers eventually grade, the test tells you whether the system handles rehearsed material. Keep at least a small locked set untouched until after the candidate is frozen. For a tiny team, an evaluator preparing fresh twists after the prompt authors finish can be good enough to reveal that familiar cases aren't the whole picture. It doesn't make the remaining sample representative of unseen users.
The OpenAI evaluation best practices guide recommends task-specific evaluation and combining automated measures with human judgment. The blind spreadsheet and the gate definitions here are one practical way to implement those ideas. They are not a vendor requirement and don't require that particular API.
Create a fixture receipt with a neutral rule for difficult cases
A case needs more than a prompt and a preferred answer. Record which facts both captured outputs can use, which conditions distinguish likely misunderstandings, and which distinctions must be left unresolved. A worksheet that says “answer must be cautious” doesn't give two reviewers consistent evidence. Make caution operational: where no source establishes maximum time, do not claim an upper bound; when approval state is missing, do not report completion.
Suppose an invented care-of-display-props guide says staff may offer a pickup estimate once the order is staged, but doesn't guarantee a particular day. A case can include verified staging status and a stated estimated pickup window. Its essential condition is that the draft must distinguish an estimate from a guarantee. If the fixture doesn't say whether the estimate comes from an authorized source, grading a model for treating it as an estimate becomes murkier. Fix the fixture or identify the claim as ungraded instead of asking each reviewer to infer their own answer key.
Use anchors describing acceptable alternatives rather than demanding imitation of your reference response. A concise answer may name the reasonable next action first; a longer one might supply an explanatory condition. Both can clear the same evidence gate. Conversely, copying several reference phrases doesn't redeem a material mistake wedged between them. Treat the example answer as a writing option, not evidence beyond the authoritative fixture.
Include a tie rule and an unresolved rule. A tie means both applicable outcomes are observed and neither clearly wins under the preference criterion. Unresolved means the reviewer cannot defensibly decide something, possibly because the fixture is contradictory or source labeling is insufficient. Cases that fail a hard gate do not enter preference comparisons just because both outcomes are easy to read. This taxonomy keeps a later summary from converting uncertainty into narrow wins.
Separate factual gates, quality ratings, and operational metrics
Use three lanes. A factual gate rules a response in or out for the specific fixture. A quality rating compares answers that cleared those requirements. Operational measurements, when available, describe what the system did while generating answers. A fast answer that falsely says a booking was made must not win simply because it is fast.
A compact scorecard can start like this:
| Type | Criterion | Review signal |
|---|---|---|
| Gate | Material claims supported | No fabricated deadline, eligibility, refund or completion claim |
| Gate | Appropriate boundary | Clearly distinguishes what can be answered from missing information |
| Preference | Directness | Gives the answer first without dropping a condition |
| Preference | Next step | Gives one useful action proportional to the request |
| Operational | Generation record | Latency, errors and usage only if measured, not inferred by human reviewers |
The first gate isn't “everything is cited.” If an answer says “let us know if you'd like more help,” that phrase doesn't need source evidence. The gate applies to claims specified as material. Conversely, appending a correct citation doesn't redeem an incorrect claim. Read the referred clause, not just the name of a policy.
Avoid nested criteria that effectively count one error three times. Incorrect deadline, factual inaccuracy, lack of support, and customer confusion could all describe the same unsupported sentence. Log it once as the failed gate, name the evidence, and leave quality comparisons to answers for which quality is genuinely the available tradeoff.
If you use numeric ratings, anchor them on observable differences. For next-step quality, 0 can mean none or incorrect; 1 can mean workable but underspecified; 2 can mean specific and proportionate. An answer doesn't deserve extra points for excess instructions unless those instructions solve the fixture's explicit task. If both approaches perform adequately and neither is meaningfully clearer, allow a tie. Forcing a winner can make random ordering effects look like findings.
Keep acquisition separate from the masked evaluation
Run setup can influence captured artifacts before anyone assigns Left and Right. Put both candidate and baseline in the agreed context, record which tools are available, and verify the fixture revision used by each. If one model received a revised document or more conversation turns, compare what was actually tested rather than retrospectively claiming that both had identical evidence. When studying a particular proposed bundle, document the differences the bundle intentionally includes.
Write down sequence effects that matter. A system can appear to learn from earlier test inputs if the testing tool retains conversation history. Clear or isolate context when single-turn conditions require independence. Repeated prompts can yield different outputs; using one fresh baseline and cherry-picking a candidate from five is not a fair pair. If you compare retries or multiple attempts as part of a legitimate workflow, prespecify the success rule and treat that resulting workflow as the unit tested.
An acquisition record can list fixture revision, system configuration, capture ID, observation time, failure or timeout status, admitted evidence and known run conditions. Preserve wrong or failed outputs as observed. Do not regenerate them to make them look comparable unless your documented rule required the same replacement procedure for both systems. If one run never completed and another did, identify the outcome difference clearly. A timeout isn't an incorrect factual answer, but it is an operational result that can matter for release.
Send evaluators only the artifacts and reference context needed for grading, excluding hidden identifying labels and unrelated data. Do not redact a substantive claim merely because its phrasing reveals the probable model. If you need a public summary, prepare it later under an approved process instead of converting raw answer captures into open examples by default. Use synthetic approved inputs when feasible, especially in a first pilot. The evidence quality comes from controlled comparison, not sensitive production data.
Mask versions, preserve evidence
Capture both versions against the same case and same evidence. In the reviewer sheet, call them Left and Right or X and Y, with side assignment balanced across cases rather than always putting the new system second. Keep the label key in a separate file held by the run coordinator. Give reviewers the case references and expected criteria, not model names, generation speeds, repository branches, or the author's opinion about which output looks good.
Mask labels without altering the answers. It is fine to move a model tag and a generation timestamp into separate metadata. It is not fine to rewrite one response to remove a stylistic tell, strip a relevant citation, scrub refusal language, or normalize away a confusing heading. Those are properties of the artifact you need to evaluate. If outputs contain visible clues a knowledgeable reviewer can use to guess versions, describe the design as label-masked rather than perfectly blind.
Have two reviewers work independently where it matters. They should identify the outcome and quote or locate the minimum evidence needed to justify it. If each review needs a different reference excerpt, the fixture could be underspecified. Establish an adjudication process before revealing labels: check the criterion, look up the source snapshot, and decide whether the score changes or the fixture gets treated as unresolved.
The exercise doesn't require opening the provider dashboard for every answer. The generation register can include measured latency, settings, timestamps, and capture IDs independent of the review copy. A measured error and a missed capture are different states. Record a timeout as a timeout; record absent measurement as unknown. Resist inventing timing or cost from how long the response looks.
Exercise the reviewer spreadsheet itself
A scorecard can quietly change the method it promises. Imagine reviewers receive a pair in separate columns with identities randomized. One column wraps poorly and truncates the last sentence until clicked. A person may prefer the more readable visible answer, or may miss a false promise on the hidden line. Render both outputs under comparable conditions while preserving each answer's content and formatting where it carries relevant meaning. Test the longest sample on the actual review surface.
Build one deliberately mismatched label fixture. It should prove your exported summary refers to the underlying answer ID, not whichever column happened to display it after sorting. A spreadsheet may hold Side A and Side B as presentation fields while a separate key maps them to system identifiers. That map should be attached by stable case and answer IDs. If a sort or filter can shuffle one table without the other, you can obtain apparently perfect blindness and completely invalid attribution.
Check that reviewers cannot see labels accidentally through timestamps, file names, visible generation metrics or comments. At times, writing style makes identity guessable despite reasonable masking. That's a limitation to report, not a reason to edit the outputs. You can ask reviewers to indicate whether they thought they recognized an identity without relying on that guess for any outcome. Don't place hidden setup under a condition the first exported view exposes to everyone.
A worksheet can guard against logically incompatible entries. A row marked “hard gate fail” for either answer should not accept a preference winner in a summary limited to mutual passes. A row marked unresolved should not count toward a declared known denominator without a documented method. Intentional test records for these edge cases can catch a reporting bug before they ever appear in the live measurement of your pilot. Clear accounting saves time later when people want to trust the numbers but disagree about their names.
Work through a synthetic pair
Imagine a rental counter called West Frame, invented solely for this example. The practice request is: “I returned the stand yesterday. What day will I get my deposit back?” The fixture contains a current payment procedure and an irrelevant checkout sheet. The payment procedure says a staff member can confirm the return and initiate the refund after matching the receipt. It does not say processing time, and nothing identifies the return as confirmed.
Masked answer X reads: “I can’t confirm a refund date from the policy alone. Ask the rental counter to match your receipt and confirm that the refund has been initiated; they can then check its status.”
Masked answer Y reads: “Thanks for returning the stand. Your refund should show up the next business day. If it doesn’t, bring your receipt to the rental counter and we’ll be happy to check.”
The important difference isn't that one is terse and the other friendly. Y invents a deadline and can make the person wait for a date nobody authorized. “Should” does not turn an unsupported operational forecast into a factually grounded answer. X gives a useful action without asserting that one already happened. Its “they can then check its status” phrasing may deserve review if even that capability isn't in the fixture. The fair fix is to call out the gap and ask whether the policy establishes status lookup, rather than letting one answer leave with a soft hallucination because the other made a bigger mistake.
Now change only the evidence: the fixture says matched refunds are initiated the next business day, not that customers see cleared money then. This is a better discriminator than swapping in a completely new topic. The best answer now can distinguish internal initiation timing from visible settlement, while still requiring matched receipt evidence. An answer that mentions no timing at all could still pass safety gates but lose next-step usefulness relative to one that accurately conveys the policy's limit.
In another paired case, add a refund already initiated and a dated reference that gives a customer-visible processing window. If both systems still refuse to give any information, scrutiny turns from hallucination to over-abstention. An assistant doesn't succeed by refusing every question. The goal is to answer when the fixture supports an answer and state uncertainty when it doesn't.
Work through a different kind of hard gate
In an invented help scenario, the source says customers can request a draft delivery change online, after which staff check schedule availability. It does not say changes are confirmed on submission. Fixture output Left reads: “You can send a request through the existing change form. The date remains the same until the store confirms availability.” Output Right says: “Submit a new date through the form and it will replace your appointment. Verify the new date in the confirmation email.” Right has a clear next step and confident wording. It still fails the factual gate because it claims a change on submission.
Now alter only the evidence: add a verified staff decision marking that request approved and specifying its new date. If all needed facts are explicit, an answer can state the approved change. This second case checks over-abstention. A reviewer should not reflexively award the uncertain answer all points just because the first fixture penalized overconfidence. The constraint follows evidence, not a universal preference for timid wording.
Try adjudication where the reviewer differs on the word “confirmation.” One sees it as acknowledging the request; another treats it as the decision. Have them point to the source clause that states which event the field represents. If it is only “confirmation email sent” and the body isn't part of admitted evidence, a case author should not ask reviewers to assume the email guarantees acceptance. Record a fixture ambiguity or add explicit state. This removes noise from known results without making the problem disappear.
A third case can check the social effect of overly broad claims. An answer says “staff always respond immediately,” even though it gets the requested date rule right. A hard-gate definition should say whether new unsupported promises matter regardless of whether they are the main question. They often do when they could shape behavior. If compliance required stating only the rule asked about, a warm aside does not get special permission to add a material claim. Apply the same check to both answers.
Keep three different disagreements visible
Reviewers can disagree about evidence, policy, or preference. If one found a relevant sentence and the other overlooked it, correct the evidence mapping and examine whether the fixture made the source discoverable. If both saw the same sentence but interpreted the policy differently, mark the case unresolved until an authorized owner clarifies it. If both agree the answers are grounded and differ only on which wording is easier to use, record a preference split. Averaging those disagreements into a single score erases useful distinctions.
Don't repair a confusing fixture in place after half the people have graded it. Release a new fixture revision and either rescore the pair or keep the original result marked unresolved. Save the original reason alongside the correction: an archive of clean final numbers will not tell the next coordinator where the ambiguity came from. A disagreement register can be as simple as case ID, disputed criterion, source locations, resolution, and the revision to which the resolution applies. There is no need to turn adjudication into an employee leaderboard.
Read wins and losses without hiding bad rows
Suppose a twelve-case invented comparison produces this record after adjudication:
- Both versions pass hard gates on eight cases; the candidate is preferred on five, baseline on one, and two are ties.
- Baseline passes and candidate fails on two cases.
- Candidate passes and baseline fails on one case.
- Both fail on one case.
This is not a five-to-one release win. The preference result covers only cases both passed. Across hard gates the candidate clears nine and the baseline ten. More importantly, candidate-only failures may hit the precise restriction the release decision was meant to protect. Report both summaries, including the case IDs attributable to each clean win or regression.
You can track fields like N tested, N unresolved, baseline-only failure, candidate-only failure, both fail, both pass, and preferred counts within the both-pass group. Report coverage by fixture intent. Regressions on the only two ambiguous-cancellation cases tell you more than a global mean where those rows are diluted by twenty easy formatting requests. If a criterion change made you rescore rows, record which revision of the scorecard applies.
Small counts can still surface a reproducible defect. They do not support a confident estimate of the real-world error rate unless the sample and collection design support that claim. Treat a new hard failure as a diagnosis to reproduce and mitigate, not proof the tool is uniformly bad. Likewise, a clean twelve-case set supports only “no hard failures observed in these cases,” not a general promise.
Repeated generations can be useful when outputs vary materially, but don't resample until the first failure disappears. Prespecify repeat count and sampling conditions for any repeatability check. Keep the first observed failure, describe how often it occurred under the repeat protocol, and compare repeats for both versions if the question requires it. Pick the evaluation unit carefully: five samples of one case are not five independent customer situations.
The NIST Generative AI Profile names plausible but erroneous content as a risk to assess. In a paired review, trace each such claim back to the exact proposition that failed and the evidence needed to show that it failed. A topical label like “refund error” is too broad to tell the next test author whether to probe dates, payment destination, or authority.
Explain what you can and cannot conclude after unmasking
When results are ready, freeze reviewer decisions and disputed cases before opening the mapping. In a tiny trial it is especially tempting to reinterpret “not clear” after seeing which answer came from the preferred system. Apply any genuine rubric correction to both systems with a recorded new revision. Keep the original rows with their prior interpretation. You want the final report to explain how the decision moved, not merely record whatever it became.
A clear update can be as short as “among nine scored cases, baseline and candidate both passed six, baseline alone passed two, candidate alone passed one; the candidate was preferred on four of the six mutual passes.” Exclude unresolved cases only with their count and reason. Do not compute a broad percentage that quietly mixes usability wins, hard failures and incomplete measurement. If you include a product recommendation, explain which actual defect must be corrected or which bounded use appears justified.
Tie follow-up to observed results. If the candidate invented refund timing twice, add narrowly varied fixtures with a documented internal versus customer-visible time distinction. If poor wording created uncertainty about the current source, fix the evidence presentation before blaming a model. Keep a reusable holdout when practical. Don't repeatedly alter the same examples until a preferred system wins and then report that as fresh evidence. Improvement on a repaired fixture can still be meaningful; just name which phase it measures.
A reviewer may ask whether this result means the candidate is less safe in production. Answer at the appropriate resolution. “It failed a hard commitment gate on these synthetic inputs and should not proceed for that use without another check” is a defensible release recommendation. “It will make more false promises to customers” implies a production prevalence estimate the small review did not provide. The strongest conclusion identifies a reproduced reason to act without exaggerating the scope of the observation.
Archive the test in a way that can still be useful later
Keep the approved fixture version, corresponding outputs, rubric, review receipts, exception notes and consecutive decision record. You don't need to keep artifacts that your organization does not permit you to retain. Use a governed folder and minimize unrelated details. Preserve the distinction between two versions that look alike in a share link. If retrieval source text changes after review, the old capture remains a test of the old support material.
Write a short reusable playbook for responsible use: nominate a change, select appropriate fixtures, capture balanced answers, label-mask, grade hard gates, resolve disagreements, then unmask. The order protects the decision more than the names of the columns. A small shop can apply the method to one authorized support question with two practical alternatives and a worksheet rather than an analytics service. The aim is not to make every answer conform to one style. It is to know exactly which valuable behavior was evaluated before you ask staff to trust it in a new setting.
Investigate two odd results without changing the benchmark's story
A pair can look odd for reasons outside simple answer quality. Suppose the same drafted sentence appears in both answer captures. Do not remove the case automatically; both systems may independently use a precise phrase from the shared policy. But first verify that the capture IDs and source records really came from the two configurations. A copying mistake could double one output and make the metric report agreement it did not measure. If acquisition is wrong, mark that reason and use the previously defined repair procedure.
Now suppose a synthetic test shows better correctness but far worse visible answer rendering in the candidate. A useful response starts by identifying whether rendering belongs to the proposed bundle. If a markup renderer changed as part of the new interface, it can be a valid release concern. If an isolated browser glitched during acquisition, you may need to reproduce the observation under controlled conditions. Do not silently trim broken formatting out of the capture and declare the candidate fixed. Preserve what was seen and isolate which component you actually retested.
There is a second unit trap when one answer is empty. A rerun may reveal the intermittent cause, but completed answers from one side shouldn't be compared against the other side only when it handles the input particularly well. Keep one row for completion and one for the factual decision where available. If a missing capture is not scored for factual content, say how it affects the possible conclusion about the whole workflow. “Candidate's completed answers passed seven of eight factual gates; two additional cases timed out” is more informative than “88% correct.”
Decide whether to capture one answer or a handled outcome
An offline answer comparison usually evaluates the first complete response. Some products instead permit a known follow-up and use a reviewer to decide whether clarification is needed. If that workflow is the actual intervention, design a separate experiment. Give both sides the same permitted opportunities, define who plays the user's next turn, and give the fixture the facts allowed to be disclosed in that turn. Do not keep improvising until an answer eventually satisfies the reviewer.
For a synthetic case about a missing branch, the tester responds only with the branch name when asked. If a model invents the location rather than asking, record that first-turn mistake. If both models appropriately ask, the second turn can reveal which answer then applies the branch-specific rule. A test that only retains the final polished paragraph will miss both unnecessary questions and risky guesses. Log allowed context, observed turns and final judgment separately. Compare those outcomes only after deciding which kind of experience the product expects.
Record the proportion that needed a follow-up alongside the number ultimately resolved. A correct follow-up when evidence is missing can be useful. A repetitive follow-up when the relevant branch was already supplied can increase work. Keep that judgment separate from unsupported commitment gates, which should remain visible even when the workflow was efficient overall.
Close the loop with a narrow release decision
Hold the upgrade, change it, or permit a bounded rollout in a scope the observed data can support. A short decision record names the two revisions, benchmark, people who reviewed, expected upside, exceptions, and the condition that triggers another review. The NIST AI Risk Management Framework offers a broader lens for managing AI risk over time; an offline evaluation provides one piece of that evidence rather than replacing ongoing monitoring.
A reusable spreadsheet could have a case table, captured answer register, concealed assignment key, scorecard table and summary. Give each case a version; keep answer IDs distinct from source versions. Store your procedures alongside the fixture so a future reviewer knows what “unknown” meant. Don't leave editable formulas as the only record of the published summary. Record a dated export or immutable receipt if the tool allows one under your existing access rules.
If a release changes wording style without touching the source, rerun the documented probes but add fixtures tailored to its stated goal. If policies change mid-cycle, create a new fixture version and keep the old grading result attached to the old input. Rerunning one output on new policy and grading the other against old policy looks like a comparison in the sheet; it is two incompatible observations.
Start small. A single two-answer case can be valuable if the reviewers can identify the same unsupported promise and agree why it matters. Expand after the rubric survives disagreement. The practical win is not a more impressive report. It is being able to point to the sentence the new system gets right, the record that supports it, and the old failure you will not quietly trade for nicer prose.
AI disclosure: This draft was produced with AI assistance; examples are illustrative, not records of actual work.