A software vendor sends a completed accessibility report. It has a familiar template, a recent date, and many rows marked Supports. What does that establish about the product a customer is actually about to use? The answer depends on the product version, the scope evaluated, the explanation behind each status, and the evidence connecting the report to the deployed service.
The Information Technology Industry Council describes a completed VPAT as an Accessibility Conformance Report, or ACR. Its current page lists VPAT 2.5Rev, dated April 2025, with editions for different standards and markets. ITI also states that there is no VPAT certification. A completed template is a reporting artifact, not a certification badge issued by the template's owner. ITI's VPAT information and current editions
That does not make an ACR unimportant. A careful report can reveal precisely where a product supports accessibility requirements and where it does not. It can help a buyer ask targeted questions and a development team connect a public statement with detailed testing. The mistake is to treat the existence of the document as a substitute for reading its boundaries.
This article follows a fictional product called Portal Q through a document review. Every vendor, product behavior, user exercise, and commercial scenario below is invented. The one activity actually executed for this article was a small record-consistency script run on original fictional JSON files. It did not test a website, use a screen reader, certify a product, or validate an official VPAT or OpenACR document.
The focus is evidence literacy, not a determination of an organization's legal obligations. It does not restate ADA litigation or announce that one technical standard applies to every purchaser. The practical question is narrower: what does this particular report establish about this particular product, and what additional evidence is needed before relying on a broader claim?
1. Read the version numbers as a set, not as one date
Portal Q's fictional report identifies product release 4.2, report revision 1.1, and a stated standards target. Its template has its own version. Those labels refer to different objects. Updating the report date does not necessarily mean the product was retested, and updating the product does not automatically update the report.
Suppose a buyer is offered release 4.3, while the report still evaluates 4.2. That difference is not proof that 4.3 is inaccessible. It is a gap between the offered configuration and the evaluated one. The buyer needs a change description and relevant evidence rather than an assurance that a higher version number must be better.
Now introduce a second report revision. Revision 1.2 corrects a typographical error in a component name but describes no new testing. Calling it a new accessibility evaluation would overstate the change. The document can be newer while the underlying observation remains the same.
Conversely, imagine that the product version remains 4.2 but a customer changes a configuration that controls the navigation interface. The unchanged product number does not prove that the earlier evaluated configuration still matches the service. A useful report identifies material configuration boundaries, not merely the marketing name printed on a subscription invoice.
For our fictional review, create a compact identity line containing the product release, evaluated configuration, report revision, evaluation dates, and standards target. Add the exact artifact being considered: the hosted service, a native application, an exported document, or some combination. This identity line is an author-designed reading aid, not a new official reporting requirement.
The result prevents a common substitution. A report for a desktop authoring application should not silently become evidence for every document a customer exports with it. A report for a public portal should not automatically cover a separately supplied mobile application. The relationship may be close, but the scope still needs to be established.
At the end of this first pass, the conclusion is deliberately specific: the file describes Portal Q 4.2 under its stated conditions. Whether it supports a claim about the offered 4.3 configuration remains an evidence question. That is a useful finding before anyone begins counting the status labels.
2. Map the report's scope to the job the user needs to finish
Give Portal Q four modules: search, appointment selection, account administration, and document export. Its fictional ACR covers search and appointment selection. The buyer intends to use all four. The report therefore addresses only part of the intended service, even if every statement within its declared scope is accurate.
An evaluation should not be criticized merely for being scoped. A limited report can be entirely appropriate for a limited product or request. The problem arises when a narrow scope is presented as coverage of the whole customer deployment. The buyer's usage map and the report's scope map need to be compared explicitly.
For our case, the comparison has three possible states. Search and appointment selection are included. Account administration is excluded. Document export is not mentioned clearly enough to classify. Excluded and unclear should not be merged: the first is a known boundary, while the second calls for clarification of the document.
Suppose the export module creates a receipt that users must retain to complete their application. The scope gap now affects a required task, not an optional decorative feature. The importance of the gap comes from the intended workflow, not from the percentage of modules listed in the ACR.
A report covering two of four modules is not necessarily fifty-percent accessible, because the modules are not equal units of accessibility. One may be rarely used; another may block the only route to an essential service. A module count can describe coverage but cannot, without a defensible model, become a universal access score.
Configuration adds another dimension. The buyer may enable an external identity service or a customized appointment calendar absent from the evaluation. Our fictional report should not be stretched to cover those additions merely because the overall service still carries the Portal Q name. Their behavior and integration require their own evidence.
A useful review output is a scope mismatch list with practical consequences. For this case, it identifies two covered modules, one excluded module, one unclear module, and a required receipt task that depends on the unclear module. That list gives the supplier a concrete question to answer and the buyer a concrete boundary to preserve in any public accessibility claim.
3. Read a status together with its explanation
Section508.gov's ACR FAQ identifies the familiar conformance labels and says that partially supported or unsupported provisions should be accompanied by an explanation of what is not met. The explanation is therefore substantive information, not an optional footnote to a colored status cell. Section508.gov ACR and VPAT FAQ
Create two fictional rows. Row One says Partially Supports and explains that the calendar's month selector can be operated with a keyboard but the date-grid interaction has an identified limitation. Row Two also says Partially Supports but explains that a rarely used administrative export has an identified limitation. The same label does not tell the buyer whether the impact on the intended public task is comparable.
The reader needs to connect the exception to a component, a condition, and an observed result. An explanation saying some controls may be problematic is less useful than one identifying which control, under what configuration, and what the evaluator observed. That difference concerns the specificity of evidence, not the attractiveness of the wording.
Now imagine a row marked Supports whose explanation describes an unresolved failure in the same evaluated scope. That apparent inconsistency calls for clarification. It should not be silently resolved by keeping the favorable label and ignoring the unfavorable explanation, nor by assuming the label was deliberately deceptive. The report may contain an editing error, a scope distinction, or a substantive disagreement that needs to be made explicit.
A different row is marked Not Applicable. The buyer should ask why the criterion is outside the product's functionality rather than treat the label as either a pass or a failure. In our fictional case, a feature can become relevant when a customer enables a module that the original evaluation did not include. A correct earlier applicability statement may no longer cover the configured service.
The report's own terminology also matters. A custom phrase such as planned support could describe a future intention rather than current behavior. It should not be counted among established support statements merely because the word support appears in it. A future plan and an observed result belong to different evidence categories.
For Portal Q, the review notes should preserve the exact claimed state and the explanation needed to interpret it. The goal is not to turn every exception into a disqualifying alarm or every favorable label into a guarantee. It is to understand which functions the report says work, which limitations it acknowledges, and which statements cannot yet be reconciled.
4. A successful page is not necessarily a successful process
WCAG 2.2's conformance provisions concern full pages and complete processes, rather than allowing a claim to exclude an inconvenient part of a page or a required step. Its published Recommendation at the time of this review is dated December 12, 2024. This is a technical conformance boundary, not a statement that every legal framework requires that particular WCAG version. WCAG 2.2 conformance requirements
Our fictional appointment process has four steps: identify an available service, select a date, confirm the booking, and retrieve the required receipt. A demonstration successfully completes the first three. The receipt step then presents a barrier for the evaluated interaction method. Reporting the demonstration as a successfully completed appointment process would omit the final requirement built into our example.
A second test reaches the receipt by starting from a direct internal link supplied by a developer. That can help investigate the receipt page itself. It does not establish that a user can reach the same state through the ordinary public workflow. The entry point is part of the test conditions and should remain visible in the result.
The same issue appears with error recovery. A demonstration that enters correct information on the first attempt may never encounter the validation message or correction path. If the intended task includes recovering from a rejected entry, a report of only the successful straight-through path leaves that state unevaluated. This is a gap in the demonstrated coverage, not proof that the untested state fails.
To make our case concrete, define two fictional paths. Path A contains the four successful-input states. Path B contains those states plus a rejected-date message and a return to the calendar. Testing Path A alone does not test the two additional states in Path B. The reviewer should not count a shared page address as evidence that every interaction state was exercised.
A useful task record describes the starting state, actions, expected outcome, observed outcome, and the configuration used. In this fictional exercise, those details reveal why a three-step demonstration and a direct-link check do not establish completion of the full required public process.
The conclusion should match the observation: three steps completed under the stated conditions, with the required receipt path unresolved or blocked as described. That statement is less sweeping than the claim the portal is accessible, but it is far more useful for deciding what evidence or correction is needed next.
5. Ask what the evaluation method could actually observe
Section508.gov distinguishes automated, manual, and hybrid testing approaches. Its guidance emphasizes that automated tools have limitations and that human evaluation is needed for aspects they cannot determine. It does not support a universal claim that one scanner catches a fixed percentage of every product's accessibility problems. Section508.gov overview of testing methods
Our fictional report says automated scanning was performed on twenty pages. That establishes a stated method and sample, not the complete set of interactions evaluated. If the report does not describe keyboard operation, dynamic states, or other relevant observations, the reader should not invent those tests on the evaluator's behalf.
Suppose the scan produces thirty findings. Ten concern repeated instances of one shared navigation component, while twenty concern other elements. A development team might repair the shared component once and remove ten visible findings. That is a useful result, but thirty findings were not necessarily thirty independent underlying defects.
The opposite counting issue also occurs. A single reported defect in the shared navigation can affect many pages and tasks. A summary that counts it as one minor item solely because it occupies one row would ignore its reach. The record should distinguish the number of observations, the underlying issue, and the affected scope.
Now imagine that a second scanning run produces zero findings after the shared issue and the other detectable issues are addressed. The result supports the statement that the configured checks reported no findings on the tested material during that run. It does not establish that every criterion, state, or interaction method was evaluated.
A stronger evidence packet names the tool and version, the rules enabled, the sample, the run date, and the treatment of results requiring human review. These details allow a later reviewer to understand the observation. They do not make the tool capable of judging questions outside its design.
For Portal Q, the method review identifies a concrete gap: the reported scan does not show whether the required receipt workflow was operated using the interaction methods relevant to the intended users. That gap can be resolved with additional evidence. It should not be papered over by a scanner score or by a generic statement that the product was tested.
The distinction is equally important when a report describes expert review. The phrase expert tested does not identify the scope or findings by itself. A human evaluator can provide valuable judgment, but the public summary still needs enough method and result information to support the claim being made.
6. A sample needs a reason, not just a size
W3C published WCAG-EM 2.0 as a Group Note on July 23, 2026. It describes a methodology for evaluating digital products through scope definition, exploration, sampling, evaluation, and reporting, and explicitly says it does not add to or replace WCAG's requirements. It should not be confused with a new WCAG conformance level. WCAG Evaluation Methodology 2.0, dated Group Note
Invent a Portal Q deployment with sixty named screens. Fifty are variations on a shared information-page layout. Ten contain distinct functions such as appointment selection, account changes, or document generation. A sample of ten information pages examines ten screens but none of the ten distinct functional screens in this deliberately constructed inventory.
A second sample also contains ten screens, but includes two information-page variations and eight distinct functional screens. The sample size is identical; the represented functionality is different. The example does not prove the second sample is sufficient. It demonstrates why size alone cannot establish coverage.
Add interaction states. One selected screen has an initial view, an error view, and a confirmation view. Listing its URL once does not tell us which states were examined. If the test log shows only the initial view, the other two remain unobserved in the supplied evidence even though they share the same address.
For a reproducible review, distinguish the inventory from the selected sample and the selected sample from the states actually exercised. Those three sets can differ. A planned test is not an executed test, and an executed test without a recorded result cannot support the same conclusion as a documented observation.
Suppose the plan selected twelve screen-state combinations, but two could not be reached because the test account lacked a required role. Ten combinations were evaluated. The correct result is ten completed evaluations and two access-limited cases, not twelve passes or two accessibility failures. The obstacle to testing is itself a relevant limitation, but it should not be mislabeled as a product finding.
The sample also needs to match the intended configuration. If a buyer enables a specialized module, evidence from a deployment without it leaves a scope question. It may be possible to reuse unchanged observations, but the reuse should be justified by the components and conditions, not by a desire to avoid revisiting the report.
For our fictional product, the sample review now provides specific information: which screen families appeared, which functions were represented, which states were exercised, and which planned checks could not be completed. That information is much more useful than the bare assertion that ten pages were tested thoroughly.
7. User observations add evidence without becoming universal claims
W3C WAI recommends combining evaluation with disabled users and standards-based conformance assessment. It cautions that a small number of participants cannot represent everyone with similar or different disabilities. The two forms of evaluation contribute different evidence rather than replacing each other. W3C WAI, Involving Users in Evaluating Web Accessibility
Consider a wholly fictional exercise involving eight participants and one appointment task. Five complete it without assistance, two complete it only after the facilitator supplies a workaround, and one cannot complete it. A summary saying seven of eight completed the task would conceal the assistance distinction. A summary saying only five completed would omit the two assisted completions unless its definition is explained.
Neither summary estimates the experience of all disabled users. We have supplied no sampling design that would justify that generalization, and the participants' relevant characteristics and tools have not yet been described. The useful evidence is the observed interaction, the obstacle, and the conditions of the exercise.
Suppose both assisted participants encountered the same calendar limitation. That observation can motivate a focused investigation. It does not establish that every person using the same assistive technology will encounter the same result, or that people using other methods will not. The report should preserve what was observed without extending it beyond the study.
The facilitator's intervention matters too. If the workaround requires instructions unavailable to ordinary users, the assisted completion should not be presented as proof that the public interface independently supports the task. On the other hand, recording the workaround can help the team understand what information or functionality is missing.
A different participant might finish the task while experiencing an unnecessary burden, such as repeated backtracking or an unclear confirmation. Task completion is one outcome, not the entire experience. Our fictional report can describe the burden without inventing a formal conformance failure that has not been evaluated against a specific criterion.
No such participant exercise was conducted for this article. The scenario illustrates how to read one if supplied. A real report should identify its actual scope and protect participant privacy while preserving enough context to interpret the observations. Personal stories should not be generalized into a universal success claim or dismissed merely because they do not constitute a statistically representative study.
For Portal Q, user evidence would complement the scope and test records by showing how the task works in practice for the people involved. It would not turn an ACR into a guarantee that every future user, configuration, or task is covered.
8. Machine-readable reports make consistency checks easier, not conformance automatic
Section508.gov's ACR library provides reports in readable HTML and OpenACR YAML formats. It describes OpenACR as a machine-readable schema intended to help accessibility experts create and share reports. Structured data can make a report easier to process, but a valid file format does not establish the truth of every reported result. Section508.gov Accessibility Conformance Report Library
To demonstrate that distinction, I created an original seven-row JSON fixture and actually ran a small Python consistency checker against it. The fixture is not a VPAT template, is not OpenACR, and uses invented criterion letters rather than real standards identifiers. Portal Q, its components, its statuses, and its evidence identifiers are all fictional.
Each row contains a product, version, component, criterion letter, status, and evidence reference. The checker groups rows by the first four fields. It then looks for repeated rows within a group, conflicting status labels within the same group, and references to evidence identifiers absent from the fixture's evidence dictionary.
The initial file contains two identical search entries for version 4.2 and Criterion A. It contains a calendar entry with a partial-support label. Two dialog entries share version 4.2 and Criterion C but carry conflicting support labels. A media entry has a not-applicable label. A login entry points to evidence identifier E999, which does not exist in the dictionary.
The observed output contains seven rows and five distinct grouping keys. It identifies one duplicate group, one conflicting-status group, and one missing evidence reference. These are actual results from the local script, not invented claims that a real product failed accessibility testing.
The duplicate is a document-consistency issue. The conflicting labels require investigation of the report's scope or editing history. The missing reference prevents the reader from locating the claimed supporting record within this toy packet. None of those findings tells us whether a person can use the fictional interface, because the script never operates an interface.
That boundary is deliberately built into the output. It labels accessibility conformance and evidence relevance as not evaluated. A tool can be correct about the narrow checks it performed while providing no answer to the broader question a buyer ultimately cares about.
The exercise is useful because these layers are easy to confuse in practice. A spreadsheet that parses successfully may still contain contradictory claims. A report that has no duplicate rows may still describe the wrong product version. A reference that exists may still point to irrelevant evidence. Machine readability helps expose some problems, but it does not erase the need to interpret the records.
9. A clean structural result can conceal an irrelevant evidence link
I then created a second, mechanically altered fixture and ran the same checker. One duplicate search row was removed. The earlier dialog claim was assigned version 4.1, separating it from the version 4.2 claim. The login row's nonexistent E999 reference was replaced with E01, an identifier that does exist.
The observed second result contains six rows and six distinct grouping keys, with no duplicate group, conflicting-status group, or missing reference under the script's rules. Its structural check passes. But the example is deliberately not a repaired or approved accessibility report.
Why not? E01's fictional description concerns search, not login. The checker verifies existence, not relevance. Changing the login pointer makes the file structurally cleaner without supplying evidence that supports the login claim. A human reader can see the mismatch from the descriptions, while this particular script does not attempt that judgment.
The dialog version change also needs justification in a real record. Our fixture simply stipulates the altered metadata for the experiment. In an actual report, an editor would need evidence that the two observations concern different versions before changing the field to remove a conflict. Otherwise the adjustment would hide disagreement rather than resolve it.
The comparison therefore demonstrates two different propositions. The initial file has detectable structural problems. The altered file passes those limited checks but still lacks a justified relationship between at least one claim and its cited evidence. Passing the checker is not equivalent to passing an accessibility evaluation.
A third actually executed control removes the login row entirely from the mechanically altered fixture. It produces five rows and five distinct keys, again with a clean structural result. The checker has no separate inventory of required components, so it cannot detect that login coverage has disappeared. A shorter, cleaner file can therefore contain less evidence. This control is not a suggestion to delete inconvenient findings; it demonstrates why a consistency check needs to be interpreted alongside a separate scope inventory. Otherwise an omitted claim can become less visible precisely because no contradictory row remains to flag.
All three fixtures, the checker, and its recorded output were retained for reproducibility. The initial fixture's SHA-256 digest begins a610e476a11c73d0; the mechanically altered fixture's digest begins 0358e70b9e81ca53. These prefixes distinguish the two saved inputs in the accompanying exercise packet; they are not signatures, certifications, or proof that the fictional claims are true.
This gives a buyer a useful way to frame automated report review. A consistency tool can flag matters to investigate and reduce clerical mistakes. Its output should describe the properties checked, not convert a clean record structure into a broad product endorsement.
For Portal Q, the right next action after the clean structural result is not to stamp the report approved. It is to obtain relevant login evidence and establish the dialog version history. Those are specific unresolved questions exposed by the example, not reasons to reject structured reporting altogether.
10. Connect the summary report to a reproducible finding
Section508.gov distinguishes an ACR's overview from a more detailed accessibility test report intended to help product teams address issues. Its guidance identifies information such as report and product versions, testing details, and findings. The summary and the detailed report serve related but different purposes. Section508.gov, Essential Elements of an Accessibility Test Report
Give Portal Q a fictional test record T-17. It identifies release 4.2, the appointment module, a particular browser and interaction method, the starting state, the attempted action, the expected outcome, and the observed failure. An ACR row references T-17 in explaining a limitation. That link is much more useful than an unqualified statement that testing occurred.
Now suppose a proposed fix changes the calendar. A new test record T-18 repeats the previously failing action under the same stated conditions and observes the expected outcome. That is evidence of a specific retest in our fictional case. It does not automatically establish that every other calendar behavior, interaction method, or configured deployment was re-evaluated.
The report revision should explain what changed. If T-18 addresses only the month selector, a broad edit removing every calendar exception would need additional support. If the original failure concerned the date grid, a passing month-selector test would not resolve that original issue at all. Matching the test to the finding matters more than whether the new record says pass.
A record can also fail to reproduce an earlier issue for reasons unrelated to a fix. The test account may have a different role, the application may display another interface, or the evaluator may start from a different state. A meaningful comparison checks those conditions before concluding that the original observation was wrong.
For our fictional packet, keep T-17 and T-18 as separate dated records and describe their relationship. Replacing T-17 with the passing result would remove the history needed to understand the limitation and its resolution. Preserving history does not require presenting an obsolete defect as current; the status can show that the specific issue was addressed under specified conditions.
This creates a disciplined bridge from report to finding to correction to retest. The bridge is not a promise that the product will never regress. It gives a reader enough evidence to understand which claim changed and why, instead of treating the report's latest date as the only meaningful signal.
11. Compare intended outcomes, not the percentage of favorable rows
Imagine two fictional reports prepared against the same invented set of one hundred review items. Product A marks ninety supported and ten unsupported. Product B marks eighty supported and twenty unsupported. A simple count favors A. It does not yet tell the buyer which product permits the required appointment-and-receipt task to be completed.
In the exercise, one of A's unsupported items prevents access to the only receipt route required by the buyer's workflow. B's unsupported items concern functions outside that particular workflow. That difference matters to the intended task, but it does not establish that B is generally accessible or that a legal purchasing exception automatically applies. It identifies a question the raw row counts concealed.
Reverse the intended use. A second customer needs one of B's unsupported functions every day and does not use the receipt feature. The same two reports now have different practical implications. A universal ranking based only on ninety versus eighty would conceal both customers' actual needs.
The lesson is not that a buyer can ignore applicable requirements by assigning convenient weights. It is that a report-level percentage cannot replace the identified requirements and workflow. Any procurement decision must still follow the applicable process, and this fictional comparison does not authorize one.
The denominator can be misleading as well. One report may include more standards, modules, or explicitly not-applicable items than another. If Product A has eighty favorable rows out of one hundred and Product B forty out of fifty, both display eighty percent under that arithmetic. Without matching the row definitions, the equality says little about their relative support.
A useful comparison instead identifies the relevant product versions, scope, criteria, and known limitations side by side. Where the reports differ in coverage, mark the mismatch rather than fill it with a guessed score. Where a required function has no adequate evidence, identify the missing demonstration or test record.
For Portal Q, the comparison result is therefore not a trophy for having the greenest table. It is a set of substantiated capabilities, known limitations affecting the intended task, and specific unresolved evidence requests. That is the information a responsible decision needs before price, implementation effort, or a planned future fix is considered alongside accessibility.
12. A deployment matrix shows what the report has not established
Finish with an original configuration example. A buyer intends to support two browsers, two assistive-technology configurations, and two user roles. Under this simplified design, there are eight combinations. A report describes observations for four of them. That is four documented combinations, not proof that the other four fail and not proof that they behave identically.
The exact four matter. Testing both browsers with only one assistive-technology configuration leaves the other configuration unobserved. Testing both roles with only one browser leaves another gap. A count of four out of eight cannot describe the pattern without the combination list.
Suppose a shared component is unchanged across the eight combinations. That fact may inform a test strategy, but the reader still needs the evaluator's justification for any evidence reused across configurations. Our arithmetic does not determine which combinations require separate evaluation. It simply exposes the difference between a complete list of intended configurations and the observations actually supplied.
Now add a customer-specific theme that changes the calendar and receipt presentation. The earlier four observations describe the previous configuration unless evidence establishes otherwise. The buyer should preserve that boundary in the rollout decision and in any public description of the report. An unchanged subscription name is not evidence that the interface remained unchanged.
For a final Portal Q handoff, the evidence packet can be concise but specific. It identifies the evaluated release and configuration, the required user process, the ACR's scope, the detailed findings linked to important exceptions, the recorded retests, and the unresolved deployment combinations. The mechanically clean JSON result remains in a separate category: a document-consistency observation, not product conformance.
This packet supports several clear conclusions. The original report covers two named modules. The required receipt route needs additional evidence. One fictional calendar finding has a bounded retest record rather than a universal clearance. The intended deployment includes combinations the supplied observations do not establish. The local consistency checker found and then ceased to flag particular structural issues, while leaving evidence relevance untested.
None of those findings is improved by replacing it with a vague pass or fail badge. A good ACR helps people understand a product's accessibility, including its limitations. Reading it well means preserving the relationship between the version, the scope, the claim, and the observation. That relationship is what makes the report useful after the sales conversation ends and someone actually needs to complete the task.