A rising line on an AI-incident dashboard can mean that more harmful events occurred. It can also mean that more events became visible, an editor changed the inclusion rule, a backlog was processed, or a database split several records that had previously been grouped together. Some combinations of those changes can happen at once. The graph does not identify its own explanation.
Definitions are not settled simply because two databases use the same word. The OECD's May 2024 paper distinguishes actual harm, described as an AI incident, from potential danger, described as an AI hazard. Its purpose includes interoperability while allowing jurisdictions flexibility over scope. Those definitions should not be confused with a universal legal reporting duty. OECD, Defining AI incidents and related terms
The AI Incident Database's editorial guide, Version 2 dated October 1, 2026, includes alleged harm or near-harm events in which AI is implicated. It distinguishes source reports from incident records and cautions that database inclusion does not independently adjudicate every allegation. The version and date matter: this is an editorial standard, not a court's finding about every listed event. AIID Editorial and Research Guide, Version 2
This article uses those differences as a starting point for an original analytical casebook. It does not reproduce either organization's operating manual or classify any named company as unsafe. Every event, organization, dashboard, reviewer, and numerical dataset introduced below is fictional unless explicitly identified as a primary-source fact. None of the invented entries was submitted to an actual database.
The goal is to make a public dashboard answerable to its records. A useful count should say what entered the numerator, which evidence supports the classification, how records changed, and what remains unknown. A useful response record should also show whether the reported problem was addressed, rather than treating a larger archive as proof that the underlying system became safer.
1. Preserve the disagreement instead of hiding it inside one label
Create a fictional review file containing thirty candidate events. For this exercise, twelve have documentary support for a realized adverse consequence and an AI contribution. Eight describe a specifically documented danger that was interrupted before the consequence occurred. Six describe an adverse consequence but leave AI's role unresolved. Four have been shown, within our invented evidence, not to involve an AI contribution.
These categories are definitions for this exercise, not a ruling on how any real database must classify an event. A narrow dashboard could display twelve supported realized-harm events. A broader dashboard could display twenty supported realized-harm or interrupted-danger events. A preliminary investigation queue could contain twenty-six candidates after excluding the four resolved non-AI cases.
All three totals can be arithmetically correct. They are not three estimates of precisely the same quantity. A comparison that reports twelve versus twenty-six without explaining the scope difference would invite readers to infer disagreement about the number of actual harms when part of the difference is a decision about what to include.
The six unresolved cases deserve an explicit status. Treating all six as established AI-caused harms would outrun the evidence. Treating them as disproven would also outrun it. A dashboard can display both the count of supported cases and the count awaiting a particular factual determination without forcing uncertainty into a yes-or-no answer.
Now imagine that new evidence resolves two of those six as AI-related realized harms and one as unrelated. The categories become fourteen supported harms, eight interrupted dangers, three unresolved cases, and five unrelated cases. The total candidate file remains thirty. The narrower dashboard grows from twelve to fourteen even though the events themselves are not new; the evidence changed.
That distinction is useful to readers trying to understand a sudden update. In our example, the increase is two newly supported classifications, not two newly occurring events. The published explanation should identify the transition rather than let a counter imply a chronology that did not happen.
A definition crosswalk can therefore be more informative than a ranking. It preserves the categories each source actually uses and shows how a proposed comparison handles them. If a category cannot be mapped without losing meaning, mark it as not directly comparable. Interoperability should make differences visible, not erase them merely to produce one convenient total.
2. Count documents, events, and independent evidence separately
A fictional archive contains fourteen source documents. Nine concern Event A, four concern Event B, and one concerns Event C. The archive therefore contains fourteen documents associated with three event records under the case file's stipulated boundaries. Fourteen is not the event count.
Within Event A's nine documents, seven republish substantially the same original account. Two are separate original accounts. The seven copies can show how widely a claim circulated, but they do not supply seven independent observations of what happened. Even the two original accounts need to be examined for whether they relied on the same witness or underlying document.
To make that dependence explicit, suppose the two accounts both cite a single inspection memo. The number of publishers is now different from the number of identified underlying evidentiary sources. A headline that says nine independent reports confirmed the event would be unsupported by the fictional file.
Event B presents the opposite difficulty. Its four documents describe two affected workplaces during one software update. Whether the dashboard counts one update-related episode or two site-level events depends on the chosen unit. The record should preserve both the common update and the distinct workplace effects so that an analyst does not have to guess which dimension the total represents.
For our demonstration dashboard, keep one parent episode with two linked site records. Display the number of parent episodes and the number of affected sites in separate fields. This is an author-designed representation for the exercise, not a claim that any external database uses that structure. Its advantage is that the counting choice remains inspectable.
A third distinction concerns people affected. A single episode can affect several people, and the same person can appear in more than one episode. Adding all reported affected-person counts can therefore overstate unique people if overlaps are not resolved. It can also understate them when records report only a subset. The absence of identifying information may make an exact deduplication impossible, and publishing sensitive identities is not an acceptable shortcut.
For this case file, the honest dashboard shows what it can establish: document count, parent-episode count, site count where known, and reported impact with an explicit overlap limitation. That is a richer account than one enormous number labeled incidents, especially when the number combines objects that answer different questions.
3. Separate a failed output from the consequence attributed to it
Imagine a fictional inventory assistant that recommends ordering 500 replacement components. A later complaint says the recommendation caused a 5,000-dollar loss. The complaint is evidence of an allegation. To evaluate the stated causal chain in this invented case, we supply several additional records rather than declare the output sufficient proof.
The input snapshot contains an erroneous stock balance. The model output recommends the order. An operator changes the quantity to 300. A purchasing system applies a separate minimum-order rule and issues an order for 400. A credit memo later refunds half the purchase price. Each record describes a different part of the chain, and the final financial effect cannot be read directly from the first output.
Suppose the fictional price is ten dollars per component. The 400-component purchase costs 4,000 dollars. A 2,000-dollar refund leaves 2,000 dollars of net cash outflow, before any value assigned to retained components or other costs. That calculation does not establish the total economic harm, because the case file has not yet specified whether the remaining components were useful, wasted, or recoverable.
It also does not establish that the model alone caused the result. The inaccurate input, human modification, purchasing rule, and eventual remediation are all visible in our stipulated chain. An account can describe the model's role without deleting those other steps or assigning legal responsibility that the records have not established.
A useful additional test would compare the recorded recommendation with the actual action. If the operator had rejected the recommendation and placed no order, the output would not establish the same realized purchasing loss. If the order had been initiated independently before the recommendation, the chronology would undermine a simple claim that the recommendation triggered it.
These are evidentiary distinctions, not excuses for a defective system. They help identify what would need to change: the input check, the recommendation, the authority to act, the purchasing rule, or a combination. A dashboard that stores only the model name and a dollar figure can obscure exactly the part of the process an investigation needs to understand.
The public-facing result in this fictional case is therefore a bounded account of the sequence and the unresolved valuation. It is not an unsupported declaration that the entire initial allegation was confirmed, nor a dismissal because a person also participated. The records support a more specific explanation than either slogan.
4. Keep occurrence, discovery, and publication on separate clocks
A fictional dashboard adds one hundred records during January. Thirty concern events that occurred in earlier months, while seventy concern January events. An additional fifteen January events are discovered and added in February. For the completed observation window we have invented, January additions total one hundred but January occurrences total eighty-five.
The difference is not an arithmetic error. The first series groups records by when they entered the dashboard; the second groups events by when they happened. The late discoveries change the second series retrospectively. They do not change the number of records that were actually added during January.
Now suppose a chart freezes January occurrences on January 31, before the fifteen later discoveries. Comparing that incomplete January value with a mature earlier month can create an apparent improvement that disappears after the backlog arrives. The chart needs either a consistent observation lag or a clear warning that the latest period is incomplete.
The relevant dates can multiply. A person may experience an event, recognize a consequence later, notify an organization later still, and then see a public report. A database may add the report after another delay. A date field called incident date is not self-explanatory unless the documentation says which of those events it represents.
For our fictional records, preserve three dates at minimum where the evidence supports them: the event date, first documented awareness, and public addition date. If the event date is known only to the month, do not invent a day merely because a charting library expects one. A fabricated first-of-month date can distort apparent clusters and reporting-delay calculations.
The same caution applies to ongoing events. A problem spanning several weeks may have a beginning, a discovery, a mitigation, and a later confirmation of recovery. Counting it once per day would answer a different question from counting the episode once. A dashboard can report duration without multiplying episodes invisibly.
For the January example, a useful update says that fifteen later-discovered January events were added to the occurrence series. Readers can then distinguish historical revision from a new burst of events. The chart becomes an account of both what happened and when the evidence became available.
5. Exposure and visibility are separate missing quantities
NHTSA's standing-general-order crash-reporting page supplies a useful sector-specific warning. It says differences in telemetry and awareness affect reporting, cautions against assuming the data represent all crashes statistically, and explains that the reported counts are not normalized by exposure such as miles traveled. This is not an AI-wide reporting rule, but it illustrates why an incident count cannot supply its own denominator. NHTSA's crash-data interpretation limitations
Consider two fictional services with a common, deliberately fixed event definition. Service A has twenty recorded events across one million relevant operations. Service B has five across ten thousand operations. The recorded rates are two per hundred thousand operations for A and fifty per hundred thousand for B. B has fewer recorded events but a recorded rate twenty-five times higher.
That arithmetic does not yet establish that B is twenty-five times more dangerous. We have not established equal detection, equal exposure quality, or comparable circumstances. An operation could mean something different across services unless the denominator has been defined carefully. A low-stakes recommendation and a consequential automated transaction should not be treated as equivalent merely because each produced one software request.
To isolate visibility, invent two further worlds for Service A. In World One, all twenty relevant events were detected. In World Two, there were one hundred events but only twenty reached the reporting system. The public count is identical. Without evidence about missed events, the dashboard cannot choose between those worlds.
Now hold the underlying event total fixed at one hundred across two periods. A reporting improvement captures twenty in the first period and sixty in the second. The observed count triples, a two-hundred-percent increase, even though the underlying event total is unchanged by construction. This is a counterexample to the claim that a larger reported count necessarily proves a larger underlying total.
The opposite inference also fails. A falling reported count does not establish improvement if access to reporting deteriorated. Neither possibility should be asserted about a real organization without evidence. The examples identify information that a strong comparison needs, not a convenient explanation to select whenever a trend is politically unwelcome.
For our fictional dashboard, display recorded events and the stated exposure separately, then mark the detection process as known, partially described, or unknown. A rate can be calculated when its inputs are available, but its label should retain the word recorded. Readers should not have to discover in a footnote that a chart of supposed real-world risk actually measures only the cases that reached an archive.
6. A collection pipeline can create a break in the series
The OECD monitor's current methodology describes machine-assisted classification of news-derived events, a four-day window for adding related coverage, and a methodological change beginning in November 2024. It explicitly warns that later articles can be misclassified as separate events. Those documented features matter when interpreting a trend; the page does not make every collected event an independently adjudicated fact. OECD AI Incidents and Hazards Monitor methodology
Use a different fictional archive to examine what a collection change can do. During an initial period, its collector reads ten local publications. During a later period, it reads thirty, including twenty that had never been monitored. The number of captured stories rises from fifty to one hundred. That observation alone cannot separate a change in underlying events from the expanded coverage.
A useful comparison could retain the original ten-publication panel and count its later stories separately. Suppose those original sources produce sixty later stories, while the added sources produce forty. The total increase is fifty stories: ten from the old panel and forty from the added panel. The bridge makes one part of the measurement change visible without assuming that even the old panel's increase equals a change in event incidence.
Why not? The old panel can change its own coverage intensity. Editors may assign more reporters to AI, publish more follow-ups, or cover events in different locations. A stable list of outlets is not automatically a stable observation process. The panel comparison is useful, but it solves only the particular problem it was designed to address.
Now imagine that the collector changes its duplicate rule. Five records previously displayed as separate are merged into two episodes. The total falls by three without any disappearance of the underlying reports. A revision log should identify the merge, retain discoverable links, and distinguish it from the withdrawal of a factual allegation. These are different editorial actions with different meanings for readers.
The consequences can be tested on a frozen snapshot. Recompute the old records under the new grouping rule, or state that such a back-series is unavailable. Comparing an old-method total directly with a new-method total may still be informative if clearly labeled, but it cannot support a clean growth claim that assumes the counting procedure stayed unchanged.
For our casebook, the dashboard's methodological history becomes part of the data. The date a source was added, a field redefined, or a duplicate rule changed can explain a discontinuity that would otherwise look like a sudden change in technology. That explanation must come from the actual pipeline history rather than from speculation about what the curve probably means.
7. Two reviewers can disagree without either being dishonest
The CSET taxonomy page for AIID explains that its harm definition does not encompass every AIID incident. It also describes independent classification by two annotators followed by review and quality control, while acknowledging possible mistakes. A taxonomy layer and the underlying archive therefore need not produce identical totals. CSET AI Harm Taxonomy overview
Create an original review exercise with thirty records and a single author-defined yes-or-no criterion. Reviewer A marks twenty yes; Reviewer B marks eighteen yes. Fourteen records receive yes from both. That leaves six yes only from A and four yes only from B. The union of their yes labels is twenty-four, leaving six records marked no by both.
They agree on twenty of thirty records: fourteen shared yes labels and six shared no labels. Their raw agreement is approximately 66.67 percent. That figure is not their accuracy. We have supplied no independently established truth against which to score either reviewer, and agreement alone cannot create one.
The discrepancy is also more informative than the two marginal totals. Saying twenty versus eighteen might suggest a disagreement about only two records. In fact, the reviewers disagree on ten. The small difference between the totals conceals a much larger difference in which records they included.
An adjudicator then examines those ten disputed cases. Under the exercise's stated rule and evidence, three of A's six unique yes labels and two of B's four unique yes labels are retained. The final yes set contains nineteen records: fourteen originally shared plus five resolved yes decisions. That result is not obtained by averaging twenty and eighteen, even though the numerical average happens to equal nineteen here.
To see why the procedure matters, change the adjudicated outcomes while leaving the original reviewer totals alone. If only two disputed cases qualify, the final total is sixteen. If eight qualify, it is twenty-two. The original totals do not determine the resolution because the decision depends on the evidence in the disputed cases.
For a public dashboard, the most useful record of this process is not a theatrical contest over which reviewer won. It is the criterion applied, the evidence that resolved each disagreement, and the version of the resulting classification. If an unresolved disagreement remains material, the database can show that status rather than force a falsely unanimous answer.
This analysis also protects against a misleading quality claim. Double review is a process feature, not automatic proof of correct labels. It can expose ambiguity and improve consistency, but the published result still needs to be traceable to its supporting evidence. In our fictional exercise, no amount of agreement statistics substitutes for examining the ten disputed records.
8. A completed field can still contain no usable evidence
The OECD's February 2025 common-reporting-framework paper proposes twenty-nine criteria intended to support interoperability across jurisdictions and sectors while allowing local adaptation. A shared structure can make reports easier to compare; it does not establish that every completed field contains verified information. OECD, Towards a common reporting framework for AI incidents
Our fictional dashboard contains one hundred records with a loss-amount field. Sixty contain a numeric value supported by the record's stated evidence. Twenty say the amount is unknown. Ten say it is withheld. Ten say the field is not applicable under the exercise's definition. Every field is filled, but only sixty contain a usable number.
If the analysis concerns the ninety records for which a loss amount could apply, numeric availability is sixty out of ninety, approximately 66.67 percent. If it concerns all one hundred records, the corresponding proportion is sixty percent. A dashboard claiming one-hundred-percent completeness simply because no cell is empty would answer a software-validation question, not the substantive evidence question.
The categories must remain distinct. Unknown does not mean zero. Withheld does not mean unreported to every authority. Not applicable does not mean a missing value should be imputed. Collapsing all three into zero would introduce forty unsupported numeric replacements, one for each record lacking a numeric amount.
Suppose the sixty supported values sum to 120,000 dollars. Dividing by sixty yields a mean of 2,000 dollars among those records. Dividing by all one hundred yields 1,200 dollars only if the denominator is explicitly all records and the numerator remains the known subtotal; it is not an estimate of the average actual loss across the complete file without additional assumptions.
A careful display can state the known subtotal and the number of records contributing to it, with the unresolved categories alongside. It need not invent values to be useful. A reader can see both the documented amount and why the total impact is not fully established.
Privacy introduces a legitimate reason to limit public detail. If a small group could be identified from a combination of fields, publishing more granular information may create a new risk. That does not require abandoning accountability. An analyst can document the existence and purpose of a restriction, publish safe aggregates where justified, and explain which conclusions the withheld detail prevents outsiders from independently checking.
The design challenge is to make absence interpretable. A field should tell readers whether evidence was unavailable, outside scope, deliberately protected, or contradicted. Treating every non-numeric state as a clerical failure can pressure a reporting system to produce confident-looking data that its sources do not support.
9. Severity is not a synonym for the number of entries
Invent two collections. Collection A contains one episode that temporarily blocks access to a thousand customer accounts. Collection B contains ten episodes, each permanently destroying one unique archival item. The event counts are one and ten. The affected-account and destroyed-item counts are a thousand and ten. None of those totals, alone, supplies a defensible overall ranking of severity.
The consequences differ in kind, duration, and reversibility. We have not specified what customers lost during the temporary interruption, whether alternatives were available, or the value and recoverability of the archival items. Assigning one universal score without those facts would conceal the judgments doing the work.
A simple monetary example makes another distinction visible. Suppose one fictional episode causes ten documented losses of one hundred dollars each, while another causes one documented loss of one thousand dollars. The known dollar subtotal is the same. The distribution across people is different. A reader may reasonably care about both, and the dashboard should not force one to stand in for the other.
Even identical dollar subtotals need compatible valuation. An initial claimed loss, an independently documented payment, a refunded amount, and a remaining unrecovered balance describe different stages. Adding all four for the same transaction would multiply the loss by counting several descriptions of it. Subtracting every announced refund before evidence of payment would assume a remedy had occurred.
For an original mini-ledger, let an affected customer document a 1,000-dollar payment. The organization announces a full refund, but the available receipt shows only 600 dollars paid so far. The recorded remaining cash loss is 400 dollars under those stipulated facts, with a further 400-dollar promised payment pending. The announcement does not close that difference.
This example also shows why reversibility should be a separate field rather than an optimistic adjective. A promised reversal and a completed reversal are not the same state. Some consequences may persist after money is returned; our toy ledger does not attempt to value them. A financial reconciliation should not be presented as a complete assessment of every harm the event may have caused.
For a useful severity display, preserve the observable dimensions the records actually support. The dashboard might report documented financial amounts, duration where known, affected populations with overlap limits, and remediation status. It can also state when consequences are qualitatively different and cannot responsibly be collapsed into one ranking. Refusing an unsupported single score is not refusing to measure; it is keeping the measurements faithful to their units.
10. Closing tasks is not the same as resolving incidents
NIST describes its AI Risk Management Framework as voluntary and identifies the January 2023 release of version 1.0. Its current page also says that version is being revised. A reference to the framework should therefore identify the version and should not be presented as a certificate that an organization has eliminated risk or satisfied every applicable legal duty. NIST AI Risk Management Framework overview
Consider an original response ledger with twenty-five fictional incidents. Each incident has exactly two required tasks under our invented response plan: one corrective action and one affected-party follow-up. There are fifty tasks. At the reporting date, thirty tasks are complete. The completion rate is sixty percent at the task level.
How many incidents are fully resolved under this plan's two-task rule? The total alone does not say. If thirty completed tasks are paired across fifteen incidents, fifteen are complete and ten have neither task complete. Alternatively, five incidents can have both tasks complete while each of the other twenty has one complete. That arrangement also gives thirty completed tasks, but only five complete incidents.
Five and fifteen are the minimum and maximum possible completed-incident counts under these exact assumptions. To reach fewer than five, the twenty-five incidents could accommodate at most twenty-five first completed tasks plus fewer than five second tasks, which would not reach thirty. To exceed fifteen would require more than thirty completed tasks because each completed incident needs two.
The task dashboard therefore supports a range of twenty to sixty percent fully completed incidents until the links between tasks and incidents are inspected. It does not justify reporting sixty-percent incident resolution merely because sixty percent of tasks are done. The mismatch is a property of the invented record structure, not a criticism of any real response program.
Now add a quality condition. A task marked complete may mean a patch was prepared, not that it was deployed to every affected installation. Our example's response plan must say what evidence closes the task. Otherwise the same status can represent a proposed fix in one record and verified deployment in another.
The affected-party task can have its own ambiguity. Sending a notice is not the same as confirming that the person regained access or received compensation. Depending on the stated purpose, each may be a legitimate milestone. The dashboard should name the milestone it actually measures rather than collapse communication, correction, and recovery into a single reassuring word.
For our twenty-five incidents, the immediate analytical step is to examine the task-to-incident links and completion evidence. That is a specific, answerable follow-up. It is more useful than celebrating thirty closed tasks or dismissing the entire response effort because some cases remain open.
11. A successful retest has a boundary
Take a fictional incident involving an automated scheduling assistant. The identified defect occurs when a cancellation arrives after a booking update but before a confirmation message. The response team creates three test sequences that reproduce the documented timing pattern, then modifies the system and observes all three pass. These are invented test results, not tests we ran against a real product.
The result supports a narrow statement: the revised system passed those three specified sequences under the recorded test conditions. It does not establish that every cancellation sequence is safe, that every deployment runs the revision, or that previously affected users have been made whole.
Add five fictional installations. Deployment records confirm the revision at two, show the old version at two, and leave the fifth unverified. Calling the defect fixed everywhere would contradict the inventory. Calling nothing fixed would also discard the two documented deployments. The operational picture is partial: two verified revised installations, two old, and one unknown.
A dashboard could record separate statuses for the proposed correction, test result, deployment coverage, and affected-user remedy. Those fields answer different questions. They also identify the next evidence needed without demanding that a single status word carry every dimension of the response.
There is a further distinction between a defect recurring and the same report being rediscovered. Suppose a later document describes the original scheduling failure but is added to the archive after the fix. If it concerns the same original event, it is not evidence of a post-fix recurrence. The relevant dates and event links must be checked before reopening the technical conclusion.
Conversely, a genuinely new event after deployment cannot be dismissed simply because the earlier three tests passed. It may expose a new condition, an incomplete rollout, or a mistaken original diagnosis. Our exercise does not select among those explanations. It shows why the new record should be compared with the exact scope of the previous test rather than with the broad public phrase problem solved.
A credible response history is therefore cumulative. It preserves what was tested, where the correction was verified, which cases were remedied, and what later evidence changed. That history lets readers recognize real progress without treating a limited test as a universal guarantee.
12. Publish a change account alongside the total
Finish with a fictional dashboard that begins the month with thirty-five accepted event records. During the month it receives twelve new candidate records. Seven are accepted, three remain unresolved, and two are excluded under its stated criteria. Separately, editors merge three existing duplicate records into other existing records and reclassify two previously accepted entries as outside scope.
The accepted total at month's end is thirty-seven: start with thirty-five, add seven, subtract three duplicates, and subtract two reclassifications. The net increase of two does not mean that only two new candidate events were reported. It also does not mean that twelve new accepted events occurred. The change account exposes the different operations hidden inside the headline total.
The seven newly accepted records can themselves concern different dates. Suppose four concern older events and three concern events occurring this month. The additions series, accepted-record series, and occurrence series now diverge in understandable ways. A chart should say which series it displays and whether historical values were updated.
Preserve the excluded and merged identifiers in a way that lets readers understand corrections without unnecessarily repeating sensitive allegations. A correction should not become a silent disappearance that leaves earlier citations impossible to interpret. At the same time, retaining provenance is not a reason to keep prominently presenting a disproven allegation as if it remained established.
Our fictional dashboard can now make several defensible statements. It has thirty-seven accepted records under a dated rule. Seven records were newly accepted, of which three concern the current month. Three candidates remain unresolved. Five prior records left the accepted total through deduplication or reclassification, with those reasons distinguished. None of these statements claims to count every harmful AI event in the world.
That final limitation does not make the dashboard worthless. The records can identify recurring failure patterns, expose unanswered questions, support investigation, and track whether a promised remedy reached affected people. Those uses do not require pretending that the archive is a complete census or that every editorial decision is a legal verdict.
The strongest public incident reporting is neither an alarm counter nor a reassurance counter. It is an inspectable account of events, evidence, classification, and response. Its definitions tell readers what the numbers mean, and its revision history tells them why those numbers changed. Only then does the dashboard's rising or falling line become something a reader can responsibly interpret.