A name appears in a search result from a historical collection. Before that result becomes a claim about a person, several different questions need answers. Did the software read the letters correctly? Does the name appear in the document or only in its description? Is it the person the researcher means? And what does the surrounding passage actually say about that person?
Those questions are especially consequential when people search intelligence and investigative records. An apparent connection can spread much faster than the explanation that a search matched the wrong person, joined two columns, or returned a later catalog description. A useful search system supplies candidates. It does not, simply by returning a result, settle identity or establish an allegation.
The Library of Congress explains that Chronicling America's searchable newspaper text is machine-generated and can contain errors arising from features such as unusual type, small fonts, markings and variable image quality. That warning describes the discovery layer, not a reason to abandon digitized collections. Library of Congress, Chronicling America FAQ
This article follows an original ten-record experiment and several related editorial decisions. Every person, document, event and institutional setting in the examples is fictional. I actually executed the search rules on hand-authored strings and retained the inputs and outputs. I did not run an OCR engine, examine newly discovered intelligence files, test a real archive's search performance or identify a real person from ambiguous records.
The distinction matters to the result. We can inspect exactly why these invented strings were retrieved and why some were missed. We cannot turn their percentages into estimates of error in the JFK collection, another archive or a commercial OCR service. What the experiment provides is a practical model for separating retrieval, transcription and identification before making a public claim.
1. The matching text may belong to a different layer
NARA's catalog guide says that catalog descriptions cannot list every person's name and that OCR-based name results can vary. It also describes searches across descriptions, extracted text and other contributed material. A matched name therefore needs to be located within the specific layer that supplied it. NARA, Using the National Archives Catalog
Consider a fictional folder whose title is Mara Ellison, but whose only digitized page is an invoice without that name in its body. A title search can correctly retrieve the folder. A writer would nevertheless be wrong to quote the invoice as though its text named Mara. The search worked according to its scope; the interpretation changed the source of the match.
Now give a second folder a broad title, Bridge Accounts, while a page inside it names Mara. A title-only search may miss that folder even when its page text is accurately searchable elsewhere. The absence belongs to the chosen field, not necessarily to the record. These two cases show why the phrase searched the archive is incomplete without an account of which fields were searched.
For the experiment below, keep a raw-text field separate from an optional title. The distinction is visible in the inputs rather than guessed after the result appears. It permits two different queries: one limited to raw text and another that searches title or text. Neither is inherently the correct query for every question. They answer different discovery needs.
An editorial note creates a third possibility. A researcher might add a proposed identification to a local spreadsheet. If the spreadsheet is later included in a combined search, the proposed name can produce a hit even though neither the page nor the official description supplied it. A circular argument begins if that hit is then cited as independent support for the proposal.
A practical match record therefore identifies its field before interpreting its significance. In our case it can say raw text, title, or researcher note. This is an author-designed distinction, not a claim about a particular archive's internal indexing implementation. It helps prevent a useful annotation from quietly acquiring the authority of an original document.
The question to ask after a promising result is concrete: show the passage or field that matched. If that cannot be done, the result remains a lead with an unresolved basis. A name printed in the interface above a thumbnail is not necessarily a transcription of anything visible on the page.
2. Build a test in which the relevant records are known
The fictional collection contains ten records, labeled P01 through P10. Six are stipulated to concern one invented bridge engineer, Mara Ellison. Two concern a different invented person with the identical name, associated with a dental practice. One concerns Marie Ellison, an invented theater worker. The final record contains an unrelated river-gauge sentence.
These identities are part of the constructed test, not deductions made by the program. They let us evaluate the search rules against a known target set. In a real historical investigation, deciding which records concern the target person would be part of the difficult work rather than an answer supplied in advance.
P01 contains the engineer's full name correctly. P02 replaces the two lowercase letters in the middle of Ellison with digit ones, producing E11ison. P03 splits the surname after Elli, followed by a hyphen and a line break before son. P04 uses M. Ellison. P07 uses the surname alone. P08 is designated as a relevant image-only record with no searchable text supplied.
The distracting records make the test less forgiving. P05 contains the full name Mara Ellison but concerns the fictional dental namesake. P06 contains Marie Ellison. P09 has Mara Ellison in its title but only an unrelated dental-supply invoice sentence in its text. P10 mentions inspection of a river gauge and contains none of these names.
All of those strings were written specifically for this exercise. The corrupted spellings are deliberate inputs, not observed OCR errors produced by a tested model. P08's image-only designation is likewise a scenario condition; no image was created or processed. Stating these boundaries prevents a small demonstration from masquerading as a benchmark of real recognition technology.
The research target is also deliberately narrow: retrieve records about the stipulated bridge engineer. If the target instead were every appearance of the exact character string Mara Ellison, P05 would be relevant. The same retrieved set can receive a different evaluation when the information need changes. That is why the target definition belongs beside the result.
Before running a query, our packet therefore contains the ten inputs, the six target labels, and the distinction between text and title. Nothing in the result is allowed to rewrite those labels merely to improve the score. When a rule misses P02 or finds P05, the discrepancy is recorded rather than rationalized away.
3. The exact-name search is precise about letters, not identity
A literal, case-insensitive search for Mara Ellison in raw text returns P01 and P05. The program has done exactly what was requested: both strings contain the phrase. Only P01 belongs to the target engineer. P05 belongs to the stipulated namesake. Five of the six relevant records remain outside the returned set.
In information retrieval, precision measures the relevant share of retrieved items, while recall measures the retrieved share of relevant items. Those are the definitions used in Manning, Raghavan and Schütze's Introduction to Information Retrieval. They require a relevance judgment, not merely a count of search hits. The authors' online treatment of precision and recall
For this run, one relevant result out of two retrieved gives fifty-percent precision. One retrieved target record out of six gives roughly 16.7-percent recall. These are exact consequences of our fixture labels and query rule. They are not measurements of an archive, an OCR engine, or the likelihood that an unfamiliar name match identifies the desired person.
The outcome exposes two independent problems. The literal query misses altered or abbreviated forms of the target name. At the same time, the exact spelling admits a different person. Improving the recognition of letters would address some missed candidates, but would not remove the namesake. Character correctness and person identity are separate properties.
Suppose an editor sees only the two returned snippets. Without the target labels, the editor cannot know from the shared name alone that one is the engineer and one is the dental worker. A search interface that emphasizes the identical words may make them look strongly connected even though our constructed facts say otherwise.
Nor does the small returned set establish that the engineer rarely appears in the collection. The search found one of six relevant records because of the way their text was represented. A statement that there was only one relevant record would confuse retrieval with an exhaustive review. In this controlled case we can demonstrate that error directly because we know the hidden remainder.
This first run gives the researcher a useful starting list, not a conclusion about identity or frequency. Its limitation is specific enough to investigate: which relevant forms were excluded by the phrase rule, and which returned record needs namesake discrimination? Those questions guide the next runs without changing what the first one actually observed.
4. Normalization recovers candidates without repairing history
The second run uses a deliberately simple normalization rule. It removes a hyphen immediately followed by a line break, replaces digit one with lowercase l, collapses whitespace, and compares without regard to letter case. These are local experimental rules. They are not presented as universally safe corrections to historical documents.
The normalized phrase search returns P01, P02, P03 and P05. P02 becomes findable because its two digit ones are transformed into letters. P03 becomes findable because its split surname is joined. Three of the four returned records concern the engineer, giving seventy-five-percent precision and fifty-percent recall in this fixture.
The new result is better under both of our chosen measures, but the dental namesake remains. The two successes concern how a string was represented. They do not add evidence that people sharing the normalized string are the same person. That unresolved identity question survives even though the retrieval score improves.
The rule also has costs that the favorable total should not conceal. A digit one may be an actual number, and a hyphen may be meaningful rather than a line-ending artifact. Replacing characters globally can create a useful search derivative while producing text unsuitable for quotation. Our fixture was designed to reveal two recoverable forms; it is not evidence that the same transformations improve every collection.
For that reason, the original P02 and P03 strings remain unchanged in the saved input. The program transforms them during matching and records the rule. A reader can therefore explain why a query succeeded without being shown a silently cleaned version as the original transcription. Search normalization should not become an undocumented editorial rewrite.
Imagine adding a fictional inventory code containing the digit one. The same global replacement would alter that code even if no name search required it. A researcher comparing original identifiers could then manufacture a discrepancy. This is why a transformation useful for candidate generation can be inappropriate for fields whose exact characters carry administrative meaning.
For the engineer search, the added candidates now deserve inspection under their original representation and context. The honest report is that a broader rule recovered P02 and P03. It is not that the program established the correct historical spelling, verified an actual page, or authenticated either record. Those would require observations this experiment did not perform.
5. A surname search changes both the workload and the misses
The third run searches the unmodified text for Ellison alone. It returns P01, P04, P05, P06 and P07. That recovers the engineer's initial-only and surname-only records, but also brings in Marie Ellison. It still misses the digit-corrupted and line-split forms because this run does not apply normalization.
Three of the five returned records belong to the target set. The result therefore has sixty-percent precision and fifty-percent recall. Its recall matches the normalized phrase search, but it finds a different selection of relevant records. Equal summary scores do not imply that two search strategies produced equivalent evidence.
Combining the three result sets gives seven distinct candidates: P01 through P07. Five concern the engineer. The union has about 71.4-percent precision and 83.3-percent recall. P08 remains missing because no searchable text was supplied. Searching the same absent field more cleverly cannot recover content that the experiment never placed there.
The union also shows why adding the three hit counts would be wrong. Two plus four plus five equals eleven query hits, but those hits refer to only seven distinct records. P01 and P05 recur across the searches. A researcher who reports eleven documents without deduplicating the identifiers would exaggerate the number examined.
For a review queue, retain the route by which each candidate entered. P02 came through normalized phrase matching, P04 through the surname rule, and P05 through all three. Multiple query routes do not give P05 three independent confirmations of identity. They merely describe three ways the same text satisfied our search conditions.
There is a practical choice here rather than a universally best query. A narrow first pass may help orient a researcher. A broader pass may recover overlooked candidates but demand more discrimination. The relevant question is what evidence the researcher examined after retrieval, not whether the result count grew impressively.
The saved union is therefore a worklist with seven identifiers and their matching routes. It is not a list of seven verified appearances by the engineer. Five are targets only because the exercise supplied their identities in advance. In real research, the worklist would still need a documented identification decision for each candidate, including explicit unresolved cases.
6. Distinguish a matching name from a matching person
NARA's definition of a biographical note says it supplies information about an individual's life and helps distinguish people with the same name. The existence of such descriptive context reflects a basic archival problem: a name string by itself may not uniquely identify a person. NARA's Biographical Note data element
Return to P01 and P05. In our invented setting, both contain Mara Ellison, but one concerns bridge inventory and the other a dental appointment list. Those subjects suggest different contexts; they do not, by themselves, prove two people. A person can have more than one role. The test's identity labels are stipulated separately, not inferred from a simplistic rule that engineers never visit dental offices.
A real identification decision needs a relationship supported by the records. Suppose a fictional personnel note explicitly links the engineer's full name with the signature M. Ellison used on the bridge estimate. That would help connect P04 to the named individual. A shared initial without that link would remain weaker evidence, especially where multiple people could satisfy it.
Now remove the personnel note from the example. The available evidence changes, so the identification status should change. It would be misleading to keep a confident match merely because the fuller scenario once supplied one. A research system needs to distinguish a candidate association from a supported identification and preserve why the label was assigned.
A useful local decision record can hold the candidate identifier, the proposed identity, the specific supporting passage, any contradiction, and the remaining uncertainty. For P04, it might say that the initial and subject matter are compatible with the engineer but that the explicit signature link has not been found. Compatible is not the same as established.
This caution becomes particularly important when the document contains an allegation. Even a correct identification does not prove the allegation; it only connects the reference to the person. If the identification itself remains uncertain, the writer has two unresolved steps rather than one. Combining them into a confident headline compounds the uncertainty.
Our fictional namesakes make the distinction visible without implicating anyone real. The aim is not to insist that every historical identity is unknowable. It is to require a stated basis for a link and to prevent the convenience of a matching string from supplying evidence that the records do not contain.
7. Page layout can create a relationship that no sentence contains
The Library of Congress describes ALTO as an XML format for storing recognized text together with page-layout information. That separation matters because where words occur can affect how they should be read. The discussion here uses an invented layout, not an ALTO file or a claim that a particular archive implements that standard. Library of Congress, ALTO Principles
Imagine a fictional page with two independent columns. The left column lists bridge correspondence, including Mara Ellison. The right column lists theater bookings, including a person called Calder. A careless flattening operation alternates fragments from the two columns. The resulting text places the names next to one another in an apparent sentence that neither original column contains.
A proximity search could now return the page for both names. That result might be useful for locating the page, but it would not establish that the two people met, corresponded, or appeared in the same account. The apparent relationship was introduced by the ordering of text fragments rather than by the described events.
The correction begins with the layout, not with a biographical theory. In this fictional case, the researcher must reconstruct which words belong to each column and which heading governs each entry. A line break lost in extraction can matter, but a line break alone also cannot settle the intended reading. The actual visual organization is the relevant evidence.
Consider a second arrangement: a two-column table whose left side contains names and whose right side contains dates. If one row is skipped during extraction, every later name may appear beside the next person's date. The individual words and numbers could all be transcribed correctly while the relationships are wrong. Character accuracy would not detect that association failure.
For an original diagnostic example, let the table contain three rows: Person A with Date One, Person B with Date Two, and Person C with Date Three. Dropping Date One while retaining the three names produces a sequence in which A can be mistakenly paired with Date Two and B with Date Three. The error is not that those dates are unreadable; it is that the row correspondence has been lost.
The appropriate publication decision is to withhold the inferred pairing until the row structure is checked. A screenshot crop that contains only the mistaken flattened text would hide the issue. A properly contextualized illustration would show enough of the source layout to explain the association, while avoiding unrelated personal information.
This is a distinct check from deciding whether Mara and M. Ellison are the same person. Layout asks which words belong together before identity analysis begins. Skipping it can give a perfectly careful namesake comparison the wrong sentence to compare.
8. Confidence scores cannot settle the identity question
In a 2023 research paper, Antonia Karaisl explains that OCR confidence is not an absolute measure: high-confidence output can be wrong and lower-confidence output can be correct. The paper explores confidence for script analysis, not a universal calibration guarantee for every recognition system. Karaisl, A Question of Confidence
To isolate the editorial issue, I created a second small control with twelve hand-assigned scores and twelve stipulated correctness labels. No OCR model produced these scores. Nine entries are labeled correct and three incorrect. Six scores are at least 0.90; among those six, five are correct and one is incorrect.
Applying a 0.90 threshold therefore retains six entries with five correct results, about 83.3 percent of the retained set. It also retains only five of the nine correct entries in the full control, about 55.6 percent. These observed arithmetic results show what our invented threshold does. They do not estimate the calibration or accuracy of a real confidence score.
The wrong retained entry has a score of 0.97. Its presence is enough to defeat a proposed rule that every value above 0.90 may be quoted without checking. Raising a threshold might change the selected set, but the threshold itself cannot supply a missing observation of the page or establish who a name refers to.
There is also a different logical gap. Suppose a name has been transcribed perfectly. The string may still refer to the dental namesake rather than the engineer. A score intended to describe recognition of characters cannot resolve that distinction merely by being high. Correct reading is necessary for some identification claims, but it is not the entire identification argument.
In a practical review queue, a score can help decide which items deserve attention first if its meaning and limits are understood. The local decision should still identify the observation needed: inspect a difficult letter, check the row alignment, or locate evidence distinguishing two people. Treating all three as one confidence problem conceals what would actually resolve each case.
For the fictional P02, the record says E11ison because I supplied that string. We do not have a measured OCR confidence for it, and the packet does not invent one. The separate twelve-entry score control is not attached retrospectively to the ten-record search experiment. Keeping those exercises distinct prevents an attractive but unsupported combined performance claim.
9. Preserve an unreadable character instead of completing a story
NARA's transcription guidance asks contributors to preserve what is written, including misspellings, and to mark an unreadable word rather than silently supply a guessed reading. It distinguishes transcription from corrections or explanatory comments added elsewhere. This is a useful boundary between a record's text and a researcher's interpretation. NARA, Transcription Tips
Imagine a fictional surname whose first character is genuinely unclear in the available scan. One reader proposes Ellison; another proposes Allison. The surrounding topic might make one interpretation seem more plausible, but plausibility is not the same as visible evidence. A publication should not make the uncertain letter disappear merely because one reading fits an appealing narrative.
An uncertainty note can be precise rather than vague. It can identify the page region, the competing readings, and the reason the image does not decide between them. If another scan or a clearly written repetition becomes available, that can be evaluated separately. Until then, the uncertainty remains part of the evidence rather than a defect to be edited out.
Now distinguish an OCR error from a source error. Suppose the fictional original clearly spells a name as Elison, with one l, while independent contextual material uses Ellison. A faithful transcription of the page would preserve Elison. An editorial note might explain the proposed connection, but changing the transcription to the expected spelling would erase what the page actually contains.
The two situations require different labels. In the first, the character is unreadable to the reviewer. In the second, the character sequence is readable but may be a historical misspelling. Neither situation authorizes pretending that the page unambiguously displays the preferred form. A search derivative can include a variant without being passed off as the literal text.
Consider a third case in which two legible records disagree. One gives a date in June, another in July. Correcting the first to July because the second looks more plausible would collapse a substantive disagreement into a transcription adjustment. The research question is now which account is reliable, not which characters the software should recognize.
For the engineer worklist, an unresolved reading should therefore carry two separate statuses if necessary: transcription uncertain and identity unresolved. Resolving the first does not automatically resolve the second. This is a more informative outcome than assigning a single confidence adjective that leaves the reader unable to see what remains unknown.
The editorial benefit is practical. A later correction can identify exactly which observation changed rather than retract a chain of conclusions built on a guessed letter. Clear uncertainty makes future research easier; an invented certainty forces the next researcher to undo an assumption before examining the evidence.
10. Field coverage can explain a missing result
The Smithsonian's Freedmen's Bureau search guide describes indexed names, dates and locations separately from transcribed text, and warns that not every name or detail was indexed. Its specific interface combines those resources in defined ways. That is an example of why a researcher should read a system's field definitions rather than assume every search box covers the same material. Smithsonian, Guide to the Freedmen's Bureau Search Portal
Our synthetic fixture demonstrates a simpler version. Searching only raw text for the exact full name returns P01 and P05. Searching title or text returns P01, P05 and P09. The added P09 is not a new discovery about the engineer. It enters because the query now includes a field containing the dental namesake's name.
Under the fixture's target labels, the expanded exact search retrieves one relevant item out of three, about 33.3-percent precision. Recall stays at one of six, about 16.7 percent. Broadening the field scope increases the hit count without recovering any additional target record in this particular experiment.
That is not an argument against metadata search. If the research need concerned the namesake's folder, P09 could be useful. It is an argument for naming the field scope and evaluating the result against the intended question. A different purpose can make the same result appropriate without changing the literal contents of the file.
Dates can create a similar misunderstanding. Suppose a fictional portal has a document-date field, while a researcher wants the date of the event described in the document. A memo written in October about a September meeting will be excluded by a September-only document-date filter. The search may be functioning correctly even though the researcher's assumed meaning of date is wrong.
Now add an undated document whose text mentions that same meeting. Requiring a populated date field could exclude it altogether. The absence would reflect the filter and metadata coverage, not an observation that the document contains no relevant account. A review log should identify such exclusions before treating a restricted result set as comprehensive.
In our ten-record exercise, P08 supplies the sharpest coverage boundary. It is stipulated to be relevant but has no text. None of the text rules can retrieve it. The appropriate next step in a real investigation would depend on what image or descriptive access is actually available; the experiment does not claim to have performed that step.
A bounded search conclusion can therefore be both useful and incomplete: these queries recovered five of the six stipulated targets, and one relevant item remained inaccessible to the supplied text layer. That statement explains the limitation without inventing the content of an unseen image or assigning blame to a search service we did not test.
11. A small overall error can affect the only name that matters
For a third control, I created an original two-thousand-character reference string beginning with Mara Ellison and filled the remaining positions with a neutral repeated character. I made two equal-length variants. One changes a letter inside Ellison. The other changes the very last filler character while leaving the name intact.
A direct position-by-position comparison finds one mismatch in each variant. One divided by two thousand is 0.05 percent. The aggregate mismatch fraction is therefore identical, yet a literal search for the full name fails in the name-error variant and succeeds in the other. Both comparisons and searches were actually executed on the synthetic strings.
This is not a standard OCR benchmark, a realistic page model or an implementation of an edit-distance evaluation. The strings have the same length and deliberately trivial filler. The control asks a limited question: can an equal overall mismatch count have different consequences for a particular name search? The observed answer is yes.
The result helps explain why a general quality summary may be insufficient for an identity-sensitive quotation. A reader interested in one name needs that name checked, not merely a favorable average across unrelated text. Conversely, one important error does not imply that every other word on the page is unusable. The scope of the finding should remain local.
Consider the reverse editorial problem. A reviewer sees many harmless spacing differences and concludes that the entire record is unreliable. In our constructed example, those differences could leave the relevant name and sentence untouched. The correct inquiry is which differences affect the claim, not whether a document can be assigned one dramatic quality label.
A useful claim-focused review can separate three questions. Does the displayed name match the proposed transcription? Does the surrounding syntax assign the claimed role or action to that name? Does contextual evidence identify the person? Our two-thousand-character control addresses only the first search-sensitive issue, and even there it uses invented text rather than a real image.
The controlled result also prevents an overclaim about our earlier normalization success. Recovering P02 and P03 shows that the selected rules handle those supplied forms. It does not establish that the remaining text is accurate, that all meaningful errors are recoverable, or that a global score predicts the reliability of every individual name.
The retained output is intentionally narrow: two strings, one mismatch each, different exact-name retrieval results. That is enough to support the practical conclusion without pretending that a toy calculation has validated an archive's digitization quality or a particular historical interpretation.
12. Publish the supported proposition, not the search interface's implication
Finish by giving an editor six proposed claims about the experiment. The first says that the exact phrase occurs in two raw-text records. That is supported by the saved run. The second says that both records concern the engineer. That is contradicted by the stipulated identities, because P05 is the namesake.
The third says that the normalized query recovered two target records missed by the exact phrase query. That is supported: P02 and P03 were added. The fourth says that their historical spelling was verified from an original page. That is unsupported because there were no page images or historical records in this experiment.
The fifth says that the combined search retrieved all relevant material. That is false under the fixture: P08 remained unretrieved. The sixth says that five of the six stipulated targets were recovered by the combined rules. That is supported, provided the account retains the scope and does not translate the fraction into a claim about a real archive.
These decisions produce more useful writing than a generic statement that the search was verified. They identify which operation was actually performed and which conclusion follows from it. A reader can disagree with the chosen research target or propose another rule while still reproducing the reported result.
For a real investigation, the comparable publication packet would connect each consequential claim to the record and the observation needed to support it. A name match would have an identification basis. A quotation would have a checked source passage. A claim about a relationship would identify the language or independent evidence establishing that relationship, not merely a pair of nearby search highlights.
The wording should preserve the difference between a record's assertion and the writer's conclusion. If a fictional memo reports that someone believed two people had met, the defensible initial statement concerns that report. It does not become a confirmed meeting simply because the memo is old, official-looking or searchable by both names.
Unresolved cases can still be reported responsibly when they serve a genuine public-interest purpose, but uncertainty should not be used as a vehicle for insinuation. An ambiguous letter is not an invitation to supply a suspect. A namesake is not a reason to publish an unrelated person's details. Where the evidence does not establish the link, the account should say so or leave the proposed identification out.
The actual result here is a reproducible demonstration, not a historical discovery. Ten invented records exposed distinct failure modes in exact matching, normalization, field scope and identity interpretation. Two additional controls showed why a score threshold and an aggregate mismatch fraction cannot replace the relevant observation. No real person's conduct was investigated or alleged.
Searching historical records is valuable precisely because it can reveal material a reader would otherwise miss. The discipline is to let the search open a question without allowing it to answer a different one by implication. Read the matched layer, preserve uncertain characters, separate namesakes, and make the published proposition no larger than the evidence that supports it.