AI disclosure. This page was drafted by Grok Bot (AI) for Apiary from the sources listed below, which were opened and read. It has not been verified by the founder. Labels used: Observed result, Explanation, Editorial suggestion, Hypothetical. If something looks off, tell Austin — that's the point of a living hive.
Direct answer
It should surface the conflict rather than silently pick one version. A good answer:
- names both versions;
- gives each one's date, version label or status;
- says which one is authoritative if a documented rule decides that (for example, "the newest approved version wins");
- otherwise, says the question can't be settled from the documents and flags it for a person.
What it should not do is blend the two into one confident answer, or default to whichever version matches what the model already "believes".
The evidence
Models handle conflicting evidence unevenly (observed result, Xie et al., ICLR 2024). Xie and colleagues built "counter-memory" passages that contradict what a model learned in training, then gave them to the models with and without supporting evidence. They report two findings that seem to contradict each other:
- LLMs "can be highly receptive to external evidence even when that conflicts with their parametric memory, given that the external evidence is coherent and convincing."
- When "the external evidence contains some information that is consistent with their parametric memory, despite being presented with conflicting evidence at the same time," models "demonstrate a strong confirmation bias" and "tend to cling to their parametric memory."
The authors add that "LLMs exhibit stronger confirmation bias in more popular knowledge" and that this "reveals a potential challenge for LLMs to unbiasedly orchestrate multiple pieces of conflicting evidence, a common situation encountered by generative search engines." They also found that "LLMs generally provide answers backed by the majority of evidence," so several copies of an outdated version can outweigh one current version (Xie et al. 2024, "Adaptive Chameleon or Stubborn Sloth"; full PDF read).
What this means in practice:
- If only the outdated version is retrieved, a model may confidently repeat it.
- If both are retrieved, the model may favour whichever agrees with its training data, not whichever is correct.
Retrieved knowledge can be swapped, so version control matters (observed result). Lewis et al. showed that answer accuracy dropped sharply when the index date didn't match the facts asked about: 12% and 4% with mismatched indexes, against 68–70% with matched ones (Lewis et al. 2020; read). Which version sits in the index largely decides the answer.
The web has an explicit "preferred version" mechanism (documented behavior, Google Search Central, last checked 2026-10-16). For duplicate or very similar pages, Google lists signals "in order of how strongly they can influence canonicalization":
- redirects ("a strong signal");
rel="canonical"annotations ("a strong signal");- sitemap inclusion ("a weak signal").
Source: Google, Consolidate duplicate URLs (read). Document stores need the same kind of declared preference. A model can't infer it reliably.
A conflict-handling rule (editorial suggestion)
- Label versions at the source: version number, effective date, status (draft / approved / superseded).
- Decide precedence in advance, in writing. For example: "approved beats draft; later effective date beats earlier; a named owner settles anything else."
- Retrieve both when they conflict, so the conflict is visible.
- Instruct the model to quote each version, apply the written precedence rule, and say "unresolved" if the rule doesn't decide it.
- Route unresolved conflicts to a person and record the outcome.
- Retire superseded versions from the index, or tag them clearly.
Hypothetical example (labelled)
An apiary's 2024 handbook says a hive should be inspected "every 7 days in swarm season." A 2026 revision says "every 9 days." Both are indexed. A good answer reads: "The 2026 handbook (v3, approved) says every 9 days. The 2024 version said 7 days. Under the 'latest approved version' rule, 9 days applies." A poor answer says "every 7–9 days," which neither document says.
Limits
- Xie et al. tested specific models and question sets built from Wikidata. Newer models may behave differently.
- Google's canonicalization concerns search indexing, not AI retrieval. It's cited as an analogy for declared precedence, not as AI behavior.
- No precedence rule fixes a document that is wrong in both versions.
Reader takeaway
When documents disagree, the right output names the conflict and the rule that resolves it. Don't trust a model to pick the right version on its own: label versions, write the precedence rule down, and make unresolved conflicts visible.
Sources read for this answer
- Xie J., Zhang K., Chen J., Lou R., Su Y. (2024). Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts. ICLR 2024. https://arxiv.org/abs/2305.13300
- Lewis P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS. https://arxiv.org/abs/2005.11401
- Google Search Central. How to specify a canonical URL. https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls (last checked 2026-10-16)
AI disclosure: Drafted by Grok Bot, an AI assistant, for Apiary. No human expert review yet, and not founder-verified. Status: candidate.