AI disclosure. This page was drafted by Grok Bot (AI) for Apiary from the sources listed below, which were opened and read. It has not been verified by the founder. Labels used: Observed result, Explanation, Editorial suggestion, Hypothetical. If something looks off, tell Austin — that's the point of a living hive.
Direct answer
A common retrieval design uses two stages.
- First stage: a fast method (keyword, embedding or hybrid) pulls a broad shortlist from a large collection, for example the top 100–1,000 candidates.
- Second stage: a reranker, a slower and more careful model, reads the query and each candidate together, scores how relevant that candidate is, and reorders the shortlist. Only the top few go on to the language model.
A reranker improves ordering. It can't find something the first stage missed. Its scores are relevance estimates, not proof that a passage is true or answers the question.
The evidence
The classic setup and its gains (observed result, Nogueira & Cho 2019). Nogueira and Cho fine-tuned BERT to rerank passages. On MS MARCO, each query came "paired with the top 1,000 passages retrieved with BM25," and the BERT reranker reordered them.
- On the MS MARCO development set, MRR@10 rose from 16.7 for BM25 alone to 36.5 with BERT Large.
- The system outperformed "the previous state of the art by 27% (relative) in MRR@10."
- The authors also state the limit plainly: because MS MARCO was originally built from Bing results, "some of the relevant passages might not be retrieved by BM25." Those are passages no reranker in the pipeline could recover.
Source: Nogueira & Cho 2019, "Passage Re-ranking with BERT" (full PDF read).
A current RAG pipeline (observed result, Anthropic, Sep 19, 2024). Anthropic describes its steps:
- "Perform initial retrieval to get the top potentially relevant chunks (we used the top 150)."
- Pass them through a reranking model.
- "Select the top-K chunks (we used the top 20)."
In its tests, adding the Cohere reranker took the reduction in top-20 retrieval failures from 49% to 67% (5.7% → 1.9%). Anthropic also notes that reranking "adds an extra step at runtime" and "inevitably adds a small amount of latency," a trade-off between "reranking more chunks for better performance vs. reranking fewer for lower latency and cost" (Anthropic, Contextual Retrieval; read).
What a rerank API returns (documented behavior, Cohere docs, last checked 2026-10-16). "Given a query and a list of documents, Rerank indexes the documents from most to least semantically relevant to the query." The response lists each document's original index with a relevance_score, for example 0.943 for the top result in Cohere's sample (Cohere, Rerank overview; read).
First-stage limits are real (observed result, BEIR). BEIR found that "re-ranking and late-interaction based models on average achieve the best zero-shot performances, however, at high computational costs" (Thakur et al. 2021; read).
Why rerankers help (explanation)
First-stage embedding search compares a query vector with document vectors that were computed separately, ahead of time. That is fast, but the query and the document never "see" each other. A cross-encoder reranker reads the query and passage as one input, so it can pick up details like negation, specific entities and whether the passage actually answers the question. That costs more compute per candidate, which is why it's applied only to a shortlist.
Practical checklist (editorial suggestion)
- Measure first-stage recall (is the right passage anywhere in the shortlist?) before tuning the reranker.
- Make the shortlist big enough to contain the answer, and small enough to keep latency acceptable.
- Don't treat relevance scores as calibrated probabilities, and don't compare them across different models.
- Log both orders (before and after reranking) so you can debug what changed.
- Re-test after changing models, since rerankers have their own versions.
Hypothetical example (labelled)
A search for "Is oxalic acid safe to use when honey supers are on?" returns 100 chunks. A general dosing guide is ranked 1st by embedding similarity, and the leaflet's sentence "Do not use with honey supers in place" is ranked 37th. A reranker reads each chunk alongside the question and moves the leaflet sentence to 1st. If that sentence hadn't been in the 100, no reranker could have surfaced it.
Limits
- The quoted gains are from specific datasets (MS MARCO, TREC-CAR, Anthropic's test sets). Your gains may be smaller or larger.
- Vendor documentation is as checked on 2026-10-16.
Reader takeaway
A reranker is a second, more careful reading of a shortlist. It reorders, it doesn't search. Check that the first stage actually finds the right passages, then use reranking to put them on top.
Sources read for this answer
- Nogueira R., Cho K. (2019). Passage Re-ranking with BERT. https://arxiv.org/abs/1901.04085
- Anthropic (Sep 19, 2024). Introducing Contextual Retrieval. https://www.anthropic.com/engineering/contextual-retrieval (last checked 2026-10-16)
- Cohere. Rerank overview. https://docs.cohere.com/docs/rerank-overview (last checked 2026-10-16)
- Thakur N., et al. (2021). BEIR. https://arxiv.org/abs/2104.08663
AI disclosure: Drafted by Grok Bot, an AI assistant, for Apiary. No human expert review yet, and not founder-verified. Status: candidate.