Your AI agent is only as good as its retriever

When an AI agent gives a wrong answer, the instinct is to blame the model and reach for a bigger one. Sometimes that is right. More often, in our experience, the agent reasoned perfectly well about the wrong information, because its search tool never surfaced the right document.

Three public benchmarks make the case that retrieval is one of the largest and cheapest levers on agent quality. None of them was run by us; we cite each one and link the source. What they show, together, is a pattern worth designing for.

Same agent, different retriever: 55.9% vs 70.1%

BrowseComp-Plus is a benchmark for "deep research" agents: hard questions that need several searches and some reasoning to answer. Unlike live-web benchmarks, it fixes the corpus at 100,195 documents with 830 queries, so you can swap one component and see what changes.

The authors ran the same GPT-5 agent with two different retrievers behind its search tool:

Source: BrowseComp-Plus, Table 1.

xychart-beta
  title "GPT-5 agent accuracy by retriever (BrowseComp-Plus)"
  x-axis ["BM25", "Qwen3-Embedding-8B"]
  y-axis "Accuracy (%)" 0 --> 100
  bar [55.9, 70.1]

Nothing about the model changed. The agent answered 14 more questions in every hundred correctly, and it needed fewer searches to do it. Better retrieval is not a trade of cost for quality here; it improved both.

The retrieval metrics explain why. Measured directly on the documents needed to answer each question, BM25 put 4.7% of the evidence documents in its top 100 results; the dense retriever found 47.7%. On nDCG@10, the standard measure of how good the top ten results are, BM25 scored 1.6 and the dense retriever 20.3 (Table 2). The agent with BM25 was reasoning over search results that rarely contained what it needed.

Retrieval turns guessing into answering

FRAMES, a benchmark of multi-hop factual questions, makes the same point from the other direction. The authors report that a model reached 0.40 accuracy with no retrieval and 0.66 with a multi-step retrieval pipeline, an improvement of more than 50%.

Multi-hop questions are exactly what agents are for: find one fact, use it to look up the next. Each hop is a search, and each search has to work.

Leaderboard embeddings fail on hard queries

It would be convenient if the fix were "use the top embedding model on the leaderboard". BRIGHT, a benchmark of 1,384 real-world queries that need reasoning to find the relevant documents, shows why it is not that simple. SFR-Embedding-Mistral, the leading model on the MTEB leaderboard when BRIGHT was published with a score of 59.0, scored 18.3 nDCG@10 on BRIGHT.

Agents ask harder questions than people type into a search box. A query like "which pricing change preceded the rise in enterprise churn" does not share many words with the documents that answer it, and its meaning depends on reasoning about time and cause. BRIGHT also found that having a model reason about the query before retrieving improved results by up to 12.2 nDCG@10 points. For agents, that is good news: the agent is already a reasoning model, so it can rewrite its own queries.

Why retrieval matters even more for agents

Three properties of agents make retrieval quality matter more than it does for a single search box:

1. An agent cannot reason its way to a document it never sees. If the right evidence is not in the results, the best outcome is "I don't know" and the usual outcome is a confident wrong answer.
2. Errors compound across steps. An agent that searches 20 times is exposed to 20 chances for a bad result to send it the wrong way. We work through that arithmetic in The reliability math of AI agents.
3. Agents query differently from people. They write long, natural-language queries, sometimes close to the whole question. Keyword matching was built for short queries. In BrowseComp-Plus's direct retrieval test, which used the full question as the query, BM25's top ten results were close to useless (nDCG@10 of 1.6).

sequenceDiagram
  participant U as User
  participant A as Agent
  participant S as Search tool
  participant I as Index
  U->>A: Question
  loop Until the agent has enough evidence
    A->>A: Rewrite the question as a query
    A->>S: search(query, filters)
    S->>I: Hybrid retrieval and rerank
    I-->>S: Top passages with IDs
    S-->>A: Compact results
    A->>A: Read, decide: answer or search again
  end
  A-->>U: Answer with citations

What to do with this

These are our recommendations from production work, informed by the results above.

Measure retrieval separately from the agent. Log the queries your agent actually sends, label which documents should come back for a sample of them, and track recall@k and nDCG@10 on that set. When agent quality drops, this tells you in minutes whether retrieval is the cause.

Start from hybrid retrieval, not keyword or vectors alone. In OpenSearch's own benchmark across seven datasets (run on OpenSearch 2.10), combining keyword and dense retrieval improved nDCG@10 by 8.12% on average over BM25, while dense-only retrieval scored 6.97% below BM25. Those tests used an older embedding model, so re-check the balance on your data, but the principle holds: keyword search catches exact names, codes and IDs; dense search catches meaning. Agents need both.

Let the agent rewrite queries, and give it filters. Expose structured filters (date ranges, document type, customer, product) as tool arguments instead of hoping the query text carries them. Encourage the agent to reformulate a failed query rather than repeat it.

Return compact, citable results. Passages with stable IDs, a title and the key fields. Long, noisy results waste context and make the next step less reliable.

Rerank before the agent reads. A reranker over the top 50 to 100 candidates is one of the cheapest quality gains in retrieval, and it means the agent reads the best five results instead of the first five.

Spend on retrieval before you spend on a bigger model. Upgrading the model raises cost on every call. Improving retrieval is a one-off engineering investment, and, as BrowseComp-Plus shows, it can also reduce the number of calls.

A caveat on benchmarks

These benchmarks are research tasks on public corpora, not your documents or your users. They are strong evidence about the direction of the effect and its size, not a promise of the same numbers on your data. The only way to know what better retrieval is worth for your agent is to measure it on your own queries, which is why the first recommendation above matters most.

Sources

• Chen et al., BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent, arXiv 2508.06600 (Tables 1 and 2).
• Krishna et al., Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (FRAMES), arXiv 2409.12941.
• Su et al., BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval, arXiv 2407.12883.
• Gaievski et al., Improve search relevance with hybrid search, generally available in OpenSearch 2.10, OpenSearch blog, last updated 18 June 2025.