How RAG fails in production: a field guide to eight failure modes

"Most setups still feel like glorified notebooks stitched together with hope and vector search." That line, from a July 2025 Hacker News thread titled Are we pretending RAG is ready, when it's barely out of demo phase?, describes a lot of the systems we are asked to fix.

The good news is that RAG systems rarely fail in mysterious ways. The same handful of problems shows up again and again, and each one has a symptom you can spot, a metric that catches it, and a known fix. This guide names eight of them.

Where things go wrong

A RAG system is a pipeline: documents are split into chunks and indexed; a question becomes a query; the query retrieves passages; the passages and the question go to a model, which writes an answer. Each failure mode below lives at a specific point in that pipeline.

flowchart LR
  D[Documents] --> C["Chunking<br/>(1)"]
  C --> X["Index<br/>(4, 5)"]
  Q[Question] --> R["Retrieval<br/>(2, 3)"]
  X --> R
  R --> K["Context assembly<br/>(6)"]
  K --> G["Generation<br/>(7)"]
  G --> A[Answer]
  E["Evaluation<br/>(8)"] -.-> R
  E -.-> G

The eight failure modes at a glance

1. Chunks cut the answer apart

Symptom. The system retrieves the right document but the answer is incomplete, because the fact spans two chunks, or a chunk like "it increased 12% in Q3" has lost the context that says what "it" is.

Catch it. Build a labeled set of real questions with the passages that answer them, and track recall@k: how often the answering passage appears in the top k results.

Fix. Split along document structure (sections, headings, list items) rather than fixed character counts, and keep a small overlap. Anthropic's contextual retrieval technique, which prepends a short model-written summary of the surrounding document to each chunk, reduced top-20 retrieval failures from 5.7% to 3.7% on its own, and to 2.9% combined with contextual BM25, in Anthropic's tests. They report a one-time cost of $1.02 per million document tokens with prompt caching.

2. Vocabulary mismatch

Symptom. Questions phrased the way users talk fail against documents written the way experts write, or the reverse: a query for an exact SKU, error code or contract number misses because a vector search matched something "similar" instead.

Catch it. Tag your eval questions by type (exact identifiers, natural-language questions, mixed) and report recall per type. A system that does well on average can fail completely on one type.

Fix. Use hybrid retrieval, combining keyword and dense search. In OpenSearch's benchmark across seven datasets on OpenSearch 2.10, hybrid search improved nDCG@10 by 8.12% on average over keyword search alone, while dense-only search scored 6.97% below it. Keyword search handles exact strings; dense search handles meaning.

3. Hybrid score swamping

Symptom. You added hybrid search and results got worse, or off-topic questions return results with high scores. A common thread on the Elastic forum, Hybrid Search high score on irrelevant documents, describes exactly this, with the explanation that "BM25 (lexical) scores tend to be much higher than knn ones, thus dominating the final score."

Catch it. For a sample of queries, record which retriever each top result came from. If one retriever supplies nearly all of the top ten, the scores are not being balanced.

Fix. Do not add raw scores from different retrievers. Use rank-based fusion (reciprocal rank fusion) or normalize scores before combining them, then tune the weights against your labeled set. Never treat a hybrid score as a confidence value: it is only meaningful for ranking within one query.

4. Stale or missing content

Symptom. Answers cite last quarter's policy, a product that was discontinued, or a document that was deleted at the source but never removed from the index.

Catch it. Monitor index lag (time from a source change to its appearance in the index) and compare document counts between source systems and the index on a schedule. Alert on drift.

Fix. Treat ingestion as a production pipeline: incremental updates, deletes that propagate, retries with dead-letter queues, and a full reconciliation job that runs regularly.

5. Filters that leak or over-restrict

Symptom. Two opposite failures with one cause. Either a user sees passages from documents they should not have access to, or permission and metadata filters are so strict that searches return nothing and the model answers from its own memory.

Catch it. Keep a test suite of users and the documents each may and may not see, and run it on every change. Track the empty-result rate in production; a sudden rise usually means a filter or mapping change.

Fix. Enforce access control inside the search query, never by filtering results after retrieval, and index permissions alongside content. Treat "no results" as a signal to show the user, not a cue for the model to improvise.

6. Too much context

Symptom. The right passage was retrieved and passed to the model, and the answer still ignored it. Teams often respond by sending more passages, which makes it worse.

Catch it. For a labeled set, plot answer accuracy against the number of passages you send. If accuracy flattens or drops as you add more, you are past the useful amount.

Fix. Send fewer, better passages: retrieve broadly, rerank, and pass only the best few. Lost in the Middle (Liu et al.) found that model performance "is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts." Put the strongest evidence first.

7. Answering without evidence

Symptom. Ask a question the documents cannot answer and the system answers anyway, fluently and wrongly.

Catch it. Include out-of-scope questions in your eval set, and measure two things: the abstention rate on those, and the groundedness rate on the rest (the share of answer claims supported by a retrieved passage).

Fix. Instruct the model to answer only from the provided passages and to say when they are insufficient, require citations, and check that every citation points to a passage that was actually retrieved. A low top retrieval score is also a useful hint to abstain, as long as you only compare scores within a single retriever (see failure 3).

8. Unvalidated evaluation

Symptom. Your automated RAG scores look healthy, but users keep reporting bad answers.

Catch it. Have people label a few hundred answers and measure how often your automated judge agrees with them. Swap the order of answers in pairwise comparisons and check whether the verdict changes.

Fix. Calibrate before you trust. LLM judges can be good: in Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, GPT-4 judges reached "over 80% agreement" with human preferences, about the level at which humans agree with each other. The same study documents "position, verbosity, and self-enhancement biases, as well as limited reasoning ability." Prefer simple pass/fail judgments with a written reason over 1-to-10 scores, re-validate the judge when you change models, and keep the human-labeled set as the reference.

Five checks before you launch

If you only have time for five things, do these. Each one catches several of the failures above.

1. A labeled question set of real user questions, including out-of-scope ones, with the passages that answer them (catches 1, 2, 6, 7).
2. Retrieval metrics tracked separately from answer quality: recall@k and nDCG@10 on that set, per query type (catches 1, 2, 3).
3. Hybrid retrieval with rank-based fusion and a reranker as the default, not an upgrade (fixes 2, 3, 6).
4. Index freshness and permission tests running on a schedule (catches 4, 5).
5. A judge validated against human labels before its scores gate any release (catches 8).

None of these needs a new model. They need the same discipline search teams have applied to relevance for years, which is the most reliable way we know to move RAG out of the demo phase.

Sources

• Ask HN: Are we pretending RAG is ready, when it's barely out of demo phase?, Hacker News, 27 July 2025.
• Anthropic, Introducing Contextual Retrieval, 19 September 2024.
• Gaievski et al., Improve search relevance with hybrid search, generally available in OpenSearch 2.10, OpenSearch blog, last updated 18 June 2025.
• Hybrid Search high score on irrelevant documents, Elastic Discuss forum.
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts, arXiv 2307.03172.
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, arXiv 2306.05685.