Find the Failing RAG Stage Before You Change the Prompt

A wrong answer does not tell you which stage broke

You are building a research assistant that answers questions from a corpus of paper abstracts. Asked which paper first paired a sequence-to-sequence generator with dense retrieval over Wikipedia, it names the wrong paper, confidently. The quick move is to rewrite the system prompt: add "cite your source", lower the temperature, try again. Sometimes the next answer is right, by accident. That tells you nothing about what broke. If the right abstract never reached the model, no wording fixes it, because the model cannot use evidence it never received.

A retrieval-augmented generation (RAG) system has at least four stages that can lose the answer, and a failure at any of them produces the same visible symptom:

1. Candidate retrieval: does the search over the corpus return the relevant document at all?
2. Reranking: does a second model keep it near the top when it narrows the candidates?
3. Prompt assembly: does the document's text actually land in the prompt, inside the context budget?
4. Generation: does the model use the evidence it was given?

Finding which stage failed, for each query, is stage attribution. It turns "the answer was wrong" into a table you can act on. The from-scratch treatment of the first stage is in the book: Chapter 5 derives BM25 and recall@k, and Chapter 6 measures embedding search against it on paraphrased questions. This article adds what those chapters stop short of: attributing failures across all four stages and putting intervals on the rates.

A five-case fixture

Start with a fixture small enough to trace by hand. Three abstracts:

The question has one relevant abstract, d2, by construction. This is an authored fixture, not a measured benchmark. Its value is that you know the right answer and can see where it gets lost.

Case A never retrieves d2. Recall@k, the share of the relevant documents that appear in the top k candidates, is 0 for this question. Nothing downstream can recover d2's contents. This is the case the prompt rewrite targets by mistake: a different prompt yields a different wrong answer, which looks like progress and is not.

Case B retrieves d2, so recall@k is 1, but the reranker drops it. The symptom is identical to case A; the fix is not. Inspect the reranker's scores for this pair, and its selection limit and truncation. An unavailable reranking endpoint is a separate service failure, not evidence of poor relevance scoring.

Case C is the first case that implicates generation, assuming the passage really contains the answer and the label is right. The evidence was in the prompt and the model still answered wrong. Only here is changing how you instruct the model the right next move.

Case D catches a boundary that is easy to skip: the reranker kept d2, but prompt assembly truncated or dropped its text. IDs in a reranker's response are not proof that the passage reached the generation request. Inspect the assembled prompt itself.

View full-size diagram

flowchart LR
    A["Query"] --> B["Candidate retrieval"]
    B --> C["Reranking"]
    C --> D["Prompt assembly"]
    D --> E["Generation"]
    B -.->|"recall miss"| F["Diagnosis: retrieval"]
    C -.->|"drops relevant ID"| G["Diagnosis: reranking"]
    D -.->|"drops passage text"| I["Diagnosis: prompt assembly"]
    E -.->|"ignores present evidence"| H["Diagnosis: generation"]

The rule that falls out of the table: a generation-only prompt change cannot repair a missing candidate. A broader change that rewrites the query or the tool calls can change retrieval, so measure that path separately. If you cannot show that the relevant document reached the prompt, a prompt fix is unverified, whatever the new answer looks like.

The attribution, in code

The same logic, with the interval you will need when the fixture becomes a real query set. It uses only the standard library and runs on Python 3.12.

import json
from math import sqrt

# A hand-labeled fixture, not a measured run. d2 is the one relevant abstract.
RELEVANT = "d2"

cases = [
    {"name": "A", "retrieved": ["d1", "d3"], "reranked": [], "prompt": [], "correct": False},
    {"name": "B", "retrieved": ["d1", "d2"], "reranked": ["d1"], "prompt": ["d1"], "correct": False},
    {"name": "C", "retrieved": ["d1", "d2"], "reranked": ["d1", "d2"], "prompt": ["d1", "d2"], "correct": False},
    {"name": "D", "retrieved": ["d1", "d2"], "reranked": ["d1", "d2"], "prompt": ["d1"], "correct": False},
    {"name": "E", "retrieved": ["d1", "d2"], "reranked": ["d1", "d2"], "prompt": ["d1", "d2"], "correct": True},
]


def diagnose(case, relevant_id):
    """Name the first stage at which the relevant abstract was lost."""
    if relevant_id is None:
        return "label_missing"
    if case.get("endpoint_error"):
        return "endpoint_unavailable"
    if relevant_id not in case["retrieved"]:
        return "retrieval"
    if relevant_id not in case["reranked"]:
        return "reranking"
    if relevant_id not in case["prompt"]:
        return "prompt_assembly"
    return "none" if case["correct"] else "generation"


def wilson(k, n, z=1.96):
    """95% Wilson interval for k successes in n trials."""
    p = k / n
    center = (p + z * z / (2 * n)) / (1 + z * z / n)
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return center - half, center + half


for case in cases:
    print(json.dumps({"case": case["name"], "failing_stage": diagnose(case, RELEVANT)}))

# Chapter 6 measured recall@3 of 0.733 on 15 paraphrased questions: 11 of 15.
low, high = wilson(11, 15)
print(f"recall@3 = 11/15 = {11 / 15:.3f}, 95% Wilson interval {low:.3f} to {high:.3f}")

Run on Python 3.12.13, it prints:

{"case": "A", "failing_stage": "retrieval"}
{"case": "B", "failing_stage": "reranking"}
{"case": "C", "failing_stage": "generation"}
{"case": "D", "failing_stage": "prompt_assembly"}
{"case": "E", "failing_stage": "none"}
recall@3 = 11/15 = 0.733, 95% Wilson interval 0.480 to 0.891

The first five lines prove only that the diagnostic logic is consistent over data written by hand. They measure no retriever and call no endpoint. The last line is about a real measurement, and it is the reason for the next section.

View full-size diagram

flowchart TD
A["Retrieved d2"] --> B["Reranker kept d2"] --> C["Prompt assembly omitted d2"] --> D["Generation never receives passage"] --> E["Fix assembly; rerank survival was insufficient"]

Put an interval on every stage

Chapter 6 measured an embedding model's recall@3 on fifteen paraphrased questions at 0.733: eleven of fifteen. The Wilson interval above runs from 0.48 to 0.89. The simpler normal interval gives 0.733 ± 1.96 × √(0.733 × 0.267 / 15) = 0.733 ± 1.96 × 0.114 = 0.733 ± 0.224, which reaches 0.96 and understates how little fifteen questions can tell you. The chapter's comparison between rankers still stands, because it was large, but a 5-point change on fifteen questions would be invisible.

Stage attribution needs the same care. With n queries, each query lands in exactly one bucket: retrieval, reranking, prompt assembly, generation or correct. Each bucket's share is a proportion with its own interval. Two practical points follow.

• Conditional rates have smaller denominators: rerank survival is measured only over the queries whose relevant document was retrieved. If 210 of 300 queries retrieve it, survival is estimated from 210, not 300, and its interval is wider.
• Queries with several relevant documents need care: recall@k for such a query is a fraction, not a yes or no. For a binomial interval, use the hit rate: the share of queries with at least one relevant document in the top k. For mean recall@k, resample queries with replacement (a bootstrap) and read the interval from the resampled means.

Size the query set before you run it

The half-width of a 95% interval on a rate p from n queries is 1.96 × √(p(1 − p) / n). At a hit rate near 0.7:

• n = 15: 1.96 × √(0.21 / 15) = 1.96 × 0.118 = 0.23, or ±23 points;
• n = 300: 1.96 × √(0.21 / 300) = 1.96 × 0.0265 = 0.052, or ±5 points.

Three hundred is the size of the SciFact test set below. When you change one stage, say a new embedding model, run both versions on the same queries and compare them with McNemar's test on the queries that flip. It is far more sensitive than comparing two separate intervals. Chapter 16 derives both the Wilson interval and the paired test.

Reranking is its own stage

It is easy to fold reranking into "search" and stop checking it separately. NVIDIA's NeMo Retriever documentation describes text reranking as a distinct step: a model that scores query-passage pairs directly, run after an initial candidate set has been assembled by embedding search (NVIDIA, NeMo Retriever text reranking overview). Embedding similarity is fast and approximate. A cross-encoder reranker scores the query and passage together and may improve their order. It can also make mistakes, which is case B.

Before a diagnosis that calls a hosted reranker, confirm the current model identifier against the vendor's documentation. Reranking model names change, and a retired identifier produces an error that looks like an outage, not a relevance failure. The NVIDIA NIM in Production course builds the embedding and reranking calls and shows how to track those identifiers.

Measured on SciFact

The fixture becomes an experiment on BEIR, the retrieval benchmark introduced by Thakur and colleagues in BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (2021). Its SciFact set has 300 test claims, 5,183 paper abstracts and about 1.1 relevant abstracts per claim. The claims are scientific statements, which suits a research assistant. The run downloads the archive from BEIR's README and checks it against the md5 listed there, 5f7d1de60b170fc8027bb7898e2efca1.

Every stage uses a small open model that runs on a free Colab GPU:

• Retrieval, top 100: BM25, or embeddings from all-MiniLM-L6-v2, the model Chapter 6 measures.
• Reranking to 10: either none, which passes on the retriever's own top 10, or the cross-encoder ms-marco-MiniLM-L-6-v2.
• Prompt assembly: whole abstracts from the top 5, in rank order, until the next one would exceed 1,024 tokens. On average about 2.4 of the 5 fit.
• Generation: Qwen2.5 1.5B is asked which abstract, by ID, gives evidence for or against the claim, or NONE. The answer is correct if it names a relevant ID.

Each claim goes to the first stage that lost all of its relevant abstracts, using the diagnose function above with "any relevant abstract survives" as the test. The code, every claim's lists and a Colab notebook are in profrodai-resources. The run cost nothing and called no API.

300 SciFact claims per pipeline, each assigned to the first stage that lost every relevant abstract. Retrieval keeps the top 100, reranking passes 10 on, and the prompt holds whole abstracts from the top 5 within 1,024 tokens.

BM25BM25 + rerankerMiniLMMiniLM + rerankerMiniLM + reranker, new prompt

View full-size figure

• lost in retrieval
• lost in reranking
• lost in prompt assembly
• lost in generation
• answered correctly

MiniLM + reranker, new prompt: 151 of 300 claims answered correctly (50%, Wilson 45% to 56%). Of the 149 misses, the largest share was lost in generation: 56. When a relevant abstract reached the prompt, the model named it 151 times out of 207.

Three readings, in the order the table supports them.

Generation is the largest loss in every pipeline. With MiniLM and the reranker, a relevant abstract reached the prompt for 207 claims, and the model named it for 107 (52%, with a Wilson interval of 45% to 58%). Most of the rest it answered NONE: across the four pipelines, NONE was the answer to 161 to 177 of the 300 claims. This is case C from the fixture, at scale. Here, changing the generation prompt is the right next move.

The upstream stages still lose about a third. For the same pipeline, 22 claims never retrieved a relevant abstract, the reranker dropped 31 more, and prompt assembly dropped 40 that survived reranking. The last group is case D: 16% of the claims whose evidence was in the top 10 lost it to the token budget, because the relevant abstract ranked below the two or three that fit. No prompt wording reaches any of these 93 claims.

The reranker matters for one retriever and not clearly for the other. Paired on the same claims, adding the cross-encoder to MiniLM fixed 32 claims and broke 9 (McNemar p = 0.0004). Adding it to BM25 fixed 21 and broke 13 (p = 0.23), which is not evidence either way at this size. Without a reranker, BM25 beat MiniLM, 31 claims to 16 (p = 0.04). With one, the two tied at 107, with 4 claims each way. Separate intervals would have shown none of this: 99 and 107 out of 300 have overlapping intervals, while the paired count tells you which claims moved.

Then change the prompt, and only that

Attribution said generation, so the follow-up changes only the generation prompt. It runs on the saved prompts of the MiniLM-and-reranker pipeline, so retrieval, reranking and assembly are identical by construction. The original question asked which abstract "gives evidence for or against the claim". The revised one says an abstract may study the same question without settling it, asks for the most relevant one, and reserves NONE for abstracts on a different topic.

Correct answers went from 107 to 151 of 300 (50%, Wilson 45% to 56%). Paired, 49 claims were fixed and 5 broken, McNemar p = 4 × 10⁻¹⁰. Given the evidence in the prompt, the model now named it 151 times out of 207 (73%). The 93 claims lost upstream did not move, which is the point: the prompt fixed the stage the attribution named and nothing else, and the ceiling for this pipeline is now 207, set by retrieval, reranking and the token budget.

Two caveats belong next to that number. I wrote the revised prompt after seeing the first run, and tested it on the same 300 claims, so the gain is an optimistic estimate; a fair test needs claims the prompt was not written against. And every SciFact test claim has a relevant abstract, so a prompt that almost never answers NONE (4 times, against 161) cannot be punished here. On a collection where some claims have no evidence, the same change would raise false citations, and that cost has to be measured before you ship it.

The Chapter 6 retrieval-evaluation exercise and the Chapter 5 retrieval-and-recall exercise are notebooks that run on Colab, so you can practice the metric on a small corpus before scaling to SciFact.

Anthropic's guidance on evaluating agents makes the general version of this point: useful evaluation separates tasks, environments, trajectories and outcomes, and a single aggregate score cannot stand in for every requirement (Anthropic, demystifying evaluations for agents, 2026-01-09). Stage attribution is a retrieval-specific instance: four criteria instead of one number.

A checkpoint before you touch any prompt

Check the decision

What would prove this diagnosis wrong

The four-way split only holds if the stages are observable in your system. If your library collapses retrieval and reranking into one opaque call, you cannot see "retrieved but not reranked", and the attribution degrades into a guess. If the relevance labels are wrong, every downstream diagnosis inherits the error; BEIR's labels are a benchmark's judgment, not ground truth for your corpus. And if the generator already knows the answer from training, as it likely does for a famous paper, case A can produce a correct answer by coincidence and look like retrieval succeeded. Holding out questions the model cannot answer from memory is part of the design.

Why this matters in a lab

Retrieval and search are research problems at frontier labs, not only product plumbing. OpenAI's current posting for a Research Engineer in retrieval and search asks for "prior experience working with vector databases, search indices, or other data stores for search and retrieval use cases." More broadly, the evaluation postings in the survey of fifteen current lab postings ask for someone who is "thinking twice before trusting a number." An interviewer who hears "we improved the prompt and accuracy went up" will ask two questions. At which stage were the failures? Is the change larger than the interval? Stage attribution with intervals answers both.