Your RAG Is Not Broken, Your Retrieval Is

A support bot tells a customer the refund window is 14 days. It is 30. The document saying 30 is sitting right there in the vector store, indexed, chunked, embedded. The model never saw it.

This is the failure mode I get called in for most often, and the conversation almost always starts in the wrong place. The team has swapped models, rewritten the system prompt twice, turned the temperature down to zero, and added a line that says "only answer from the provided context." None of it helped, because none of it touched the actual bug. The model did its job. It faithfully summarized the chunks it was handed. Those chunks were wrong.

When RAG produces a bad answer, retrieval is the culprit far more often than generation. The generation step is the visible one, so it absorbs the blame. Retrieval fails quietly, upstream, and by the time you see the output there is nothing in it that says "the right chunk was never here."

The business problem: confidently wrong is worse than blank

A RAG system that says "I don't know" is annoying. A RAG system that says "14 days" with a citation is dangerous, because it looks correct. Users trust it, act on it, and you find out from a complaint.

The cost lands in a few predictable places. Support deflection targets get missed, because every answer needs a human to double check it, which was the entire thing you were trying to avoid. Trust drops fast and comes back slowly, so one bad month of answers can push usage back to the search box for a quarter. And engineering time drains into prompt tuning that cannot possibly fix a retrieval bug.

The good news is that this is one of the cheaper problems in the LLM stack to fix. You are not retraining anything. You are changing how candidates get selected and ordered before they reach the prompt, and the wins are large and measurable.

Why single-channel retrieval misses

Most RAG systems ship with one retriever: cosine similarity over dense embeddings. It works well enough in the demo and then quietly falls over in production, because dense vectors are good at meaning and bad at specifics.

Ask "how do I fix ERR_CONN_4021" and a dense retriever will happily return chunks about connection errors in general. It understands the shape of what you want. But ERR_CONN_4021 as a token carries almost no semantic weight, so the one page that actually names the code does not stand out from twenty pages that discuss connection problems. Product codes, SKUs, function names, version numbers, and people's names all have this property. They mean nothing in embedding space and everything to the user.

Keyword search has the mirror problem. BM25 nails ERR_CONN_4021 instantly, then whiffs on "my dashboard is stuck loading" when the docs say "the console fails to render," because not a single term overlaps.

Here is the part that surprises people: on most business corpora, plain BM25 is not the weak baseline you assume it is. Benchmarked against strong commercial embedding models on text and table documents, BM25 holds its own and beats dense retrieval on several metrics. If you replaced keyword search with vectors and called it an upgrade, you may have traded one set of failures for another rather than fixing anything.

You do not have to choose. Run both, then merge.

Pattern one: hybrid search with reciprocal rank fusion

The instinct is to normalize both scores and blend them with a weight. Resist that. BM25 scores are unbounded and corpus dependent, cosine similarities sit in their own range, and any normalization you invent will need retuning every time the corpus shifts.

Reciprocal Rank Fusion sidesteps the whole problem by throwing the scores away and keeping only the ranks. A document scores 1 / (k + rank) in each list, and you sum across lists. No calibration, no tuning, and it is remarkably hard to beat.

from collections import defaultdict

def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    """Fuse ranked doc-id lists. Only order matters, never the raw scores.

    k dampens the top of each list so one retriever's #1 can't dominate the
    fused result on its own. 60 is the value from the original RRF paper and
    is a fine default; there is rarely a reason to tune it first.
    """
    scores: dict[str, float] = defaultdict(float)
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] += 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda pair: pair[1], reverse=True)


def hybrid_search(query: str, limit: int = 50) -> list[str]:
    # Each channel goes deep. Fusion is only useful if the right doc is
    # somewhere in at least one list, so don't fetch top-5 here and hope.
    dense_hits = vector_store.search(embed(query), top_k=100)   # semantic match
    sparse_hits = bm25_index.search(query, top_k=100)           # exact terms

    fused = reciprocal_rank_fusion([
        [hit.doc_id for hit in dense_hits],
        [hit.doc_id for hit in sparse_hits],
    ])
    return [doc_id for doc_id, _ in fused[:limit]]

The detail that matters most is the depth of each channel. Fetching the top 100 per retriever and fusing down to 50 is doing real work. Fetching the top 5 from each is not, because a document ranked 30th by BM25 and absent from the dense list is exactly the one RRF exists to rescue, and you never gave it the chance.

Hybrid retrieval is the floor, not the ceiling. If you ship nothing else from this post, ship this. It is a meaningful recall gain over dense-only for a few milliseconds of added latency, and it needs no model.

Pattern two: rerank, because recall is not precision

Hybrid search gets the right document into your top 50. That is not the same as getting it into the prompt.

You can only afford to put a handful of chunks in front of the model, and stuffing more in makes things worse rather than better. Long contexts dilute attention, cost more, and bury the answer in the middle where models are weakest at finding it. So the question becomes: of these 50 candidates, which 3 are actually right?

Your retrievers cannot answer that well, and it is structural. A bi-encoder embeds the query and the document separately, before it has ever seen the pairing, then compares two frozen vectors. It is fast because the document vectors are precomputed, and it is imprecise for exactly the same reason.

A cross-encoder reads the query and the document together and scores the pair directly. Every token of the query can attend to every token of the document, so it catches the difference between a page that mentions refunds and the page that states the refund window. This is far too slow to run over a whole corpus, and perfectly affordable over 50 candidates.

That asymmetry is the whole design. Retrieve wide and cheap, then rank narrow and expensive.

from sentence_transformers import CrossEncoder

# Small, fast, runs on CPU. Load once at startup, never per request.
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def retrieve(query: str, top_n: int = 3) -> list[Doc]:
    candidate_ids = hybrid_search(query, limit=50)
    candidates = doc_store.get_many(candidate_ids)

    # The cross-encoder sees query and doc together, which is the point.
    pairs = [(query, doc.text) for doc in candidates]
    scores = reranker.predict(pairs)

    ranked = sorted(zip(candidates, scores), key=lambda pair: pair[1], reverse=True)

    # Drop weak matches instead of always returning top_n. If nothing clears
    # the bar, returning nothing beats handing the model irrelevant context
    # and letting it improvise an answer.
    return [doc for doc, score in ranked[:top_n] if score > 0.3]

Two things in there earn their keep. The first is that reranking runs on the fused candidate list, so it inherits the recall that hybrid search bought you. Reranking a dense-only top 50 just reorders the same blind spots.

The second is the threshold. Most pipelines unconditionally return top_n, which means a question your corpus cannot answer still produces three chunks of loosely related text, and the model dutifully constructs something plausible out of them. That is how confident hallucinations get manufactured. Let retrieval return nothing and let the answer be "I don't have that." Users forgive a system that admits gaps and do not forgive one that invents them.

Measure retrieval on its own

Here is the trap. You upgrade retrieval, spot-check five questions, they look better, you ship. Two weeks later something regresses and you have no idea whether it was retrieval, the prompt, or the model, because you never measured the pieces separately.

Retrieval has a real advantage over generation: it is cheap to score. There is no judge model and no rubric. Either the right document came back or it did not.

def recall_at_k(eval_set: list[dict], k: int = 3) -> float:
    """eval_set entries: {"query": str, "relevant_ids": set[str]}

    Recall@k asks one question: did we get the right doc into the k chunks
    that reach the prompt? If this is 0.6, no prompt engineering saves you,
    because 40% of the time the answer was never in the context.
    """
    hits = 0
    for case in eval_set:
        retrieved = {doc.id for doc in retrieve(case["query"], top_n=k)}
        if retrieved & case["relevant_ids"]:
            hits += 1
    return hits / len(eval_set)


if __name__ == "__main__":
    # Run each stage over the same set to see where the gain actually came from.
    print("dense only  ", recall_at_k(EVAL_SET))
    print("hybrid      ", recall_at_k(EVAL_SET))
    print("hybrid+rerank", recall_at_k(EVAL_SET))

Thirty to fifty real questions with their known-correct documents is enough to start, and you build the set from your own traffic. Every question your system got wrong in production becomes a case, which sharpens the suite exactly where you break. This is the same logic behind putting LLM evals in CI, applied one layer down: score the stage in isolation so a regression tells you where it happened.

Run the three configurations above and you get an honest answer about what your changes bought. Sometimes hybrid alone closes the gap and the reranker is not worth the latency. You cannot know that by guessing.

The tradeoffs nobody mentions

Reranking costs latency, and the knob is candidate count. Rerank 50 candidates and expect roughly 50 to 200ms on a hosted reranker. Going from 100 candidates to 50 roughly halves that, and the recall you lose is usually small. Measure it on your own eval set rather than accepting a default someone picked in a tutorial.

Two indexes means two things to keep in sync. Your BM25 index and your vector store now both need updating when a document changes. A doc that exists in one and not the other produces retrieval behavior that is genuinely painful to debug. Write to both in the same job, and add a count check that alerts on drift.

Reranking cannot fix chunking. If your chunks split a table from its header, or cut a procedure in half, the reranker will faithfully find you the best of a bad set. No ranking stage recovers information that chunking destroyed. When recall stays stubbornly low after hybrid and reranking, stop tuning the retrieval stack and go read your chunks.

Retrieval is not free context. Pulling more and better chunks still means paying for tokens, and in a long-running agent those chunks accumulate across turns. That is where retrieval quality collides with context management: better retrieval means putting fewer, denser chunks in the window, not more of them.

The takeaway

When a RAG answer is wrong, log the retrieved chunks before touching the prompt. If the answer was not in there, no amount of prompt engineering was ever going to help, and you would have spent a week finding that out the slow way.

The fix is two stages and both are cheap. Run BM25 and vector search together and fuse with RRF, which needs no model and no tuning. Then rerank the fused candidates with a cross-encoder so the few chunks that reach the prompt are the right ones. Score it with recall@k on questions from your own traffic, so you know what each stage bought instead of assuming.

Most teams reach for a better model when their real problem is that the answer never made it into the context window. Fix retrieval first. The model was probably fine all along.

If your RAG system is confidently wrong often enough to be a problem, and you are not sure whether it is retrieval, chunking, or the prompt, book a consultation call and we will find out which one it actually is.