Back to blog

More retrieval, better answers? Lessons from a real RAG Fusion deployment

Research by Medrano, Verma and Chhabra shows that in a production-like RAG system, RAG Fusion raises recall, but after re-ranking and context truncation much of the gain disappears and precision even dips slightly.

RAG (retrieval-augmented generation) has become the standard pattern for enterprise assistants: the system first finds relevant documents in the organization's knowledge base, and then the language model answers based on them. One popular idea for improving the retrieval part is RAG Fusion: several rewrites are generated from one question, each is searched, and the results are merged so the chance of finding the right document (recall) goes up.

The paper "Scaling Retrieval Augmented Generation with RAG Fusion: Lessons from an Industry Deployment" by Luigi Medrano, Arush Verma and Mukul Chhabra tests this idea not in the lab but in a production-like RAG pipeline over an organization's knowledge base — under real constraints: fixed retrieval depth, a limited re-ranking budget and a latency ceiling.

The main finding is that RAG Fusion does raise raw recall, but the gain is largely neutralized after the re-ranking and truncation stage. As a result, in end-to-end evaluation the Fusion-based variants were weaker than the simple single-query baseline: Hit@10 fell from 0.51 to 0.48 in several configurations while latency increased.

Why? The exact explanation is in the paper itself, but one reasonable reading is this: the language model finally sees only a few chunks in its context window. If the re-ranker and the context cap choose the same few top chunks, the extra candidates Fusion found either never enter the context or add more noise to the ranking stage. An improvement at an intermediate stage does not necessarily translate into a better final answer.

The authors conclude that recall-focused Fusion techniques show diminishing returns once real re-ranking and context-budget constraints are applied, and that retrieval quality, system efficiency and downstream impact should be evaluated together, not retrieval improvement alone.

The practical lesson for teams building document-based assistants: (1) test each technique on your own data and real questions with an end-to-end metric, not just recall; (2) before reaching for complex techniques, get the basics right — proper chunking, metadata, hybrid search and a good re-ranker; (3) count latency and cost as part of the metric; (4) build a fixed evaluation set of real user questions so every change is comparable.

A necessary caution: these results come from one deployment and one knowledge base and do not necessarily hold for every organization. The paper's value is above all in reminding us that in RAG, as in any other system, evaluation must come before optimization.