Unlocking the Power of Reranking Models in Retrieval-Augmented Generation (RAG) Systems
Applications requiring high-performance information retrieval span a wide range of domains, including search engines, knowledge management systems, AI agents, and AI assistants. These systems demand retrieval processes that are accurate and computationally efficient to deliver precise insights, enhance user experiences, and maintain scalability. Retrieval-augmented generation (RAG) is used to enrich results, but its effectiveness is fundamentally tied to the precision of the underlying retrieval mechanisms.
The operational costs of RAG-based systems are driven by two primary factors: compute resources and the cost of inaccuracies resulting from suboptimal retrieval precision. Addressing these challenges requires optimizing retrieval pipelines without compromising performance. A reranking model can help improve retrieval accuracy and reduce overall expenses. However, despite the potential of reranking models, they have historically been underutilized due to concerns about added complexity and perceived marginal gains in information retrieval workflows.
In this article, we unveil significant performance advancements in the NVIDIA NeMo Retriever reranking model, demonstrating how it redefines the role of computing relevance scores in modern pipelines. Through detailed benchmarks, we’ll highlight the cost-performance trade-offs and showcase flexible configurations that cater to diverse applications, from lightweight implementations to enterprise-grade deployments.
What is a Reranking Model?
A reranking model, often referred to as a reranker or cross-encoder, is a model designed to compute a relevance score between two pieces of text. In the context of RAG, a reranking model evaluates the relevance of a passage to a given query. Unlike approaches that just use an embedding model, which generates independent semantic representations for each passage and relies on heuristic similarity metrics (cosine similarity, for example) to determine relevance, a reranking model directly compares the query-passage pair within the same model. This creates semantic representation one passage at a time, and then uses a heuristic metric to measure relevance.
By analyzing the patterns, context, and shared information between the query and passage simultaneously, reranking models provide a more nuanced and accurate assessment of relevance. This makes cross-encoders more accurate at predicting relevance than using a heuristic score with an embedding model, making them a critical component for high-precision retrieval pipelines.
How can Reranking Models Improve RAG?
The cost of compute to run a large language model (LLM) is considerable higher when compared to using an embedding or reranking model. This cost scales directly with the number of tokens an LLM processes. A RAG system uses a retriever to fetch the top N chunks of relevant information (which can typically range from 3-10), and then employs an LLM to generate an answer based on that information. Increasing the value of N often involves a trade-off between cost and accuracy. A higher N improves the likelihood that the retriever includes the most relevant chunk of information, but it also raises the computational expenses of the LLM step.
Retrievers typically rely on embedding models, but incorporating a reranking model into the pipeline offers three potential benefits:
- Maximize accuracy while reducing the cost of running RAG: By using a reranking model, you can reduce the number of candidates used in the second step, decreasing the cost of the LLM step while maintaining accuracy.
- Maintain accuracy while considerably reducing the cost of running RAG: By using a reranking model, you can reduce the number of candidates used in the second step, reducing the cost of the LLM step while maintaining accuracy.
- Improve accuracy and reduce the cost of running RAG: By using a reranking model, you can increase the number of candidates used in the second step, improving accuracy while reducing the cost of the LLM step.
Reranking Model Stats
With the premise understood, this section dives into the performance benchmarks. There are three numbers that need to be understood to digest the information following:
- N_Base: The number of chunks a RAG pipeline uses without a reranking (Base Case).
- N_Reranked: The number of chunks a RAG pipeline uses with a reranking.
- K: The number of candidates being ranked in Step 2 using a reranking process.
With these three variables, formulate three equations that serve as the basis of all the three scenarios:
Equation 1: N_Reranked = N_Base + (K – 1)
Equation 2: N_Reranked = N_Base + (K – 1)
Equation 3: N_Reranked = N_Base + (K – 1)
These equations demonstrate how the reranking model can be used to optimize the number of chunks used in the pipeline, reducing the cost of running RAG while maintaining or improving accuracy. By leveraging the NVIDIA NeMo Retriever reranking model, organizations can unlock the potential of RAG systems, enhancing the precision and efficiency of their information retrieval workflows.
