New day, new RAG research! We've already dedicated many articles to the Retrieval-Augmented Generation (RAG) approach and its various modifications, but honestly, it's a topic that never gets old. As one of the most popular methods for enhancing LLMs with external knowledge and retrieving the information you actually need, RAG is continually evolving. Researchers are actively working on modifications to address its limitations, such as long processing times and difficulties with handling extended contexts. But with so much research out there, it can be challenging to know what to focus on. That's where we come in. Speculative RAG is one of the latest approaches, aiming to balance efficiency and effectiveness by using two types of language models in its architecture. The researchers behind it have come up with an intriguing idea: leveraging a small language model within a larger framework. Let’s explore how they arrived at this concept of Speculative RAG.
In today’s episode, we will cover:
Overview of existing RAG systems and their limitations
Here comes Speculative RAG
How does Speculative RAG work?
How good is Speculative RAG?
Where Speculative RAG excels
Not without limitations…
Conclusion
Bonus: Resources
What Are the Limitations of Standard RAG?
As you probably know, RAG systems combine large language models (LLMs) with external information sources to answer queries. These systems aim to enhance the accuracy and relevance of responses by incorporating data retrieved from external databases.
However, existing approaches have limitations:
Original RAG incorporates all retrieved documents directly into the prompt, leading to increased length and slower response times. In other words, it struggles with long context window.
LongRAG deals with the long context window, but struggles with the limitations of current long embedding models, the inefficiencies of black-box LLMs for handling extended inputs, and its reliance on Wikipedia-specific grouping methods, which limit its generalizability.
Graph RAG, a popular method from one of our previous episodes, organizes data into a graph structure, representing text data and its interrelations. Graph-based models may struggle with non-stationary data where the relationships between variables change over time.
Self-Reflective RAG requires specialized instruction-tuning of the general-purpose LMs to generate specific tags for self-reflection. However, it needs additional tuning, which can be complex and need a lot of resources.
Corrective RAG uses an external retrieval evaluator to refine the quality of retrieved documents. It focuses solely on improving the contextual information. The problem is that it doesn’t enhance the model's reasoning capabilities.
Image credit: Speculative RAG paper
These methods often struggle with balancing efficiency and effectiveness. It’s difficult for them to do all at the same time – handling long contexts and ensuring diverse perspectives while maintaining speed and accuracy. For a broader map of where RAG is heading, see 20 advanced RAG types in 2026 — from memory systems to security-focused retrieval.
What Is Speculative RAG?
The researchers from the University of California, San Diego, and Google came up with a thought: what if a smaller, specialized language model could efficiently draft multiple answers from different document subsets, and then a larger, generalist language model could simply verify these drafts? How will it improve RAG approach?
Their innovative method, formalized in the paper Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting and published last July, demonstrates that this division of labor significantly improves both accuracy and speed in RAG tasks. By delegating the initial drafting to a specialized LM, they reduce the computational burden on the larger model, allowing it to focus solely on verification. This streamlined process not only accelerates response generation but also enhances the quality of the final output, offering a more efficient solution to knowledge-intensive tasks in natural language processing.
This research clearly demonstrates how small language models (SLMs), which are becoming one of the focus of AI industry, can be applied to larger frameworks to enhance their capabilities. In collaboration we trust!
How does Speculative RAG work?
Speculative RAG introduces two additional components to the original architecture:
RAG Drafter (Smaller, Specialized Model): Generates multiple drafts of potential answers along with rationales. Its inputs are distinct subsets of retrieved documents while on the output Draft answers and rationales for each subset of documents.
RAG Verifier, or Generalist LM (Larger, General-Purpose Model): Evaluates and verifies the drafts generated by the Drafter. It chooses the best possible answer.
Image credit: Speculative RAG paper
Let’s walk step-by-step →
→ Question and document retrieval: When a question is asked, the system first retrieves several documents from an external database that might contain relevant information.
→ Clustering documents: The retrieved documents are grouped into clusters based on their content. Each cluster represents a different perspective or aspect related to the question.
→ Creating subsets: From each cluster, one document is selected to create a subset of documents. These subsets ensure that the system has a diverse range of information without being repetitive.
→ Drafting answers: The smaller, specialized model, called the RAG Drafter, takes each subset of documents and generates a draft answer. It also explains why this answer makes sense based on the documents.
→ Parallel processing: Multiple drafts are created in parallel, each from a different subset of documents. So this approach considers various perspectives and speeds up the whole process.
→ Verification: The larger, general-purpose model, which is called RAG Verifier, evaluates these drafts. It checks the quality and accuracy of each draft without needing to process all the original documents again. It uses its pre-existing knowledge and the provided rationales to score each draft on how well it answers the question.
→ Selecting the best answer and giving the final output: The system selects the draft with the highest score as the final answer. It is presented to the user as both accurate and produced efficiently.
This approach helps to solve the problem of handling long-context by creating shorter, diverse drafts from subsets of documents, reducing the input length. It is also easily scalable as multiple RAG Drafter instances can be run in parallel without significantly increasing the overall processing time. This makes Speculative RAG adaptable to a wide range of applications, from simple queries to more complex, knowledge-intensive tasks.
Speculative RAG Benchmarks: Accuracy and Latency Results
This study exemplifies how Small Language Models (SLMs) are integrated into larger frameworks using model orchestration. SLMs, designed for exceptional reasoning, are leveraged to produce multiple RAG drafts in parallel, optimizing token count and reducing costs. By injecting relevant knowledge at inference, these specialized models achieve up to a 12.97% accuracy improvement and a 51% latency reduction without requiring fine-tuning. The best performance was achieved with the Mistral 7B SLM as the drafter and Mixtral 8x7B as the verifier.
Here are the key results of the performance of Speculative RAG on 4 benchmarks:
Latency reductions:
Image credit: Original paper
TriviaQA: Reduced latency by up to 23.41%.
MuSiQue: Reduced latency by up to 17.28%.
PubHealth: Reduced latency by up to 51.25%.
ARC-Challenge: Reduced latency by up to 26.73%.
Accuracy improvements:
Image credit: Original paper
TriviaQA: 74.24%, surpassing the most competitive baseline by 0.33%.
MuSiQue: 31.57%, improving over the best baseline by 2.15%.
PubHealth: 76.60%, significantly outperforming the best baseline by 12.97%.
ARC-Challenge: 80.55%, bettering the highest baseline by 2.14%.
Where Speculative RAG Excels: Best Use Cases
Speculative RAG really hits its stride in situations where you need quick, accurate answers to tough questions that pull from a lot of different sources. Good example would be a virtual assistant that helps doctors with medical queries. A doctor might ask about the latest treatment guidelines for a specific condition. The system needs to pull in the most relevant and up-to-date information from various medical databases, synthesize it, and deliver a precise answer quickly. It is perfect here because it can generate multiple draft responses based on different pieces of evidence, then have a larger model pick the best, most accurate one, all in a fraction of the time it would normally take.
Another great use case could be in legal research, where a lawyer needs to quickly understand how a new law might impact their case. The system can gather relevant legal texts, analyze them from different angles, and then provide a well-rounded, accurate summary. Because this approach balances speed with accuracy by offloading the heavy lifting to a smaller model, it’s also ideal when you’re working with limited computing power but still need top-notch results.
So, whether it’s a real-time application like a chatbot or a research tool where every second counts, this system is designed to deliver the goods efficiently.
Limitations of Speculative RAG
Speculative RAG offers significant improvements over traditional RAG systems, but we can’t avoid its several limitations:
Increased Complexity: Requires training an additional specialized drafter model, adding complexity compared to standard these systems.
Dependency on Rationale Quality: The effectiveness of the system heavily relies on the quality of the rationales generated by the drafter.
Performance-Latency Trade-off: While it reduces latency, the parallel processing of drafts requires more computational resources.
Verification Overhead: The additional verification step by the generalist LM can become a bottleneck in low-latency scenarios.
Risk of Overfitting: Instruction-tuning the specialized drafter could lead to overfitting or reduced generalizability.
You might want to avoid using it in scenarios where the input data is simple or straightforward, and doesn't require complex reasoning or the integration of multiple perspectives. For instance, if you’re dealing with tasks that involve basic fact retrieval or simple question-answering, a standard model or even a simpler model might be more efficient and easier to implement.
Additionally, if ultra-low latency is a must and you can't afford the extra processing step of draft verification, Speculative RAG might not be the best fit.
It's also less suitable for situations where there's a risk of overfitting during model training, especially if you're working with limited training data or a highly specialized domain.
Conclusion
Speculative RAG is a great example of combing small and large language models. As for the practical matter: there are no tools suitable for all. Speculative RAG excels at handling complex, knowledge-intensive tasks with improved efficiency and accuracy, but it may not be the best choice for simpler tasks or scenarios where ultra-low latency is crucial. As with any AI system, it's essential to assess the specific needs and constraints of your application before choosing the right approach. Speculative RAG is a promising advancement in this approach, offering a novel way to balance speed, accuracy, and resource use. However, it's vital to weigh its benefits against its limitations to determine if it’s the right fit for your use case.
Resources (links to papers)
Original RAG models combine a pre-trained language model (parametric memory) with an external knowledge source (non-parametric memory), like a dense vector index of documents such as Wikipedia. RAG retrieves relevant information from this source during generation, enhancing the model's accuracy and specificity.
Graph RAG approach presented by Microsoft organizes data into a graph structure, representing text data and its interrelations. Graph RAG is a valuable addition to RAG systems to handle query-focused summarization at scale.
LongRAG is an improved version of RAG model, which processes larger text units (4,000 tokens instead of 100 words), reducing the number of units to search through. This "long retriever" and "long reader" approach enhances accuracy and performance in extracting answers from large texts without extra training.
Self-RAG (Self-Reflective approach) allows the model to retrieve and reflect on information only when needed. It outperforms other models like ChatGPT in tasks requiring reasoning and fact-checking.
Corrective RAG (CRAG) uses an external retrieval evaluator to refine the quality of retrieved documents. It selectively focuses on key information, enhancing the accuracy and robustness of generated content.
EfficientRAG efficiently handles multi-hop questions by generating new queries without needing LLMs at each step and filtering out irrelevant information.
Golden-Retriever is a RAG model that uses reflection-based question augmentation to handle domain-specific jargon and context in industrial knowledge bases, ensuring the retrieval of the most relevant documents.
Adaptive RAG for conversational systems instead of always retrieving external knowledge, assesses the conversation context and decides if RAG is necessary. This approach improves response quality by only using RAG when beneficial, leading to more accurate and confident answers.
Modular RAG is an advanced framework that breaks down complex RAG systems into independent modules and specialized components. Unlike traditional RAG's simple "retrieve-then-generate" process, Modular RAG offers flexible and customizable configurations like routing, scheduling, and combining processes.
Speculative RAG combines two types of LMs: a smaller, specialized LM for producing multiple drafts in parallel, and a larger generalist LM that verifies these drafts to find the best answer. It enhances both effectiveness and speed of the system.
RankRAG is a framework that trains the model to both rank relevant contexts and use it to answer questions. It excels at knowledge-intensive tasks.
Multi-Head RAG uses different parts of the model’s attention mechanism to capture various aspects of a query, making it easier to find and use relevant information. It improves retrieval accuracy, especially for complex queries.
How did you like it?
Share this article with three friends and get a 1-month subscription for free! 🤍
