Introduction
In 2026, the optimization of Retrieval-Augmented Generation (RAG) context is a hot topic among conversational agents. As the complexity and size of knowledge bases continue to grow, streamlining processes to maximize efficiency while minimizing costs becomes imperative. At Kapa, we have developed an innovative method to reduce the context used by our models while maintaining remarkable answer accuracy.
The Challenge of RAG Context
When building intelligent assistants capable of answering complex questions, developers need quick and precise access to relevant information. The RAG process relies on two main steps: a retriever that identifies relevant chunks of documentation, and a generator, usually a large language model, that formulates the answer. However, most of the retrieved chunks are unnecessary, leading to unnecessary costs.
The Solution: A Pruning Step
To solve this problem, we introduced a pruning step between the retriever and the generator. A small, inexpensive language model reads the question and the retrieved chunks, discarding those that are not necessary before the main model sees them. This method has reduced about 68% of the context while maintaining 96% recall, cutting the cost of a query by a third.
Why Simple Rerank Doesn't Work
An obvious solution would have been to expose rerank scores and allow users to cut based on these scores. However, this approach fails for two reasons: rerank scores are not absolute measurements but preference orders and are not calibrated across queries. This means users cannot simply set a fixed threshold.
Economic Impact
In our system, retrieved chunks account for about two-thirds of the total cost of a query. Each fewer chunk reduces the cost by about 4%. Thus, by optimizing the context, we allow agents to have more resources for other essential tasks, such as integrating conversation histories or system prompts.
Conclusion
Optimizing RAG context is not just about cost reduction but also about improving the overall performance of AI systems. By incorporating an effective pruning step, we have maintained a high level of recall while optimizing resources. For companies looking to maximize the efficiency of their conversational agents, this approach offers a promising path.
Let's discuss your project in 15 minutes.