Introduction
In the realm of intelligent assistants and Retrieval-Augmented Generation (RAG) pipelines, integrating images poses a significant challenge. Kapa.ai, a company specializing in AI assistants, has tackled this by developing an innovative method for indexing images. This approach maximizes the value of images without inflating per-query costs. This article delves into how we developed this solution and why it is crucial for enhancing the responses provided by language models.
Why Images Are Crucial
Images in technical documentation serve two primary purposes:
- Illustrative: They clarify textual instructions. For example, a screenshot might show where the settings icon mentioned in the text is located.
- Load-Bearing: They contain essential information not present in the text, such as a wiring diagram or a specification table.
In both cases, images significantly enrich the comprehension and actionability of responses. A study conducted across three customer projects and two models showed a statistically significant preference for answers including image context (McNemar's test, p < 0.05).
Challenges of Integrating Images
The initial solution often considered is sending images to a vision-capable model at every query. However, this approach is not scalable for several reasons:
- Economic Cost: Raw images added 27% to per-query cost on GPT and 51% on Claude.
- Performance: Sending images with every query introduces latencies and costs that are hard to justify.
Our Solution: Image Indexing
To circumvent these challenges, we adopted a different approach:
- One-Time Indexing: Each image is described once at indexing time using an affordable vision model.
- Storing Descriptions: Textual descriptions of images are stored and retrieved alongside other text chunks.
This method reduces the per-query cost to a mere 1% to 6% overhead compared to a text-only solution.
Impact on Answer Quality
The results speak for themselves. With this approach, AI assistants deliver responses that are not only more accurate but also easier for users to act upon. This leads to a reduction in support tickets and an improved user experience.
Conclusion
Indexing images for RAG is an elegant solution that optimizes resources while enriching the user experience. For companies looking to enhance their technical support systems, this approach offers excellent value for money.
Let's discuss your project in 15 minutes.