Systematically Improving Your RAG - jxnl.co

Continuously Monitor and Experiment¶

Continuously monitor your system's performance and run experiments to test improvements.

• Set up monitoring and logging to track system performance over time


• Regularly review the data to identify trends and issues


• Design and run experiments to test potential improvements


• Measure the impact of changes on precision, re

... See more

Utilize both full-text search and vector search (embeddings) for retrieving relevant documents. Ideally, you should use a single database system to avoid synchronization issues.

• Implement both full-text search and vector search


• Test the performance of each method on your specific use case


• Consider using a single database system to store both types of

... See more

I think the biggest mistake around improving the system is that most people are spending too much time on the actual synthesis without actually understanding whether or not the data is being retrieved correctly. To avoid this:

• Create synthetic questions for each text chunk in your database


• Use these questions to test your retrieval system


• Calculate p

... See more

Analyze user queries and feedback to identify topic clusters, capabilities, and areas of user dissatisfaction. This will help you prioritize improvements.

Why should we do this? Let me give you an example. I once worked with a company that provided a technical documentation search system. By clustering user queries, we identified two main issues:

1. Top

... See more

Implementing clear user feedback systems (e.g., thumbs up/down) is essential for gathering data on your system's performance and identifying areas for improvement.

• Add user feedback mechanisms to your application


• Make sure the copy for these mechanisms clearly describes what you're measuring


• Ask specific questions like "Did we answer the question corr

... See more

Balance Latency and Performance¶

Finally, make informed decisions about trade-offs between system latency and search performance based on your specific use case and user requirements.

• Understand the latency and performance requirements for your application


• Measure the impact of different configurations on latency and performance


• Make trade-offs based

... See more

Ensuring relevant metadata (e.g., date ranges, file names, ownership) is extracted and searchable is crucial for improving search results.

• Extract relevant metadata from your documents


• Include metadata in your search indexes


• Use query understanding to extract metadata from user queries


• Expand search queries with relevant metadata to improve results

For... See more