Systematically Improving Your RAG - jxnl.co

1. Synthetic Data for Baseline Metrics¶

Synthetic data can be used to establish baseline precision and recall metrics for your reverse search. The simplest kind of synthetic data is to take existing text chunks, generate synthetic questions, and verify that when we query our synthetic questions, the sourced text chunk is retrieved correctly.

Benefi... See more

• Multiple indices. Splitting the document corpus up into multiple indices and then routing queries based on some criteria. This means that the search is over a much smaller set of documents rather than the entire dataset. Again, it is not always useful, but it can be helpful for certain datasets. The same approach works with the LLMs themselves.


• Cu

... See more

All of the effort spent deliberating on edge cases and long tails stems from the fact that many junior devs are not actually thinking hard enough about what the experiment should be, and what the metrics should look like.

The goal of building out these probabilistic software systems is not a milestone or a feature. Instead, what we're looking for a... See more