Improving query performance in ElasticSearch at scale
blogs.halodoc.io
Saved by Andrés
Improving query performance in ElasticSearch at scale
Andrés added
Database Partitioning is when you split the data inside the database into smaller "databases". One partition holds a part of your data. All partitions together form the entire database.
Partitioning is often used with replication, which means each partition's data is copied onto multiple nodes. This way, even though each record is in only one partition, it can be stored on different nodes for fault tolerance. A node may hold more than one partition.
When using a leader-follower replication model, the mix of partitioning and replication might look like this:
Eventually their database became a ticking time bomb ready to explode at any moment. The first instinct might be to beef up their Postgres instance, like me trying to get big at the gym. But there are physical limits to how much you can scale a single machine before costs increase exponentially. Moreover, query performance and maintenance processes
... See moreKiki's Bytes • How Notion Scaled to 100 Million Users Without Their Database Exploding
- Multiple indices. Splitting the document corpus up into multiple indices and then routing queries based on some criteria. This means that the search is over a much smaller set of documents rather than the entire dataset. Again, it is not always useful, but it can be helpful for certain datasets. The same approach works with the LLMs themselves.
- Cu
Matt Rickard • Improving RAG: Strategies
Nicolay Gerold added
Peter Hagen and added
- Scalability is crucial - systems need to be designed with the assumption that query volume, document corpus size, indexing complexity etc. could increase by 10x. What works at one scale may completely break at a higher scale.
- Sharding the index, either by document or by word, is important to distribute the indexing and querying load across machines.
Claude
Nicolay Gerold added
Tradeoffs can increase the complexity of your architecture and require load testing to ensure that a measurable benefit is obtained.