Building software
by sari and · updated 2mo ago
Building software
- The "Painless" data migration
- Another challenge was to incorporate the above changes for the existing data i.e. increasing the number of shards and defining a custom routing key for each of the document. The existing index had close to 33 MN documents at the time of the change
- Given that the routing key and the number of shards needed to changes, a
from Improving query performance in ElasticSearch at scale
Andrés added 7mo ago
- Pros
- The query speeds improve as, instead of instructing Elasticsearch to Get all the data matching... , you say Go there and get all the data matching... ( Fig 6 ). If you are querying data with multiple unique routing keys, this guarantees better or equivalent query performance as that of the default routing strategy (reducing the scatter and gat
from Improving query performance in ElasticSearch at scale
Andrés added 7mo ago
- The choice of the routing key in our case was a combination of the unique identifier (UUID) and type of the primary participant of each event in a calendar. This choice was made based on the querying patterns of the consuming application and to maintain a degree of uniqueness.
from Improving query performance in ElasticSearch at scale
Andrés added 7mo ago
- The number of partitions is fixed at the time of the index creation. Each partition is an indivisible chunk of data which can grow in size but not be split into smaller chunks. As the data grows, more nodes may be added to the cluster, but the number of partitions which eventually move to the newer nodes are limited (Fig 3)
- In some sense, as the da
from Improving query performance in ElasticSearch at scale
Andrés added 7mo ago
- Software problems tend to evolve over time into philosophy problems, a point that's come up in a few Diff pieces, like the Antithesis writeup and Asana's ontology-as-a-service ($). In this particular category, it's the philosophy of work and vocation: a bond trader, accountant, or social media manager doesn't necessarily have to spend much time nav... See more
from Building for Power Users by Byrne Hobart
sari added 7mo ago
- In Elasticsearch, these partitions are known as shards . Each node of the Elasticsearch cluster can serve queries independently for the partition it manages, thereby increasing query throughput.
from Improving query performance in ElasticSearch at scale
Andrés added 7mo ago
It does everything I want a word processing program to do and it doesn't do anything else. I don't want any help. I hate some of these modern systems where you type up a lowercase letter and it becomes a capital. I don't want a capital, if I'd wanted a capital, I would have typed the capital.
George R.R. Martinfrom The beauty of finished software
sari added 10mo ago
In a world where constant change is the norm, finished software —a software you can use forever with no unneeded change —provides a breath of fresh air. It’s a reminder that reliability, consistency, and user satisfaction can coexist in the realm of software development.
Ideas related to this collection