Building software
sari and
Building software
222. Automating Processes With Software Is HARD
Steven Sinofskyhardcoresoftware.learningbyshipping.comThe Pulse #92: GitHub’s AI-assisted developer workflow vision
Gergely Orosznewsletter.pragmaticengineer.com
The "Painless" data migration
- Another challenge was to incorporate the above changes for the existing data i.e. increasing the number of shards and defining a custom routing key for each of the document. The existing index had close to 33 MN documents at the time of the change
- Given that the routing key and the number of shards needed to changes, a
Improving query performance in ElasticSearch at scale
The formula in Fig 5 is one of the reasons why Elasticsearch does not allow the update of the number of primary shards post index creation. What the default routing guarantees is that the data will be uniformly distributed across the cluster which would reduce query hotspots when retrieving documents
The choice of the routing key in our case was a combination of the unique identifier (UUID) and type of the primary participant of each event in a calendar. This choice was made based on the querying patterns of the consuming application and to maintain a degree of uniqueness.
Improving query performance in ElasticSearch at scale
Linear doesn’t have metrics goals - the founder thinks setting specific metric goals can lead them astray. Product goals should be about how we solve our customers’ problems better.
degree of iteration in software development is inevitable. As soon as one accepts that logic, one must then also accept that software development will be subject to friction (i.e. unexpected inefficiencies). When I plan the future for a piece of software, I expect to encounter ongoing friction, and I try to allow for that. I remain astonished at how many organisations assume that a master plan for their software can be rigidly followed.