1. Another challenge was to incorporate the above changes for the existing data i.e. increasing the number of shards and defining a custom routing key for each of the document. The existing index had close to 33 MN documents at the time of the change


2. Given that the routing key and the number of shards needed to changes, a

Once a page has been crawled by a spider, the search engine stores details about that page’s contents and the links both into and out of it in a massive database called an index. This index is highly optimized so that results for any of the hundreds of millions of search requests received every day can be retrieved from it almost instantly.

1. Scalability is crucial - systems need to be designed with the assumption that query volume, document corpus size, indexing complexity etc. could increase by 10x. What works at one scale may completely break at a higher scale.


2. Sharding the index, either by document or by word, is important to distribute the indexing and querying load across machines.