The best Google Search engineering explainer just dropped. This was reverse-engineered from 1000s of leaked Google court documents. I know everyone thinks ew Google, search sucks, but no one else has a truly web scale search engine in ~25yrs. Must read for software engineers.... See more

1. Scalability is crucial - systems need to be designed with the assumption that query volume, document corpus size, indexing complexity etc. could increase by 10x. What works at one scale may completely break at a higher scale.


2. Sharding the index, either by document or by word, is important to distribute the indexing and querying load across machines.

... See more