Stability and scalability for search

1. Scalability is crucial - systems need to be designed with the assumption that query volume, document corpus size, indexing complexity etc. could increase by 10x. What works at one scale may completely break at a higher scale.


2. Sharding the index, either by document or by word, is important to distribute the indexing and querying load across machines.

... See more

Twitter is actually a host of microservices, including a user service (for listing a user’s timeline), a graph service (for tracking your network), a posting service (for posting new tweets), a profile service (for user profiles), a timeline service (for presenting your timeline), etc.; the architecture to tie all of these together and operate at s... See more