Stability and scalability for search

1. Scalability is crucial - systems need to be designed with the assumption that query volume, document corpus size, indexing complexity etc. could increase by 10x. What works at one scale may completely break at a higher scale.


2. Sharding the index, either by document or by word, is important to distribute the indexing and querying load across machines.

... See more

Decentralization adds new functionality in other domains, but when it comes to scale, we’re aiming to replicate the global experience Twitter currently offers. Existing decentralized social protocols default to local conversations because it’s a natural fit for a decentralized architecture, but our goal is to make global conversations possible whil... See more

On Scalable Logging Platform

Twitter is actually a host of microservices, including a user service (for listing a user’s timeline), a graph service (for tracking your network), a posting service (for posting new tweets), a profile service (for user profiles), a timeline service (for presenting your timeline), etc.; the architecture to tie all of these together and operate at s... See more

Ensuring availability during peak traffic by maintaining all GPU instance types could lead to prohibitively high costs. To avoid the financial strain of idle instances, we implemented a “standby instances” mechanism. Rather than preparing for the maximum potential load, we maintained a calculated number of standby instances that match the increment... See more