Stability and scalability for search

Elasticsearch at Twitter

Elasticsearch is a search engine based on the Lucene library. It is a popular open source tool widely used in industry and is known for its distributed nature, speed, scalability, and simple REST APIs.

The Search Infrastructure team builds infrastructure to host search as a service. Since we are such a central infrastructur... See more

The solution: The ingestion service

To meet these unique demands, the Search Infrastructure team implemented the Ingestion Service to gracefully handle Twitter’s traffic trends. The Ingestion Service queues requests from the client service into a single Kafka topic per Elasticsearch cluster. Worker clients then read from this topic and send the req... See more