Stability and scalability for search
blog.x.com
Stability and scalability for search
New Amazon Builders' Library article: Resilience lessons from the lunch rush. Fantastic overview of techniques to prevent overload in distributed systems, and highly entertaining real-world stories that draw parallels. https://t.co/mmWU9b1bGS
David Yanacekx.com
How would you build a modern Search Engine using Embeddings and Transformer models? Here is an article on how @wilsonzlin build a web search engine in two months, capable of indexing 280 million pages with 3 billion embeddings.
The Ingest process (simplified):
1. Crawl raw HTML page and normalize to... See more