Scalability and Efficiency Challenges in Large-Scale Web Search Engines

B. Barla Cambazoglu, Ricardo Baeza-Yates
2015 Proceedings of the Eighth ACM International Conference on Web Search and Data Mining - WSDM '15  
The main goals of a web search engine are quality, e ciency, and scalability. In this tutorial, we focus on the last two goals, providing a fairly comprehensive overview of the scalability and e ciency challenges in large-scale web search engines. In particular, the tutorial provides an in-depth architectural overview of a web search engine, mainly focusing on the web crawling, indexing, and query processing components. The scalability and e ciency issues encountered in these components are
more » ... ented at four di↵erent granularities: at the level of a single computer, a cluster of computers, a single data center, and a multi-center search engine. The tutorial also points at open research problems and provides recommendations to researchers who are new to the field.
doi:10.1145/2684822.2697039 dblp:conf/wsdm/CambazogluB15 fatcat:kxoaikbpu5cbvbpgvljm54h2sy