Scalability and efficiency challenges in large-scale web search engines

B. Barla Cambazoglu, Ricardo Baeza-Yates
2014 Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval - SIGIR '14  
The main goals of a web search engine are quality, e ciency, and scalability. In this tutorial, we focus on the last two goals, providing a fairly comprehensive overview of the scalability and e ciency challenges in large-scale web search engines. In particular, the tutorial provides an in-depth architectural overview of a web search engine, mainly focusing on the web crawling, indexing, and query processing components. The scalability and e ciency issues encountered in these components are
more » ... ented at four di↵erent granularities: at the level of a single computer, a cluster of computers, a single data center, and a multi-center search engine. The tutorial also points at open research problems and provides recommendations to researchers who are new to the field.
doi:10.1145/2600428.2602291 dblp:conf/sigir/CambazogluB14 fatcat:ldl4l4nas5hcvcclhvdq5755zu