Crawling the Web [chapter]

Gautam Pant, Padmini Srinivasan, Filippo Menczer
2004 Web Dynamics  
The large size and the dynamic nature of the Web make it necessary to continually maintain Web based information retrieval systems. Crawlers facilitate this process by following hyperlinks in Web pages to automatically download new and updated Web pages. While some systems rely on crawlers that exhaustively crawl the Web, others incorporate "focus" within their crawlers to harvest application-or topic-specific collections. In this chapter we discuss the basic issues related to developing an
more » ... astructure for crawlers. This is followed by a review of several topical crawling algorithms, and evaluation metrics that may be used to judge their performance. Given that many innovative applications of Web crawling are still being invented, we briefly discuss some that have already been developed.
doi:10.1007/978-3-662-10874-1_7 fatcat:wz2wsoi3d5h2vebem2rof2jv2e