Crawling the web for structured documents

Julián Urbano, Juan Loréns, Yorgos Andreadakis, Mónica Marrero
2010 Proceedings of the 19th ACM international conference on Information and knowledge management - CIKM '10  
Structured Information Retrieval is gaining a lot of interest in recent years, as this kind of information is becoming an invaluable asset for professional communities such as Software Engineering. Most of the research has focused on XML documents, with initiatives like INEX to bring together and evaluate new techniques focused on structured information. Despite the use of XML documents is the immediate choice, the Web is filled with several other types of structured information, which account
more » ... or millions of other documents. These documents may be collected directly using standard Web search engines like Google and Yahoo, or following specific search patterns in online repositories like SourceForge. This demo describes a distributed and focused web crawler for any kind of structured documents, and we show with it how to exploit general-purpose resources to gather large amounts of real-world structured documents off the Web. This kind of tool could help building large test collections of other types of documents, such as Java source code for softwareoriented search engines or RDF for semantic searching.
doi:10.1145/1871437.1871773 dblp:conf/cikm/UrbanoLAM10 fatcat:3gqugbw6pventfkfeo5di5fhbu