Highly efficient algorithms for structural clustering of large websites

Lorenzo Blanco, Nilesh Dalvi, Ashwin Machanavajjhala
2011 Proceedings of the 20th international conference on World wide web - WWW '11  
In this paper, we present a highly scalable algorithm for structurally clustering webpages for extraction. We show that, using only the URLs of the webpages and simple content features, it is possible to cluster webpages effectively and efficiently. At the heart of our techniques is a principled framework, based on the principles of information theory, that allows us to effectively leverage the URLs, and combine them with content and structural properties. Using an extensive evaluation over
more » ... ral large full websites, we demonstrate the effectiveness of our techniques, at a scale unattainable by previous techniques.
doi:10.1145/1963405.1963468 dblp:conf/www/BlancoDM11 fatcat:4abvitwunfh33dlpzm57i6xdtq