Document Similarity Self-Join with MapReduce

Ranieri Baraglia, Gianmarco De Francisci Morales, Claudio Lucchese
2010 2010 IEEE International Conference on Data Mining  
Given a collection of objects, the Similarity Self-Join problem requires to discover all those pairs of objects whose similarity is above a user defined threshold. In this paper we focus on document collections, which are characterized by a sparseness that allows effective pruning strategies. Our contribution is a new parallel algorithm within the MapReduce framework. This work borrows from the state of the art in serial algorithms for similarity join and MapReducebased techniques for
more » ... rity join. The proposed algorithm shows that it is possible to leverage a distributed file system to support communication patterns that do not naturally fit the MapReduce framework. Scalability is achieved by introducing a partitioning strategy able to overcome memory bottlenecks. Experimental evidence on real world data shows that our algorithm outperforms the state of the art by a factor 4.5.
doi:10.1109/icdm.2010.70 dblp:conf/icdm/BaragliaML10 fatcat:ueq5jsum4bewthlveisuwjybna