A parallel graph partitioning algorithm to speed up the large-scale distributed graph mining

ZengFeng Zeng, Bin Wu, Haoyu Wang
2012 Proceedings of the 1st International Workshop on Big Data, Streams and Heterogeneous Source Mining Algorithms, Systems, Programming Models and Applications - BigMine '12  
For the large-scale distributed graph mining, the graph is distributed over a cluster of nodes, thus performing computations on the distributed graph is expensive when large amount of data have to be moved between different computers. A good partitioning of distributed graph is needed to reduce the communication between computers and scale a system up. Existing graph partitioning algorithms incur high computation and communication cost when applied on large distributed graphs. A efficient and
more » ... alable partitioning algorithm is crucial for large-scale distributed graph mining. In this paper, we propose a novel parallel multi-level stepwise partitioning algorithm. The algorithm first efficiently aggregates the large graph into a small weighted graph, and then makes a balance partitioning on the weighted graph based on a stepwise minimizing RatioCut Algorithm. The experimental results show that our algorithm generally outperforms the existing algorithms and has a high efficiency and scalability for large-scale graph partitioning. Using our partitioning method, we are able to greatly speed up PageRank computation.
doi:10.1145/2351316.2351325 dblp:conf/kdd/ZengWW12 fatcat:njhkucpffnc4jikyewnmxezysm