1,909 Hits in 3.7 sec

Distributed Edge Partitioning for Trillion-edge Graphs [article]

Masatoshi Hanai, Toyotaro Suzumura, Wen Jun Tan, Elvis Liu, Georgios Theodoropoulos, Wentong Cai
2019 arXiv   pre-print
We propose Distributed Neighbor Expansion (Distributed NE), a parallel and distributed graph partitioning method that can scale to trillion-edge graphs while providing high partitioning quality.  ...  As a result, Distributed NE can partition a trillion-edge graph using only 256 machines within 70 minutes.  ...  Scalability to Trillion-edge Graph Figure 10(j) shows the scalability of Distributed NE to trillion-edge graph.  ... 
arXiv:1908.05855v2 fatcat:5atwo3vjfbhozksdhiw5c25j24

Partitioning Trillion-Edge Graphs in Minutes

George M. Slota, Sivasankaran Rajamanickam, Karen Devine, Kamesh Madduri
2017 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS)  
We propose Distributed Neighbor Expansion (Distributed NE), a parallel and distributed graph partitioning method that can scale to trillion-edge graphs while providing high partitioning quality.  ...  As a result, Distributed NE can partition a trillion-edge graph using only 256 machines within 70 minutes. PVLDB Reference Format:  ...  Scalability to Trillion-edge Graph Figure 10(j) shows the scalability of Distributed NE to trillion-edge graph.  ... 
doi:10.1109/ipdps.2017.95 dblp:conf/ipps/SlotaRDM17 fatcat:osvwtvqz35dkdh6dhvhcemih4i

Scaling Techniques for Massive Scale-Free Graphs in Distributed (External) Memory

Roger Pearce, Maya Gokhale, Nancy M. Amato
2013 2013 IEEE 27th International Symposium on Parallel and Distributed Processing  
We present techniques to process large scale-free graphs in distributed memory.  ...  We show results on two clusters with local NVRAM storage that are capable of traversing trillion-edge scale-free graphs.  ...  SCALE 36 IS A GRAPH WITH OVER 1 TRILLION EDGES.  ... 
doi:10.1109/ipdps.2013.72 dblp:conf/ipps/PearceGA13 fatcat:e6l6dyhzj5bo5nceq7iglhvi3i

Scaling graph traversal to 281 trillion edges with 40 million cores

Huanqi Cao, Yuanwei Wang, Haojie Wang, Heng Lin, Zixuan Ma, Wanwang Yin, Wenguang Chen
2022 Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming  
The successor of Sunway TaihuLight, New Sunway, is equipped with nearly 10 PB memory and over 40 million cores, which brings the opportunity to process hundreds of trillions of edges graphs.  ...  Together with sub-iteration direction optimization, core group -aware core subgraph segmenting, and a new on-chip sorting mechanism using RMA, we achieve 180,792 GTEPS on a graph with 281 trillion edges  ...  We partition an R-MAT synthetic graph with 281 trillion edges and 17.6 trillion vertices to 103912 nodes. Vertices are first evenly distributed across nodes.  ... 
doi:10.1145/3503221.3508403 fatcat:qutnrxs4cfdsxbwsvh4tymwlqy

Partitioning Trillion-edge Graphs in Minutes [article]

George M Slota, Sivasankaran Rajamanickam, Karen Devine, Kamesh Madduri
2016 arXiv   pre-print
We introduce XtraPuLP, a new distributed-memory graph partitioner designed to process trillion-edge graphs.  ...  Further, we show that using XtraPuLP partitions for distributed-memory graph analytics leads to significant end-to-end execution time reduction.  ...  We also thank Henning Meyerhenke, Peter Sanders, and Christian Schulz for providing the source code for their partitioner.  ... 
arXiv:1610.07220v1 fatcat:f6dfzvx32zhqjd7c2muqykla3q

Storing and Querying Large-Scale Spatio-Temporal Graphs with High-Throughput Edge Insertions [article]

Mengsu Ding, Muqiao Yang, Shimin Chen
2020 arXiv   pre-print
ii) supporting up to 1 trillion new edges that are streamed in daily, and (iii) minimizing cross-machine communication for query processing.  ...  We propose and evaluate PAST, a framework for efficient PArtitioning and query processing of Spatio-Temporal graphs. Experimental results show that PAST successfully achieves the above goals.  ...  vertices, and ∼100 trillion edges.  ... 
arXiv:1904.09610v2 fatcat:6ysnlkkc5vam7nwscdqdk52mdy

PyTorch-BigGraph: A Large-scale Graph Embedding System [article]

Adam Lerer, Ledell Wu, Jiajun Shen, Timothee Lacroix, Luca Wehrstedt, Abhijit Bose, Alex Peysakhovich
2019 arXiv   pre-print
trillions of edges.  ...  PBG uses graph partitioning to train arbitrarily large embeddings on either a single machine or in a distributed environment.  ...  ACKNOWLEDGEMENTS We would like to acknowledge Adam Fisch, Keith Adams, Jason Weston, Antoine Bordes and Serkan Piantino for helping to formulate the initial ideas that led to this work, as well as Maximilian  ... 
arXiv:1903.12287v3 fatcat:k4oalzdhjnckzi57pugwx2hs4i


Ha-Myung Park, Sung-Hyon Myaeng, U. Kang
2016 Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD '16  
In this paper, we propose PTE (Pre-partitioned Triangle Enumeration), a new distributed algorithm for enumerating triangles in enormous graphs by resolving the structural inefficiency of the previous MapReduce  ...  How can we enumerate triangles from an enormous graph with billions of vertices and edges?  ...  INTRODUCTION How can we enumerate trillion triangles from an enormous graph with billions of vertices and edges?  ... 
doi:10.1145/2939672.2939757 dblp:conf/kdd/ParkMK16 fatcat:773isyezm5exzfds4fz636mb2e

Graphing trillions of triangles

Paul Burkhardt
2016 Information Visualization  
This is true for many graph problems, such as listing all triangles in a graph.  ...  Enabling scalable graph exploration for Big Graphs requires new approaches to algorithms, architectures, and visual analytics.  ...  Acknowledgements The author thanks David G Harris and Louis W Ibarra for their helpful comments.  ... 
doi:10.1177/1473871616666393 pmid:28690426 pmcid:PMC5480623 fatcat:j2slz5cyubb45itdlgqaksjlrq


Amitabha Roy, Laurent Bindschaedler, Jasmina Malicevic, Willy Zwaenepoel
2015 Proceedings of the 25th Symposium on Operating Systems Principles - SOSP '15  
In terms of capacity scaling, Chaos is capable of handling a graph with 1 trillion edges representing 16 TB of input data, a new milestone for graph processing capacity on a small commodity cluster.  ...  First, Chaos partitions for sequential storage access, rather than for locality and load balance, resulting in much lower pre-processing times.  ...  We would also like to thank Florin Dinu for his feedback, help in setting up the cluster and for motivating us to keep working on graph processing.  ... 
doi:10.1145/2815400.2815408 dblp:conf/sosp/0002BMZ15 fatcat:o7nlnv6wj5evlnmuquoi525hfe

Traversing Trillions of Edges in Real Time: Graph Exploration on Large-Scale Parallel Machines

Fabio Checconi, Fabrizio Petrini
2014 2014 IEEE 28th International Parallel and Distributed Processing Symposium  
In this paper we describe the algorithmic design and the optimization techniques that led to the unprecedented processing rate of 15.3 trillion edges per second on 64 thousand BlueGene/Q nodes, that allowed  ...  the in-memory exploration of a petabyte-scale graph in just a few seconds.  ...  We would like to thank Susan Coghlan, Kalyan Kumaran, and Ray Loy at ANL and David Fox and Scott Futral at LLNL for their help in the access to Mira and Sequoia.  ... 
doi:10.1109/ipdps.2014.52 dblp:conf/ipps/ChecconiP14 fatcat:oifstrhvpvambgy5bggs4npdhe

One trillion edges

Avery Ching, Sergey Edunov, Maja Kabiljo, Dionysios Logothetis, Sambavi Muthukrishnan
2015 Proceedings of the VLDB Endowment  
one trillion edges.  ...  Industry graphs can be two orders of magnitude larger -hundreds of billions or up to one trillion edges.  ...  Piccolo [36] executes distributed graph computations on top of partitioned tables.  ... 
doi:10.14778/2824032.2824077 fatcat:wfivemamp5grpjc5mwqgrmfd5u

Streaming graph partitioning for large distributed graphs

Isabelle Stanton, Gabriel Kliot
2012 Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '12  
Using our streaming partitioning methods, we are able to speed up PageRank computations on Spark [32], a distributed computation system, by 18% to 39% for large social networks.  ...  A standard approach distributes the graph over a cluster of nodes, but performing computations on a distributed graph is expensive if large amount of data have to be moved.  ...  Our intent is to use this style of solution for graphs that include trillions of edges, yet in our initial experiments our largest graph has 1.4 million edges.  ... 
doi:10.1145/2339530.2339722 dblp:conf/kdd/StantonK12 fatcat:nqaphisol5bozfusje4zxuqdze

Standard Framework for Comparison of Graph Partitioning Techniques

Mudasser Iqbal, Dr. Syed Saif-ur-Rahman
2015 Journal of Independent Studies and Research - Computing  
Graph Partitioning is used to distribute graph partitions across nodes for processing. It is very important in the pre-processing step for distributed graph processing.  ...  In Math and Computer Science, many different distributed graph processing solutions use different partitioning approaches.  ...  The notable graph is the WWW which now contains more than 50 billion web pages and more than one trillion unique URLs [1] .  ... 
doi:10.31645/jisrc/(2015).13.1.0008 fatcat:7a3q4cevlbhgzf4n2zw5pxygmq

A parallel graph partitioning algorithm to speed up the large-scale distributed graph mining

ZengFeng Zeng, Bin Wu, Haoyu Wang
2012 Proceedings of the 1st International Workshop on Big Data, Streams and Heterogeneous Source Mining Algorithms, Systems, Programming Models and Applications - BigMine '12  
For the large-scale distributed graph mining, the graph is distributed over a cluster of nodes, thus performing computations on the distributed graph is expensive when large amount of data have to be moved  ...  A efficient and scalable partitioning algorithm is crucial for large-scale distributed graph mining. In this paper, we propose a novel parallel multi-level stepwise partitioning algorithm.  ...  ACKNOWLEDGMENTS The authors would like to thank the anonymous reviewers for their valuable comments and suggestions to improve the quality of this paper.This work is supported by the National Natural Science  ... 
doi:10.1145/2351316.2351325 dblp:conf/kdd/ZengWW12 fatcat:njhkucpffnc4jikyewnmxezysm
« Previous Showing results 1 — 15 out of 1,909 results