Improving sparse data movement performance using multiple paths on the Blue Gene/Q supercomputer

Huy Bui, Eun-Sung Jung, Venkatram Vishwanath, Andrew Johnson, Jason Leigh, Michael E. Papka
2016 Parallel Computing  
In situ analysis has been proposed as a promising solution to glean faster insights and reduce the amount of data to storage. A critical challenge here is that the reduced dataset is typically located on a subset of the nodes and needs to be written out to storage. Data coupling in multiphysics codes also exhibits a sparse data movement pattern wherein data movement occurs among a subset of nodes. We evaluate the performance of data movement for sparse data patterns on the IBM Blue Gene/Q
more » ... omputing system "Mira" and identify performance bottlenecks. We propose a multipath data movement algorithm for sparse data patterns based on an adaptation of a maximum flow algorithm together with breadth-first search that fully exploits all the underlying data paths and I/O nodes to improve data movement. We demonstrate the efficacy of our solutions through a set of microbenchmarks and application benchmarks on Mira scaling up to 131,072 compute cores. The results show that our approach achieves up to 5X improvement in achievable throughput compared with the default mechanisms.
doi:10.1016/j.parco.2015.09.002 fatcat:4ze4glj42neqhdyec6yv4xfx3a