K MapReduce: A scalable tool for data-processing and search/ensemble applications on large-scale supercomputers

Motohiko Matsuda, Naoya Maruyama, Shin'ichiro Takizawa
2013 2013 IEEE International Conference on Cluster Computing (CLUSTER)  
K MapReduce (KMR) is a high-performance MapReduce system in the MPI enviromuent, targeting large-scale supercomputers such as the K computer. Its objectives are to ease programming for data-processing and to achieve efficiency by utilizing the large amount of memory available in large scale supercomputers. In KMR, shuffling operation exchanges key-value pairs in a scalable way by collective communication algorithms utilizing the K's interconnect. Mapping and reducing operations are
more » ... d to achieve even greater efficiency in modern multi-core machines. Sorting is optimized using fixed length packed keys instead of variable-length raw keys, which is extensively used inside of shuffling and reducing operations. Besides the MapReduce operations, KMR provides routines for collective file reading for affinity-aware optimizations. This paper presents the results of experimental performance studies of KMR on the K computer. Affinity-aware file loading improves the performance by about 42% over a non-optimized implementation. We also show how KMR can be used to program real-world scientific applications such as meta-genome search and replica exchange molecular dynamics.
doi:10.1109/cluster.2013.6702663 dblp:conf/cluster/MatsudaMT13 fatcat:w5geh2geszc2rnb7453sisooby