K MapReduce: A scalable tool for data-processing and search/ensemble applications on large-scale supercomputers
2013 IEEE International Conference on Cluster Computing (CLUSTER)
K MapReduce (KMR) is a high-performance MapReduce system in the MPI enviromuent, targeting large-scale supercomputers such as the K computer. Its objectives are to ease programming for data-processing and to achieve efficiency by utilizing the large amount of memory available in large scale supercomputers. In KMR, shuffling operation exchanges key-value pairs in a scalable way by collective communication algorithms utilizing the K's interconnect. Mapping and reducing operations are
... d to achieve even greater efficiency in modern multi-core machines. Sorting is optimized using fixed length packed keys instead of variable-length raw keys, which is extensively used inside of shuffling and reducing operations. Besides the MapReduce operations, KMR provides routines for collective file reading for affinity-aware optimizations. This paper presents the results of experimental performance studies of KMR on the K computer. Affinity-aware file loading improves the performance by about 42% over a non-optimized implementation. We also show how KMR can be used to program real-world scientific applications such as meta-genome search and replica exchange molecular dynamics.