Iteration Based Collective I/O Strategy for Parallel I/O Systems

Zhixiang Wang, Xuanhua Shi, Hai Jin, Song Wu, Yong Chen
2014 2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing  
MPI collective I/O is a widely used I/O method that helps data-intensive scientific applications gain better I/O performance. However, it has been observed that existing collective I/O strategies do not perform well due to the access contention problem. Existing collective I/O optimization strategies mainly focus on the I/O phase efficiency and ignore the shuffle cost that may limit the potential of their performance improvement. We observe that as the size of I/O becomes larger, one I/O
more » ... on from the upper application would be separated into several iterations to complete. So, I/O requests in each file domain do not necessarily issue to the parallel file system simultaneously unless they are carried out within the same iteration step. Based on that observation, this paper proposes a new collective I/O strategy that reorganizes I/O requests within each file domain instead of coordinating requests across file domains, such that we can eliminate access contentions without introducing extra shuffle cost between aggregators and computing processes. Using benchmark workloads IOR, we evaluate our new strategy and compare with the conventional one. The proposed strategy achieves up to 47%-63% I/O bandwidth improvement compared to the existing ROMIO collective I/O strategy.
doi:10.1109/ccgrid.2014.61 dblp:conf/ccgrid/WangSJWC14 fatcat:yidrtylnnrafpmdl3i7bycygne