Bandwidth Efficient All-reduce Operation on Tree Topologies

Pitch Patarasuk, Xin Yuan
2007 2007 IEEE International Parallel and Distributed Processing Symposium  
We consider efficient implementations of the all-reduce operation with large data sizes on tree topologies. We prove a tight lower bound of the amount of data that must be transmitted to carry out the all-reduce operation and use it to derive the lower bound for the communication time of this operation. We develop a topology specific algorithm that is bandwidth efficient in that (1) the amount of data sent/received by each process is minimum for this operation; and (2) the communications do not
more » ... incur network contention on the tree topology. With the proposed algorithm, the all-reduce operation can be realized on the tree topology as efficiently as on any other topology when the data size is sufficiently large. The proposed algorithm can be applied to several contemporary cluster environments, including high-end clusters of workstations with SMP and/or multi-core nodes and low-end Ethernet switched clusters. We evaluate the algorithm on various clusters of workstations, including a Myrinet cluster with dual-processor SMP nodes, an InfiniBand cluster with two dual-core processors SMP nodes, and an Ethernet switched cluster with single processor nodes. The results show that the routines implemented based on the proposed algorithm significantly outperform the native MPI Allreduce and other recently developed algorithms for high-end SMP clusters when the data size is sufficiently large.
doi:10.1109/ipdps.2007.370405 dblp:conf/ipps/PatarasukY07 fatcat:v6ouncsar5acheytrt7pwteq3y