Efficient Coflow Scheduling Without Prior Knowledge

Mosharaf Chowdhury, Ion Stoica
2015 Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication - SIGCOMM '15  
Inter-coflow scheduling improves application-level communication performance in data-parallel clusters. However, existing efficient schedulers require a priori coflow information and ignore cluster dynamics like pipelining, task failures, and speculative executions, which limit their applicability. Schedulers without prior knowledge compromise on performance to avoid head-of-line blocking. In this paper, we present Aalo that strikes a balance and efficiently schedules coflows without prior
more » ... edge. Aalo employs Discretized Coflow-Aware Least-Attained Service (D-CLAS) to separate coflows into a small number of priority queues based on how much they have already sent across the cluster. By performing prioritization across queues and by scheduling coflows in the FIFO order within each queue, Aalo's non-clairvoyant scheduler reduces coflow completion times while guaranteeing starvation freedom. EC2 deployments and trace-driven simulations show that communication stages complete 1.93× faster on average and 3.59× faster at the 95th percentile using Aalo in comparison to per-flow mechanisms. Aalo's performance is comparable to that of solutions using prior knowledge, and Aalo outperforms them in presence of cluster dynamics.
doi:10.1145/2785956.2787480 dblp:conf/sigcomm/ChowdhuryS15 fatcat:mcejrhhldzbujjwtgnskmwtvem