Analyzing and adjusting user runtime estimates to improve job scheduling on the Blue Gene/P

Wei Tang, Narayan Desai, Daniel Buettner, Zhiling Lan
2010 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS)  
Backfilling and short-job-first are widely acknowledged enhancements to the simple but popular first-come, first-served job scheduling policy. However, both enhancements depend on user-provided estimates of job runtime, which research has repeatedly shown to be inaccurate. We have investigated the effects of this inaccuracy on backfilling and different queue prioritization policies, determining which part of the scheduling policy is most sensitive. Using these results, we have designed and
more » ... mented several estimation-adjusting schemes based on historical data. We have evaluated these schemes using workload traces from the Blue Gene/P system at Argonne National Laboratory. Our experimental results demonstrate that dynamically adjusting job runtime estimates can improve job scheduling performance by up to 20%.
doi:10.1109/ipdps.2010.5470474 dblp:conf/ipps/TangDBL10 fatcat:jmhukxs3grf4fk7bpjhq55ctei