Toward Understanding the Impact of Staleness in Distributed Machine Learning [article]

Wei Dai, Yi Zhou, Nanqing Dong, Hao Zhang, Eric P. Xing
<span title="2018-10-08">2018</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
Many distributed machine learning (ML) systems adopt the non-synchronous execution in order to alleviate the network communication bottleneck, resulting in stale parameters that do not reflect the latest updates. Despite much development in large-scale ML, the effects of staleness on learning are inconclusive as it is challenging to directly monitor or control staleness in complex distributed environments. In this work, we study the convergence behaviors of a wide array of ML models and
more &raquo; ... ms under delayed updates. Our extensive experiments reveal the rich diversity of the effects of staleness on the convergence of ML algorithms and offer insights into seemingly contradictory reports in the literature. The empirical findings also inspire a new convergence analysis of stochastic gradient descent in non-convex optimization under staleness, matching the best-known convergence rate of O(1/√(T)).
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1810.03264v1">arXiv:1810.03264v1</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/drjb6zxycnawpgd7j62r4d7gbu">fatcat:drjb6zxycnawpgd7j62r4d7gbu</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200909015148/https://arxiv.org/pdf/1810.03264v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/39/f5/39f593a6ba742751f46011d97661d20d7437d016.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1810.03264v1" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>