A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit <a rel="external noopener" href="https://arxiv.org/pdf/2007.13985v1.pdf">the original URL</a>. The file type is <code>application/pdf</code>.
Stochastic Normalized Gradient Descent with Momentum for Large Batch Training
[article]
<span title="2020-07-28">2020</span>
<i >
arXiv
</i>
<span class="release-stage" >pre-print</span>
Preserved Fulltext
Stochastic gradient descent (SGD) and its variants have been the dominating optimization methods in machine learning. Compared with small batch training, SGD with large batch training can better utilize the computational power of current multi-core systems like GPUs and can reduce the number of communication rounds in distributed training. Hence, SGD with large batch training has attracted more and more attention. However, existing empirical results show that large batch training typically
<span class="external-identifiers">
<a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2007.13985v1">arXiv:2007.13985v1</a>
<a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/dpfahtshefezjg4zjtzfkwmjlu">fatcat:dpfahtshefezjg4zjtzfkwmjlu</a>
</span>
more »
... to a drop of generalization accuracy. As a result, large batch training has also become a challenging topic. In this paper, we propose a novel method, called stochastic normalized gradient descent with momentum (SNGM), for large batch training. We theoretically prove that compared to momentum SGD (MSGD) which is one of the most widely used variants of SGD, SNGM can adopt a larger batch size to converge to the ϵ-stationary point with the same computation complexity (total number of gradient computation). Empirical results on deep learning also show that SNGM can achieve the state-of-the-art accuracy with a large batch size.
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200910112545/https://arxiv.org/pdf/2007.13985v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext">
<button class="ui simple right pointing dropdown compact black labeled icon button serp-button">
<i class="icon ia-icon"></i>
Web Archive
[PDF]
<div class="menu fulltext-thumbnail">
<img src="https://blobs.fatcat.wiki/thumbnail/pdf/94/8c/948c28201ac06e6f9e24c3dc7d8249b39e0290c7.180px.jpg" alt="fulltext thumbnail" loading="lazy">
</div>
</button>
</a>
<a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2007.13985v1" title="arxiv.org access">
<button class="ui compact blue labeled icon button serp-button">
<i class="file alternate outline icon"></i>
arxiv.org
</button>
</a>