Stochastic Normalized Gradient Descent with Momentum for Large Batch Training [article]

Shen-Yi Zhao, Yin-Peng Xie, Wu-Jun Li
<span title="2020-07-28">2020</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
Stochastic gradient descent (SGD) and its variants have been the dominating optimization methods in machine learning. Compared with small batch training, SGD with large batch training can better utilize the computational power of current multi-core systems like GPUs and can reduce the number of communication rounds in distributed training. Hence, SGD with large batch training has attracted more and more attention. However, existing empirical results show that large batch training typically
more &raquo; ... to a drop of generalization accuracy. As a result, large batch training has also become a challenging topic. In this paper, we propose a novel method, called stochastic normalized gradient descent with momentum (SNGM), for large batch training. We theoretically prove that compared to momentum SGD (MSGD) which is one of the most widely used variants of SGD, SNGM can adopt a larger batch size to converge to the ϵ-stationary point with the same computation complexity (total number of gradient computation). Empirical results on deep learning also show that SNGM can achieve the state-of-the-art accuracy with a large batch size.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2007.13985v1">arXiv:2007.13985v1</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/dpfahtshefezjg4zjtzfkwmjlu">fatcat:dpfahtshefezjg4zjtzfkwmjlu</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200910112545/https://arxiv.org/pdf/2007.13985v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/94/8c/948c28201ac06e6f9e24c3dc7d8249b39e0290c7.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2007.13985v1" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>