An Optimistic Acceleration of AMSGrad for Nonconvex Optimization [article]

Jun-Kun Wang, Xiaoyun Li, Belhal Karimi, Ping Li
<span title="2020-11-03">2020</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
We propose a new variant of AMSGrad, a popular adaptive gradient based optimization algorithm widely used for training deep neural networks. Our algorithm adds prior knowledge about the sequence of consecutive mini-batch gradients and leverages its underlying structure making the gradients sequentially predictable. By exploiting the predictability and ideas from optimistic online learning, the proposed algorithm can accelerate the convergence and increase sample efficiency. After establishing a
more &raquo; ... tighter upper bound under some convexity conditions on the regret, we offer a complimentary view of our algorithm which generalizes the offline and stochastic version of nonconvex optimization. In the nonconvex case, we establish a non-asymptotic convergence bound independently of the initialization. We illustrate the practical speedup on several deep learning models via numerical experiments.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1903.01435v3">arXiv:1903.01435v3</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/7aundvaafba5tcqn4disoxpi4u">fatcat:7aundvaafba5tcqn4disoxpi4u</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20201106100145/https://arxiv.org/pdf/1903.01435v3.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/f6/80/f680f4fec7af8ff2d5b2ae91b4eb8690de0165c3.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1903.01435v3" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>