A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit <a rel="external noopener" href="https://arxiv.org/pdf/1903.01435v3.pdf">the original URL</a>. The file type is <code>application/pdf</code>.
An Optimistic Acceleration of AMSGrad for Nonconvex Optimization
[article]
<span title="2020-11-03">2020</span>
<i >
arXiv
</i>
<span class="release-stage" >pre-print</span>
We propose a new variant of AMSGrad, a popular adaptive gradient based optimization algorithm widely used for training deep neural networks. Our algorithm adds prior knowledge about the sequence of consecutive mini-batch gradients and leverages its underlying structure making the gradients sequentially predictable. By exploiting the predictability and ideas from optimistic online learning, the proposed algorithm can accelerate the convergence and increase sample efficiency. After establishing a
<span class="external-identifiers">
<a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1903.01435v3">arXiv:1903.01435v3</a>
<a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/7aundvaafba5tcqn4disoxpi4u">fatcat:7aundvaafba5tcqn4disoxpi4u</a>
</span>
more »
... tighter upper bound under some convexity conditions on the regret, we offer a complimentary view of our algorithm which generalizes the offline and stochastic version of nonconvex optimization. In the nonconvex case, we establish a non-asymptotic convergence bound independently of the initialization. We illustrate the practical speedup on several deep learning models via numerical experiments.
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20201106100145/https://arxiv.org/pdf/1903.01435v3.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext">
<button class="ui simple right pointing dropdown compact black labeled icon button serp-button">
<i class="icon ia-icon"></i>
Web Archive
[PDF]
<div class="menu fulltext-thumbnail">
<img src="https://blobs.fatcat.wiki/thumbnail/pdf/f6/80/f680f4fec7af8ff2d5b2ae91b4eb8690de0165c3.180px.jpg" alt="fulltext thumbnail" loading="lazy">
</div>
</button>
</a>
<a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1903.01435v3" title="arxiv.org access">
<button class="ui compact blue labeled icon button serp-button">
<i class="file alternate outline icon"></i>
arxiv.org
</button>
</a>