8 Hits in 7.9 sec

Sharper Rates for Separable Minimax and Finite Sum Optimization via Primal-Dual Extragradient Methods [article]

Yujia Jin, Aaron Sidford, Kevin Tian
2022 arXiv   pre-print
Our algorithms all build upon techniques related to the analysis of primal-dual extragradient methods via relative Lipschitzness proposed recently by [CST21]. (1) Separable minimax optimization.  ...  We generalize our algorithms for minimax and finite sum optimization to solve a natural family of minimax finite sum optimization problems at an accelerated rate, encapsulating both above results up to  ...  Acknowledgments We thank Yair Carmon, Arun Jambulapati and Guanghui  ... 
arXiv:2202.04640v1 fatcat:6decqyz3avdale43hx2zwebt6m

The Complexity of Nonconvex-Strongly-Concave Minimax Optimization [article]

Siqi Zhang, Junchi Yang, Cristóbal Guzmán, Negar Kiyavash, Niao He
2021 arXiv   pre-print
This paper studies the complexity for finding approximate stationary points of nonconvex-strongly-concave (NC-SC) smooth minimax problems, in both general and averaged smooth finite-sum settings.  ...  In the averaged smooth finite-sum setting, our proposed algorithm improves over previous algorithms by providing a nearly-tight dependence on the condition number.  ...  near-optimal algorithms for (finite-sum) NC-SC minimax optimization.  ... 
arXiv:2103.15888v1 fatcat:7htfnvvs25ftnblfkyrrriwpza

Bring Your Own Algorithm for Optimal Differentially Private Stochastic Minimax Optimization [article]

Liang Zhang, Kiran Koshy Thekumparampil, Sewoong Oh, Niao He
2022 arXiv   pre-print
We provide a general framework for solving differentially private stochastic minimax optimization (DP-SMO) problems, which enables the practitioners to bring their own base optimization algorithm and use  ...  We study differentially private (DP) algorithms for smooth stochastic minimax optimization, with stochastic minimization as a byproduct.  ...  Palaniappan and Bach [36] first introduced the use of variance reduction methods into finite-sum minimax optimization problems.  ... 
arXiv:2206.00363v1 fatcat:fatjyif7lre5jmk27frq4kviwe

Optimal Extragradient-Based Bilinearly-Coupled Saddle-Point Optimization [article]

Simon S. Du, Gauthier Gidel, Michael I. Jordan, Chris Junchi Li
2022 arXiv   pre-print
error term for bounded stochastic noise that is optimal up to a constant prefactor.  ...  Building upon standard stochastic extragradient analysis for variational inequalities, we present a stochastic accelerated gradient-extragradient (AG-EG) descent-ascent algorithm that combines extragradient  ...  under grant number N00014-18-1-2764 and also the Vannevar Bush Faculty Fellowship program under grant number N00014-21-1-2941 and NSF grant IIS-1901252 to MIJ.  ... 
arXiv:2206.08573v3 fatcat:7jls7yoiyfhwbnj42ki4hszzw4

Stability and Generalization of Stochastic Gradient Methods for Minimax Problems [article]

Yunwen Lei, Zhenhuan Yang, Tianbao Yang, Yiming Ying
2021 arXiv   pre-print
For the convex-concave setting, our stability analysis shows that stochastic gradient descent ascent attains optimal generalization bounds for both smooth and nonsmooth minimax problems.  ...  In this paper, we provide a comprehensive generalization analysis of stochastic gradient methods for minimax problems under both convex-concave and nonconvex-nonconcave cases through the lens of algorithmic  ...  Acknowledgments We are grateful to the anonymous reviewers for their constructive comments and suggestions. The work of Yiming Ying is partially supported by NSF grants IIS-1816227 and IIS-2008532.  ... 
arXiv:2105.03793v2 fatcat:ynds6aguvbao5mkbzo2ek4wiry

Some Adaptive First-order Methods for Variational Inequalities with Relatively Strongly Monotone Operators and Generalized Smoothness [article]

A. A. Titov, S. S. Ablaev, M. S. Alkousa, F. S. Stonyakin, A. V. Gasnikov
2022 arXiv   pre-print
Sharper rates for separable minimax and finite sum optimization via primal-dual extragradient methods. arXiv preprint arXiv:2202.04640.  ...  We provide the motivation for such an approach and obtain theoretical results of the proposed method.  ...  -We present some numerical experiments, which demonstrate the effectiveness of the proposed methods. We start with some auxiliaries. Let E be a finite-dimensional vector space and E * be its dual.  ... 
arXiv:2207.09544v2 fatcat:riwd5zxynrbgbl556jo4thb7fa

Exploiting Smoothness in Statistical Learning, Sequential Prediction, and Stochastic Optimization [article]

Mehrdad Mahdavi
2014 arXiv   pre-print
In the last several years, the intimate connection between convex optimization and learning problems, in both statistical and sequential frameworks, has shifted the focus of algorithmic machine learning  ...  In particular we examine how smoothness of loss function could be beneficial or detrimental in these settings in terms of sample complexity, statistical consistency, regret analysis, and convergence rate  ...  In what follows, we present a primal-dual prox method for such non-smooth cost functions and prove its regret bound.  ... 
arXiv:1407.5908v1 fatcat:vlevdkb23bfibombrkqttvtlp4

On Seven Fundamental Optimization Challenges in Machine Learning

Konstantin Mishchenko
In this thesis, we discuss new developments in optimization inspired by the needs and practice of machine learning, federated learning, and data science.  ...  The fourth challenge is related to the class of adaptive methods.  ...  ACKNOWLEDGEMENTS The last inequality is trivially satisfied for all k ≥ 0. We are not ready to prove Theorem 6.5.6. Below we repeat its statement and prove the proof right afterwards.  ... 
doi:10.25781/kaust-nv6ui fatcat:2ooj4nwi5bbcjbswy4ebl5hkhm