Filters








165 Hits in 4.3 sec

Asynchronous Stochastic Proximal Methods for Nonconvex Nonsmooth Optimization [article]

Rui Zhu, Di Niu, Zongpeng Li
2018 arXiv   pre-print
To fill this theoretical gap, in this paper, we propose and analyze asynchronous parallel stochastic proximal gradient (Asyn-ProxSGD) methods for nonconvex problems.  ...  However, compared to asynchronous parallel stochastic gradient descent (AsynSGD), an algorithm targeting smooth optimization, the understanding of the behavior of stochastic algorithms for nonsmooth regularized  ...  Concluding Remarks In this paper, we study asynchronous parallel implementations of stochastic proximal gradient methods for solving nonconvex optimization problems, with convex yet possibly nonsmooth  ... 
arXiv:1802.08880v3 fatcat:othkev23cjbo7kbnqmm7i2ux4y

Asynchronous Delay-Aware Accelerated Proximal Coordinate Descent for Nonconvex Nonsmooth Problems

Ehsan Kazemi, Liqiang Wang
2019 PROCEEDINGS OF THE THIRTIETH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE AND THE TWENTY-EIGHTH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE  
However, developing efficient methods for the nonconvex and nonsmooth optimization problems with certain performance guarantee remains a challenge.  ...  Proximal coordinate descent (PCD) has been widely used for solving optimization problems, but the knowledge of PCD methods in the nonconvex setting is very limited.  ...  Related Works Proximal Gradient Algorithms: Proximal gradient methods for nonsmooth regularization are among the most important methods for solving composite optimization problems.  ... 
doi:10.1609/aaai.v33i01.33011528 fatcat:weexm7jahbapnne5kkieox3k44

The Sound of APALM Clapping: Faster Nonsmooth Nonconvex Optimization with Stochastic Asynchronous PALM [article]

Damek Davis, Brent Edmunds, Madeleine Udell
2016 arXiv   pre-print
We introduce the Stochastic Asynchronous Proximal Alternating Linearized Minimization (SAPALM) method, a block coordinate stochastic proximal-gradient method for solving nonconvex, nonsmooth optimization  ...  SAPALM is the first asynchronous parallel optimization method that provably converges on a large class of nonconvex, nonsmooth problems.  ...  We introduce SAPALM, the first asynchronous parallel optimization method that provably converges for all nonconvex, nonsmooth problems of the form (1.1).  ... 
arXiv:1606.02338v1 fatcat:jpphrmyfojcpjif5adaiqmtby4

Proximal Stochastic Methods for Nonsmooth Nonconvex Finite-Sum Optimization

Sashank J. Reddi, Suvrit Sra, Barnabás Póczos, Alexander J. Smola
2016 Neural Information Processing Systems  
We analyze stochastic algorithms for optimizing nonconvex, nonsmooth finite-sum problems, where the nonsmooth part is convex.  ...  For example, it is not known whether the proximal stochastic gradient method with constant minibatch converges to a stationary point.  ...  Final Discussion In this paper, we presented fast stochastic methods for nonsmooth nonconvex optimization.  ... 
dblp:conf/nips/ReddiSPS16 fatcat:pctzoj4hifhuhkqpdlg56ia2fi

Faster Gradient-Free Proximal Stochastic Methods for Nonconvex Nonsmooth Optimization

Feihu Huang, Bin Gu, Zhouyuan Huo, Songcan Chen, Heng Huang
2019 PROCEEDINGS OF THE THIRTIETH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE AND THE TWENTY-EIGHTH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE  
Recently, the first zeroth-order proximal stochastic algorithm was proposed to solve the nonconvex nonsmooth problems.  ...  Proximal gradient method has been playing an important role to solve many machine learning tasks, especially for the nonsmooth problems.  ...  Although the above zeroth-order stochastic methods can effectively solve the nonconvex optimization, there are few zeroth-order stochastic methods for the nonconvex nonsmooth composite optimization except  ... 
doi:10.1609/aaai.v33i01.33011503 fatcat:jaudjs4vobbo3fitemtbzkqvdu

Asynchronous Proximal Stochastic Gradient Algorithm for Composition Optimization Problems

Pengfei Wang, Risheng Liu, Nenggan Zheng, Zhefeng Gong
2019 PROCEEDINGS OF THE THIRTIETH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE AND THE TWENTY-EIGHTH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE  
We prove that the algorithm admits the fastest convergence rate for both strongly convex and general nonconvex cases.  ...  Recently, several stochastic composition gradient algorithms have been proposed, however, these methods are still inefficient and not scalable to large-scale composition optimization problem instances.  ...  Also partially supported by the Hunan Provincial Science & Technology Project Foundation (2018TP1018, 2018RS3065) and the Fundamental Research Funds for the Central Universities.  ... 
doi:10.1609/aaai.v33i01.33011633 fatcat:bdljx46xhzf6fchpwdq5odz33i

Asynchronous Delay-Aware Accelerated Proximal Coordinate Descent for Nonconvex Nonsmooth Problems [article]

Ehsan Kazemi, Liqiang Wang
2019 arXiv   pre-print
However, developing efficient methods for the nonconvex and nonsmooth optimization problems with certain performance guarantee remains a challenge.  ...  Proximal coordinate descent (PCD) has been widely used for solving optimization problems, but the knowledge of PCD methods in the nonconvex setting is very limited.  ...  Then, (Xu and Yin 2015) proposed a block stochastic gradient method for nonconvex and nonsmooth problems.  ... 
arXiv:1902.01856v1 fatcat:n7tmuljtdbd6phxj4whtimr2vi

Asynchronous Optimization Methods for Efficient Training of Deep Neural Networks with Guarantees [article]

Vyacheslav Kungurtsev and Malcolm Egan and Bapi Chatterjee and Dan Alistarh
2020 arXiv   pre-print
However, for nonsmooth and nonconvex objectives, few convergence guarantees exist beyond cases where closed-form proximal operator solutions are available.  ...  In this paper, we analyze for the first time the convergence of stochastic asynchronous optimization for this general class of objectives.  ...  In particular, we show that generic asynchronous stochastic subgradient methods converge with probability one for a general class of nonsmooth and nonconvex problems.  ... 
arXiv:1905.11845v2 fatcat:4nt2sccpvbfsjobmaa4hrcvy6y

A Model Parallel Proximal Stochastic Gradient Algorithm for Partially Asynchronous Systems [article]

Rui Zhu, Di Niu
2018 arXiv   pre-print
In our algorithm, worker nodes communicate with the parameter servers asynchronously, and each worker performs proximal stochastic gradient for only one block of model parameters during each iteration.  ...  amount of training data using proximal stochastic gradient descent (ProxSGD).  ...  Xu and Yin (2015) study block stochastic proximal methods for nonconvex problems.  ... 
arXiv:1810.09270v1 fatcat:oigvzw6pfzfrpnlnqjn6zzsfba

Fast Stochastic Methods for Nonsmooth Nonconvex Optimization [article]

Sashank J. Reddi, Suvrit Sra, Barnabas Poczos, Alex Smola
2016 arXiv   pre-print
This paper builds upon our recent series of papers on fast stochastic methods for smooth nonconvex optimization [22, 23], with a novel analysis for nonconvex and nonsmooth functions.  ...  We analyze stochastic algorithms for optimizing nonconvex, nonsmooth finite-sum problems, where the nonconvex part is smooth and the nonsmooth part is convex.  ...  Acknowledgment We thank Guanghui Lan for very enlightening discussions.  ... 
arXiv:1605.06900v1 fatcat:qfqwgh3vmnb57bxwzxasyr6zci

Asynchronous Schemes for Stochastic and Misspecified Potential Games and Nonconvex Optimization [article]

Jinlong Lei, Uday V. Shanbhag
2019 arXiv   pre-print
By Since any stationary point of the potential function is a Nash equilibrium of the associated game, we believe this paper is amongst the first ones for stochastic nonconvex (but block convex) optimization  ...  Consequently, we design two asynchronous inexact proximal BR schemes to solve the problems, where in each iteration a single player is randomly chosen to compute an inexact proximal BR solution with rivals  ...  Mini-batch stochastic approximation methods were developed by [14] for nonconvex stochastic composite optimization.  ... 
arXiv:1711.03963v3 fatcat:pjyc7l6knradfi6s5bcod4fs7u

Asynchronous Variance-reduced Block Schemes for Composite Nonconvex Stochastic Optimization: Block-specific Steplengths and Adapted Batch-sizes [article]

Jinlong Lei, Uday V. Shanbhag
2020 arXiv   pre-print
We propose an asynchronous variance-reduced algorithm, where in each iteration, a single block is randomly chosen to update its estimates by a proximal variable sample-size stochastic gradient scheme,  ...  We consider the minimization of a sum of an expectation-valued coordinate-wise L_i-smooth nonconvex function and a nonsmooth block-separable convex regularizer.  ...  Mini-batch stochastic approximation methods were developed in [14] for nonconvex stochastic composite optimization.  ... 
arXiv:1808.02543v4 fatcat:niypvn6nijfv5gpcjzgiziouva

Faster Gradient-Free Proximal Stochastic Methods for Nonconvex Nonsmooth Optimization [article]

Feihu Huang, Bin Gu, Zhouyuan Huo, Songcan Chen, Heng Huang
2019 arXiv   pre-print
Recently, the first zeroth-order proximal stochastic algorithm was proposed to solve the nonconvex nonsmooth problems.  ...  Proximal gradient method has been playing an important role to solve many machine learning tasks, especially for the nonsmooth problems.  ...  Although the above zeroth-order stochastic methods can effectively solve the nonconvex optimization, there are few zeroth-order stochastic methods for the nonconvex nonsmooth composite optimization except  ... 
arXiv:1902.06158v1 fatcat:eotzjizo6jbdfcjrcpr56mdbpi

Asynchronous Decentralized Successive Convex Approximation [article]

Ye Tian, Ying Sun, Gesualdo Scutari
2020 arXiv   pre-print
The optimization problem consists of minimizing a (possibly nonconvex) smooth function--the sum of the agents' local costs--plus a convex (possibly nonsmooth) regularizer, subject to convex constraints  ...  We study decentralized asynchronous multiagent optimization over networks, modeled as static (possibly directed) graphs.  ...  Current techniques from centralized (nonsmooth) SCA optimization methods [10, [34] [35] [36] , error-bound analysis [22] , and (asynchronous) consensus algorithms, alone or brute-forcely put together  ... 
arXiv:1909.10144v2 fatcat:ph2265im5rdrvfkr5gbdnwpurm

A Proximal Zeroth-Order Algorithm for Nonconvex Nonsmooth Problems [article]

Ehsan Kazemi, Liqiang Wang
2018 arXiv   pre-print
In this paper, we focus on solving an important class of nonconvex optimization problems which includes many problems for example signal processing over a networked multi-agent system and distributed learning  ...  proposes a proximal zeroth-order primal dual algorithm (PZO-PDA) that accounts for the information structure of the problem.  ...  [12] proposed an asynchronous stochastic optimization algorithm with zeroth-order methods and proved a convergence rate of O(1/ √ R).  ... 
arXiv:1810.10085v1 fatcat:mn6maoymorgsfhdzddvfskfzpm
« Previous Showing results 1 — 15 out of 165 results