11,191 Hits in 10.7 sec

Multiplicative Weights Update as a Distributed Constrained Optimization Algorithm: Convergence to Second-order Stationary Points Almost Always [article]

Ioannis Panageas and Georgios Piliouras and Xiao Wang
2020 arXiv   pre-print
We show that MWU converges almost always for small enough stepsizes to critical points that satisfy the second order KKT conditions.  ...  We analyze a variant of a well-established algorithm in machine learning called Multiplicative Weights Update (MWU) for the maximization problem max_x∈ D P(x), where P is non-concave, twice continuously  ...  What is less understood is the problem of convergence to second order stationary points in constrained optimization (under the weaker assumption that we do not have access to the subgradient of the indicator  ... 
arXiv:1810.05355v3 fatcat:jei5e5vgqvau3kyrvzg3y3kiky

Parallel Selective Algorithms for Nonconvex Big Data Optimization

Francisco Facchinei, Gesualdo Scutari, Simone Sagratella
2015 IEEE Transactions on Signal Processing  
., sequential) ones, as well as virtually all possibilities "in between" with only a subset of variables updated at each iteration.  ...  We propose a decomposition framework for the parallel optimization of the sum of a differentiable (possibly nonconvex) function and a (block) separable nonsmooth, convex one.  ...  ACKNOWLEDGMENT The authors are very grateful to Loris Cannelli and Paolo Scarponi for their invaluable help in developing the C++ code of the simulated algorithms.  ... 
doi:10.1109/tsp.2015.2399858 fatcat:hpgqi3mgxnbptkbltn3u6pgae4

Optimization for deep learning: theory and algorithms [article]

Ruoyu Sun
2019 arXiv   pre-print
Second, we review generic optimization methods used in training neural networks, such as SGD, adaptive gradient methods and distributed methods, and theoretical results for these algorithms.  ...  When and why can a neural network be successfully trained? This article provides an overview of optimization algorithms and theory for training neural networks.  ...  Acknowledgement We would like to thank Leon Bottou, Yan Dauphin, Yuandong Tian, Levent Sagun, Lechao Xiao, Tengyu Ma, Jason Lee, Matus Telgarsky, Ju Sun, Wei Hu, Simon Du, Lei Wu, Quanquan Gu, Justin Sirignano  ... 
arXiv:1912.08957v1 fatcat:bdtx2o3qhfhthh2vyohkuwnxxa

Steepest Descent Algorithms for Optimization Under Unitary Matrix Constraint

Traian E. Abrudan, Jan Eriksson, Visa Koivunen
2008 IEEE Transactions on Signal Processing  
In many engineering applications we deal with constrained optimization problems with respect to complex-valued matrices.  ...  This paper proposes a Riemannian geometry approach for optimization of a real-valued cost function of complex-valued matrix argument W, under the constraint that W is an unitary matrix.  ...  It is known that in a stationary scenario (i.e., the matrices involved in the cost function are time invariant) the SD algorithm together with the Armijo step size rule [1] almost always converges to  ... 
doi:10.1109/tsp.2007.908999 fatcat:75kgratdcbd7zaa5dtsyelhwtm

Reweighted nonnegative least-mean-square algorithm

Jie Chen, Cédric Richard, José Carlos M. Bermudez
2016 Signal Processing  
This algorithm builds on a fixed-point iteration strategy driven by the Karush-Kuhn-Tucker conditions.  ...  It was shown to provide low variance estimates, but it however suffers from unbalanced convergence rates of these estimates.  ...  In order to address these problems, it is of interest to derive a variant of the NNLMS algorithm that satisfies the following requirements: The coefficients should converge to the fixed point satisfying  ... 
doi:10.1016/j.sigpro.2016.03.017 fatcat:3uczdohtzfez5dod6zwra4b2pq

Stochastic Majorization-Minimization Algorithms for Large-Scale Optimization [article]

Julien Mairal (INRIA Grenoble Rhône-Alpes / LJK Laboratoire Jean Kuntzmann)
2013 arXiv   pre-print
Equally important, our scheme almost surely converges to stationary points for a large class of non-convex problems. We develop several efficient algorithms based on our framework.  ...  Second, we develop an online DC programming algorithm for non-convex sparse estimation.  ...  Our second analysis shows that for nonconvex problems, our method almost surely converges to a set of stationary points under suitable assumptions.  ... 
arXiv:1306.4650v2 fatcat:iyxhhvqwsbggtgrbp62swgezfm

Second-order Guarantees of Distributed Gradient Algorithms [article]

Amir Daneshmand and Gesualdo Scutari and Vyacheslav Kungurtsev
2020 arXiv   pre-print
Specifically, we establish that (i) the renowned Distributed Gradient Descent (DGD) algorithm likely converges to a neighborhood of a Second-order Stationary (SoS) solution; and (ii) the more recent class  ...  We consider distributed smooth nonconvex unconstrained optimization over networks, modeled as a connected graph.  ...  (1.1)] converges almost surely to a second-order critical point of L α .  ... 
arXiv:1809.08694v5 fatcat:yxmajtg3ffbvtagn7thtcjv3ni

Hybrid Bees Algorithm with Grasshopper Optimization Algorithm for Optimal Deployment of Wireless Sensor Networks

Hicham Deghbouch, Fatima Debbat
2021 Inteligencia Artificial  
To alleviate this shortcoming, this paper proposes a hybrid algorithm that utilizes the strength of the GOA to enhance the exploitation phase of the BA.  ...  To prove the effectiveness of the proposed algorithm, it is applied for WSNs deployment optimization with various deployment settings.  ...  Because sensors are constrained in terms of energy, the movements of sensors should be reduced as much as possible in order to conserve energy.  ... 
doi:10.4114/intartif.vol24iss67pp18-35 fatcat:xzjbl5yk4bcgpcmffhxmntp2we

Bat Algorithm (BA) [chapter]

2018 Swarm Intelligence  
Therefore, almost all the new algorithms can be referred to as nature-inspired.  ...  In [30] , a model has been proposed for classification using bat algorithm to update the weights of a Functional Link Artificial Neural Network (FLANN) classifier.  ...  ABC optimization shows fast convergence and low sensitivity to ini tial point selection than expectation the maximization method which is common.  ... 
doi:10.1201/9781315222455-6 fatcat:724x2apwyrc2ngjsx7satgyrbm

Multiplicative Algorithms for Maximum Penalized Likelihood Inversion with Non Negative Constraints and Generalized Error Distributions

Jun Ma
2006 Communications in Statistics - Theory and Methods  
We show that if there is no penalty, then this algorithm is almost sure to converge; otherwise a relaxation or line search is necessitated to assure its convergence.  ...  Given that the measurements and point-spread-function (PSF) values are all nonnegative, we propose a simple multiplicative iterative algorithm.  ...  We also demonstrate, under certain conditions, that, if the algorithm converges, it will converge to the point where the Kuhn-Tucker necessary conditions for constrained optimization are satisfied.  ... 
doi:10.1080/03610920500501478 fatcat:rebbtkuwhjbozk2qdcwppqt6hy

A Flexible and Efficient Algorithmic Framework for Constrained Matrix and Tensor Factorization

Kejun Huang, Nicholas D. Sidiropoulos, Athanasios P. Liavas
2016 IEEE Transactions on Signal Processing  
methods which help ensure that every limit point is a stationary point, as well as faster and more robust convergence in practice.  ...  We propose a general algorithmic framework for constrained matrix and tensor factorization, which is widely used in signal processing and machine learning.  ...  Moreover, both ADMM and AO guarantee that every limit point is a stationary point, but in practice AO almost always converges (as long as the updates stay bounded), which is not the case for ADMM applied  ... 
doi:10.1109/tsp.2016.2576427 fatcat:dkklj6at2zfk5eikw4kvjpelfq

Error Whitening Wiener Filters: Theory and Algorithms [chapter]

Jose C. Principe, Yadunandana N. Rao, Deniz Erdogmus
2005 Least-Mean-Square Adaptive Filters  
In every case, the algorithm converged to the saddle point in a stable manner. Note that the misadjustment in each case is almost the same.  ...  We will derive a gradient-based LMS-type update algorithm for the weights that will converge to the vicinity of the desired solution using stochastic updates.  ...  Thus, the update equation with the curvature information in (10.65) converges to the stationary point of the quadratic cost function irrespective of the nature of the stationary point.  ... 
doi:10.1002/0471461288.ch10 fatcat:pboawm7g7bdizjyyeyrfpyacia

Distributed Newton-like Algorithms and Learning for Optimized Power Dispatch [article]

Tor Anderson
2021 arXiv   pre-print
The class of distributed optimization methods explored here can be broadly described as Newton-like or second-order, implying utilization of second-derivative information of the cost functions, in contrast  ...  Distributed algorithms are naturally well-poised to address these challenges, in contrast to more traditional centralized algorithms which scale poorly and require global access to information.  ...  Finally, we would like to extend a sincere thanks to the ARPA-e NODES program for its financial support and to its leadership, including  ... 
arXiv:2103.13493v1 fatcat:uytczmdftndkdhy5spo6uej7le

Randomized gossip algorithms

S. Boyd, A. Ghosh, B. Prabhakar, D. Shah
2006 IEEE Transactions on Information Theory  
Motivated by applications to sensor, peer-to-peer and ad hoc networks, we study distributed algorithms, also known as gossip algorithms, for exchanging information and for computing in an arbitrarily connected  ...  In general, SDPs cannot be solved in a distributed fashion; however, exploiting problem structure, we propose a distributed subgradient method that solves the optimization problem over the network.  ...  Shah thanks Bob Gallager for a careful reading and suggestions that led to an improvement in the readability of the final paper.  ... 
doi:10.1109/tit.2006.874516 fatcat:tlug2vktebfqfh6nasqzdvki2u

An MM Algorithm for Split Feasibility Problems [article]

Jason Xu, Eric C. Chi, Meng Yang, Kenneth Lange
2017 arXiv   pre-print
When a feasible point does not exist, solution methods that proceed by minimizing a proximity function can be used to obtain optimal approximate solutions to the problem.  ...  Our algorithm is based on the principle of majorization-minimization, is amenable to quasi-Newton acceleration, and comes complete with convergence guarantees under mild assumptions.  ...  Convergence Analysis We now show that the set of limit points of the MM algorithm belong to the set of stationary points of the proximity function.  ... 
arXiv:1612.05614v2 fatcat:6qtzfychlzcgbpvqxb3debrmre
« Previous Showing results 1 — 15 out of 11,191 results