Filters








2,070 Hits in 2.2 sec

Safeguarded Learned Convex Optimization [article]

Howard Heaton and Xiaohan Chen and Zhangyang Wang and Wotao Yin
2020 arXiv   pre-print
However, we present a framework that uses L2O updates together with a safeguard to guarantee convergence for convex problems with proximal and/or gradient oracles.  ...  Data-driven algorithms can "learn to optimize" (L2O) with much fewer iterations and with similar cost per iteration as general-purpose optimization algorithms.  ...  Consider a convex function f : R n → R with subgradient ∂f .  ... 
arXiv:2003.01880v2 fatcat:nkn2u7epvfbilprcu6t6sgwiti

A Simple Guard for Learned Optimizers [article]

Isabeau Prémont-Schwarz, Jaroslav Vítků, Jan Feyereisl
2022 arXiv   pre-print
(Heaton et al., 2020) proposed Safeguarded L2O (GL2O) which can take a learned optimizer and safeguard it with a generic learning algorithm so that by conditionally switching between the two, the resulting  ...  If the trend of learned components eventually outperforming their hand-crafted version continues, learned optimizers will eventually outperform hand-crafted optimizers like SGD or Adam.  ...  Most recently, Heaton et al. (2020) proposed a method for safeguarding the behaviour of any learned optimizer by combining it with stochastic gradient descent (SGD) to confer the hybrid algorithm with  ... 
arXiv:2201.12426v1 fatcat:tytjnxw24neixazl6dgjeqbm4a

Variable Metric Proximal Gradient Method with Diagonal Barzilai-Borwein Stepsize [article]

Youngsuk Park, Sauptik Dhar, Stephen Boyd, Mohak Shah
2019 arXiv   pre-print
Variable metric proximal gradient (VM-PG) is a widely used class of convex optimization method.  ...  However, most such metric selections are dependent on (an expensive) Hessian, or limited to scalar stepsizes like the Barzilai-Borwein (BB) stepsize with lots of safeguarding.  ...  Introduction We tackle a convex optimization in the composite form minimize x∈R n F (x) := f (x) + g(x), (1) where x ∈ R n is the decision variable, f : R n → R is convex and differentiable, and g : R  ... 
arXiv:1910.07056v1 fatcat:fjzulluo4vbrnapggcj3coyoyq

Spectral-like gradient method for distributed optimization

Dusan Jakovetic, Natasa Krejic, Natasa Krklec Jerinkic
2019 Zenodo  
We consider a standard distributed multi-agent optimization setting where n nodes (agents) in a network minimize the aggregate sum of their local convex cost functions.  ...  We present a distributed spectral-like gradient method, wherein stepsizes are node- and iteration-varying, and they are inspired by classical spectral methods from centralized optimization.  ...  ., [3] , and parallel and distributed machine learning, e.g., [14] .  ... 
doi:10.5281/zenodo.3333533 fatcat:7beehcmizne3hanz5fns7tjiaa

Byzantine-Resilient Non-Convex Stochastic Gradient Descent [article]

Zeyuan Allen-Zhu, Faeze Ebrahimian, Jerry Li, Dan Alistarh
2021 arXiv   pre-print
We consider a variant of this procedure in the challenging non-convex case.  ...  We study adversary-resilient stochastic distributed optimization, in which m machines can independently compute stochastic gradients, and cooperate to jointly optimize over their local objective functions  ...  Non-convex Byzantine-resilient stochastic optimization.  ... 
arXiv:2012.14368v2 fatcat:pe2csof2ijf5jhryob6f73hdki

Primal-Dual Active-Set Methods for Isotonic Regression and Trend Filtering [article]

Zheng Han, Frank E. Curtis
2016 arXiv   pre-print
Isotonic regression (IR) is a non-parametric calibration method used in supervised learning.  ...  In addition, we propose PDAS variants (with safeguarding to ensure convergence) for solving related trend filtering (TF) problems, providing the results of experiments to illustrate their effectiveness  ...  , and the regularization function g : R n → R is convex but not necessarily smooth.  ... 
arXiv:1508.02452v2 fatcat:ycwhh6xh7bfpjgqw6m35qb6cdm

Dynamic Regret of Convex and Smooth Functions [article]

Peng Zhao, Yu-Jie Zhang, Lijun Zhang, Zhi-Hua Zhou
2020 arXiv   pre-print
We investigate online convex optimization in non-stationary environments and choose the dynamic regret as the performance measure, defined as the difference between cumulative loss incurred by the online  ...  Although this bound is proved to be minimax optimal for convex functions, in this paper, we demonstrate that it is possible to further enhance the dynamic regret by exploiting the smoothness condition.  ...  Online Learning and Online Convex Optimization. Foundations and Trends in Machine Learning, 4(2):107-194, 2012. Nathan Srebro, Karthik Sridharan, and Ambuj Tewari.  ... 
arXiv:2007.03479v2 fatcat:wb6hnd2lcncgbetgyzuf3fv6zy

Learning to Optimize: A Primer and A Benchmark [article]

Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, Wotao Yin
2021 arXiv   pre-print
Learning to optimize (L2O) is an emerging approach that leverages machine learning to develop optimization methods, aiming at reducing the laborious iterations of hand engineering.  ...  The practicality of L2O depends on the type of target optimization, the chosen architecture of the method to learn, and the training procedure.  ...  approximation algorithms [47] , and safeguarded learned algorithms [24] .  ... 
arXiv:2103.12828v2 fatcat:c75y3wz6cngirb2zpugjk63ymq

Safe Exploration in Model-based Reinforcement Learning using Control Barrier Functions [article]

Max H. Cohen, Calin Belta
2021 arXiv   pre-print
This paper develops a model based reinforcement learning (MBRL) framework for learning online the value function of an infinite-horizon optimal control problem while obeying safety constraints expressed  ...  We show how these LCBFs can be used to augment a learning-based control policy to guarantee safety and then leverage this approach to develop a safe exploration framework in a MBRL setting.  ...  and the associated optimal policy are safely learned online.  ... 
arXiv:2104.08171v3 fatcat:cntqewfavjaj5dh6ulnyfgb4qm

IntSGD: Adaptive Floatless Compression of Stochastic Gradients [article]

Konstantin Mishchenko and Bokun Wang and Dmitry Kovalev and Peter Richtárik
2022 arXiv   pre-print
Our theory shows that the iteration complexity of IntSGD matches that of SGD up to constant factors for both convex and non-convex, smooth and non-smooth functions, with and without overparameterization  ...  Natural compression for distributed deep learning. arXiv preprint arXiv:1905.10988, 2019. non-convex optimization. In The 33th International Conference on Machine Learning, pp. 699-707, 2016.  ...  Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013.  ... 
arXiv:2102.08374v2 fatcat:4fj5jjriirc5vehj5t3grwit5a

Secrecy Rate Maximization for Hardware Impaired Untrusted Relaying Network with Deep Learning [article]

Hamed Bastami, Majid Moradikia, Hamid Behroozi, Rodrigo C. de Lamare, Ahmed Abdelhadi, Zhigou Ding
2021 arXiv   pre-print
The resultant optimization problem is non-convex, and a suboptimal solution is obtained through the sequential parametric convex approximation (SPCA) method.  ...  Simulation results assess the effect of different system parameters on the ASR performance as well as the effectiveness of the proposed deep learning solution in large-scale cases.  ...  , resulting in a simpler non-convex optimization problem.  ... 
arXiv:2101.02749v1 fatcat:fm7y5jnmtzfkrlcempg2xomkx4

Adaptive Relaxed ADMM: Convergence Theory and Practical Implementation

Zheng Xu, Mario A. T. Figueiredo, Xiaoming Yuan, Christoph Studer, Tom Goldstein
2017 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)  
Many modern computer vision and machine learning ap- plications rely on solving difficult optimization problems that involve non-differentiable objective functions and constraints.  ...  We propose an adaptive method that automatically tunes the key algorithm parameters to achieve optimal performance without user oversight.  ...  Introduction Modern methods in computer vision and machine learning often require solving difficult optimization problems involving non-differentiable objective functions and constraints.  ... 
doi:10.1109/cvpr.2017.765 dblp:conf/cvpr/0002FYSG17 fatcat:dilvj65ayncxviltne44qxnikq

Adaptive Relaxed ADMM: Convergence Theory and Practical Implementation [article]

Zheng Xu, Mario A. T. Figueiredo, Xiaoming Yuan, Christoph Studer, and Tom Goldstein
2017 arXiv   pre-print
Many modern computer vision and machine learning applications rely on solving difficult optimization problems that involve non-differentiable objective functions and constraints.  ...  We propose an adaptive method that automatically tunes the key algorithm parameters to achieve optimal performance without user oversight.  ...  Introduction Modern methods in computer vision and machine learning often require solving difficult optimization problems involving non-differentiable objective functions and constraints.  ... 
arXiv:1704.02712v1 fatcat:bdo5mqcedfhvdnccrqlqol5644

Exact Spectral-Like Gradient Method for Distributed Optimization

Dusan Jakovetic, Natasa Krejic, Natasa Krklec Jerinkic
2019 Zenodo  
In this paper, we consider unconstrained distributed optimization problems where n nodes constitute an arbitrary connected network and collaboratively minimize the sum of their local convex cost functions  ...  The method exhibits R-linear convergence under standard as- sumptions for the nodes' local costs and safeguarding on the algorithm step-sizes.  ...  Therefore, there exists a diagonal matrix H k such that ∇F (x k ) − ∇F (x * ) = H k (x k − x * ) = H k e k , 0 H k LI. ( 30 ) The following standard lemma in the convex optimization theory, [2] will  ... 
doi:10.5281/zenodo.3591086 fatcat:4aum3yzvnbdipklaztt52hcrjq

Compression-Based Regularization with an Application to Multi-Task Learning

Matias Vera, Leonardo Rey Vega, Pablo Piantanida
2018 IEEE Journal on Selected Topics in Signal Processing  
Our approach allows an information theoretic formulation of the multi-task learning (MTL) problem which is a supervised learning framework in which the prediction models for several related tasks are learned  ...  An important property of this algorithm is that it provides a natural safeguard against overfitting, because it minimizes the average risk taking into account a penalization induced by the model complexity  ...  We derived an iterative learning algorithm from the principle of compression-based regularization that uses compression as a natural safeguard against overfitting.  ... 
doi:10.1109/jstsp.2018.2846218 fatcat:psze6s2rnrcazcwb7bdsioxqqu
« Previous Showing results 1 — 15 out of 2,070 results