A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
Filters
Nonconvex Variance Reduced Optimization with Arbitrary Sampling
[article]
2019
arXiv
pre-print
Preserved Fulltext
Other Versions
We provide the first importance sampling variants of variance reduced algorithms for empirical risk minimization with non-convex loss functions. ...
All the above results follow from a general analysis of the methods which works with arbitrary sampling, i.e., fully general randomized strategy for the selection of subsets of examples to be sampled in ...
Variance-reduced methods A particularly important recent advance has to do with the design of variance-reduced (VR) stochastic gradient methods, such as SAG (Roux et al., 2012) , SDCA (Shalev-Shwartz ...
arXiv:1809.04146v2
fatcat:vajcltes5nenzbhd23fjsswuem
Momentum Schemes with Stochastic Variance Reduction for Nonconvex Composite Optimization
[article]
2019
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
Note that this fulltext copy is not of the "primary" version of this work. The version it corresponds to is:
2019 pre-print arXiv:1902.02715v2 fatcat:adbhnhu3jjgh3kjqpbzrcjtnguOther Versions
Two new stochastic variance-reduced algorithms named SARAH and SPIDER have been recently proposed, and SPIDER has been shown to achieve a near-optimal gradient oracle complexity for nonconvex optimization ...
However, existing momentum schemes used in variance-reduced algorithms are designed specifically for convex optimization, and are not applicable to nonconvex scenarios. ...
To the best of our knowledge, this is the first known theoretical guarantee for stochastic variance-reduced type of algorithms with momentum in nonconvex optimization. ...
arXiv:1902.02715v3
fatcat:arhpvqvorngv3pqlktcslgkbbu
Global Convergence and Variance Reduction for a Class of Nonconvex-Nonconcave Minimax Problems
2020
Neural Information Processing Systems
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2022; you can also visit the original URL.
The file type is application/pdf.
We further develop a variance reduced algorithm that attains a provably faster rate than AGDA when the problem has the finite-sum structure. ...
Yet, it is known that these vanilla GDA algorithms with constant stepsize can potentially diverge even in the convex-concave setting. ...
Variance reduced algorithm. ...
dblp:conf/nips/YangKH20
fatcat:buszxqnfkjh3pbjwdvtathpx5m
ZEROTH-ORDER STOCHASTIC PROJECTED GRADIENT DESCENT FOR NONCONVEX OPTIMIZATION
2018
2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP)
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2018; you can also visit the original URL.
The file type is application/pdf.
Contributions We propose and evaluate a novel ZO algorithm for nonconvex stochastic optimization, ZO-SVRG, which integrates SVRG with ZO gradient estimators. ...
This paper addresses these challenges by presenting: a) a comprehensive theoretical analysis of variance reduced zeroth-order (ZO) optimization, b) a novel variance reduced ZO algorithm, called ZO-SVRG ...
In this paper, we study the problem of design and analysis of variance reduced and faster converging nonconvex ZO optimization methods. ...
doi:10.1109/globalsip.2018.8646618
dblp:conf/globalsip/0001LCHA18
fatcat:23chudlyinh7pobzqi6ycds6di
Convergence Analysis of Proximal Gradient with Momentum for Nonconvex Optimization
[article]
2017
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
We further propose a stochastic variance reduced APGnc (SVRG-APGnc), and establish its linear convergence under a special case of the property. ...
Due to the intractability of nonconvexity, there is a rising need to develop efficient methods for solving general nonconvex problems with certain performance guarantee. ...
Convergence Analysis of Proximal Gradient with Momentum for Nonconvex Optimization ...
arXiv:1705.04925v1
fatcat:zkcczurf7fb23fw6c5echbf4ce
Generalization Error Bounds for Optimization Algorithms via Stability
2017
PROCEEDINGS OF THE THIRTIETH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE AND THE TWENTY-EIGHTH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2022; you can also visit the original URL.
The file type is application/pdf.
and stochastic variance reduction (SVRG). ...
In particular, we decompose the generalization error for R-ERM, and derive its upper bound for both convex and nonconvex cases. ...
In order to reduce the variance in SGD, SVRG divides the optimization process into multiple stages and updates the model towards a direction of the gradient at a randomly sampled instance regularized by ...
doi:10.1609/aaai.v31i1.10919
fatcat:vx4wzrfskbcqjd6f77djirstoi
Distributed black-box optimization of nonconvex functions
2015
2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2018; you can also visit the original URL.
The file type is application/pdf.
We combine model-based methods and distributed stochastic approximation to propose a fully distributed algorithm for nonconvex optimization, with good empirical performance and convergence guarantees. ...
This is especially relevant when many samples are needed (e.g., for high-dimensional objectives) or when evaluating each sample is costly. ...
Together with the robustness, reduced bias and variance, increased stability against node failure and intrinsic privacy (due to never exchanging local samples) of the diffusion-like distributed algorithms ...
doi:10.1109/icassp.2015.7178640
dblp:conf/icassp/MacuaZZ15
fatcat:vmzzvmrpvzbmdbcnnh7hwwnxf4
Sample Efficient Stochastic Variance-Reduced Cubic Regularization Method
[article]
2018
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
We propose a sample efficient stochastic variance-reduced cubic regularization (Lite-SVRC) algorithm for finding the local minimum efficiently in nonconvex optimization. ...
Numerical experiments with different nonconvex optimization problems conducted on real datasets validate our theoretical results. ...
Previous analysis of variance-reduced method in nonconvex setting for first-order method which only focus on the upper bound of variance of e v gives E e v 2 2 an upper bound only associated with x s t ...
arXiv:1811.11989v1
fatcat:74p66hp2l5aylhowrk5srdrxlu
Generalization Error Bounds for Optimization Algorithms via Stability
[article]
2016
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2019; you can also visit the original URL.
The file type is application/pdf.
and stochastic variance reduction (SVRG). ...
Our theorems indicate that 1) along with the training process, the generalization error will decrease for all the optimization algorithms under our investigation; 2) Comparatively speaking, SVRG has better ...
In order to reduce the variance in SGD, SVRG divides the optimization process into multiple stages and updates the model towards a direction of the gradient at a randomly sampled instance regularized by ...
arXiv:1609.08397v1
fatcat:xk4edyvppfaytmqn34dhuufile
Global Optimization Methods for Extended Fisher Discriminant Analysis
2014
International Conference on Artificial Intelligence and Statistics
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2022; you can also visit the original URL.
The file type is application/pdf.
We speed up the algorithm by taking advantage of the matrix structure and proving that a globally optimal solution is a KKT point with the smallest Lagrange multiplier, which can be computed efficiently ...
Numerical experiments illustrate the efficiency and effectiveness of the extended FDA model combined with our algorithm. ...
of training samples with labels +1 and −1, respectively. ...
dblp:conf/aistats/IwataNT14
fatcat:3wv4gftanzaetcnzq5yie6i6ly
Client Selection in Nonconvex Federated Learning: Improved Convergence Analysis for Optimal Unbiased Sampling Strategy
[article]
2022
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2022; you can also visit the original URL.
The file type is application/pdf.
FedSRC-D is provable the optimal unbiased sampling in non-convex settings for non-IID FL with respect to the given bounds. ...
Moreover, based on our convergence analysis, we give a novel unbiased sampling strategy, i.e., FedSRC-D, whose sampling probability is proportional to the client's gradient diversity and local variance ...
FedSRC-D reduces variance terms more efficiently since it can reduce all variance terms in Φ, while in Φ only last term V ar( 1 mpi ĝi ) can be optimized w.r.t p i . ...
arXiv:2205.13925v1
fatcat:m3tlh7oufjgzzan7galm2bqzoy
Sparse Blind Deconvolution with Nonconvex Optimization for Ultrasonic NDT Application
2020
Sensors
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
This letter introduces the l1/l2 ratio regularization function to model the deconvolution as a nonconvex optimization problem. ...
Compared with conventional blind deconvolution algorithms, the proposed methods demonstrate the robustness and capability of separating overlapping echoes in the context of synthetic experiments. ...
The additive noise is considered as white Gaussian noise with the variance . ...
doi:10.3390/s20236946
pmid:33291739
pmcid:PMC7730569
fatcat:7agcr4t35fhqpp3x3x2zkone5y
Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization
[article]
2020
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
Other Versions
Our theoretical analyses shed some light on using Langevin dynamics based algorithms for nonconvex optimization with provable guarantees. ...
We present a unified framework to analyze the global convergence of Langevin dynamics based algorithms for nonconvex finite-sum optimization with n component functions. ...
variance in SGLD, which can be reduced with larger batch size B. ...
arXiv:1707.06618v3
fatcat:ron6imxljbgwrelspvctnkyuey
ADMM-based Adaptive Sampling Strategy for Nonholonomic Mobile Robotic Sensor Networks
2021
IEEE Sensors Journal
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2022; you can also visit the original URL.
The file type is application/pdf.
In order to tackle the nonlinearity and nonconvexity of the objective function in the optimization problem we first exploit the linearized alternating direction method of multipliers (L-ADMM) method that ...
The control, movement and nonholonomic dynamics constraints of the mobile sensors are also considered in the adaptive sampling optimization problem. ...
The proposed approach can significantly reduces the computation time in solving the nonconvex optimization problem (10a) as compared with the L-ADMM algorithm. ...
doi:10.1109/jsen.2021.3072390
fatcat:z4ttj7bexnf35c6pck7qbn77ta
Robust adaptive beamforming using worst-case performance optimization: a solution to the signal mismatch problem
2003
IEEE Transactions on Signal Processing
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL.
The file type is application/pdf.
Computer simulations with several frequently encountered types of signal steering vector mismatches show better performance of our robust beamformer as compared with existing adaptive beamforming algorithms ...
The similar type of performance degradation can occur when the signal steering vector is known exactly but the training sample size is small. ...
CONCLUSIONS A new adaptive beamformer with an improved robustness against an arbitrary unknown signal steering vector mismatch has been proposed. ...
doi:10.1109/tsp.2002.806865
fatcat:kswftf4hnndp5mpmrhubmtrly4
« Previous
Showing results 1 — 15 out of 4,409 results