2,935 Hits in 5.3 sec

A Consistently Rapid Algorithm for Solving Polynomial Equations

J. B. Moore
1976 IMA Journal of Applied Mathematics  
of polynomial equations with real or complex appears that in terms of algorithm complexity and calculation time, the new algorithm represents a considerable improvement on the various always  ...  ~ln v~~. a ide stepping kelahn <complex a), k is pos<~ble to avoid the saddle point region of F and thereby gain proximity to a polynomial zero.  ...  For multiple roots, acceleration steps are required in order to give rapid convergence.  ... 
doi:10.1093/imamat/17.1.99 fatcat:wbiftxfs6naspfdxl6tykc22ay

Learning Rate Dropout [article]

Huangxing Lin, Weihong Zeng, Xinghao Ding, Yue Huang, Chenxi Huang and John Paisley
2019 arXiv   pre-print
The uncertainty of the descent path helps the model avoid saddle points and bad local minima. Experiments show that LRD is surprisingly effective in accelerating training while preventing overfitting.  ...  As the learning rate of different parameters is dropped, the optimizer will sample a new loss descent path for the current update.  ...  Therefore, each update of LRD will still cause a decrease in the loss function according to the gradient descent theory [29, 36, 25] , while better avoiding saddle points.  ... 
arXiv:1912.00144v2 fatcat:drapxgb5uzdsfnlbwernlzloce

A Dual-Dimer Method for Training Physics-Constrained Neural Networks with Minimax Architecture [article]

Dehao Liu, Yan Wang
2021 arXiv   pre-print
The training of the PCNN-MM is searching the high-order saddle points of the objective function. A novel saddle point search algorithm called Dual-Dimer method is developed.  ...  However, the weights of different losses from data and physical constraints are adjusted empirically in PCNNs.  ...  Parameter is introduced in the algorithm to avoid the zero-division error. When the eigenvalue is close to zero, it means that the curvature is small and the saddle point almost degenerates.  ... 
arXiv:2005.00615v2 fatcat:vvv5a3slcrbjfp2gwcok7c5eoy

Identifying and attacking the saddle point problem in high-dimensional non-convex optimization [article]

Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, Yoshua Bengio
2014 arXiv   pre-print
Motivated by these arguments, we propose a new approach to second-order optimization, the saddle-free Newton method, that can rapidly escape high dimensional saddle points, unlike gradient descent and  ...  Such saddle points are surrounded by high error plateaus that can dramatically slow down learning, and give the illusory impression of the existence of a local minimum.  ...  A step of the gradient descent method always points in the right direction close to a saddle point (SGD in Fig. 2 ).  ... 
arXiv:1406.2572v1 fatcat:qnbfbkiohfdojeqqarayt7wmwm

Improved configuration space sampling: Langevin dynamics with alternative mobility

C. D. Chau, G. J. A. Sevink, J. G. E. M. Fraaije
2008 Journal of Chemical Physics  
Due to the analogy of the drift term in the Langevin equation and the update scheme in Newton's method, we expect accelerated dynamics or improved convergence in the convex part of the potential energy  ...  We employ a Broyden-Fletcher-Goldfarb-Shannon method for updating the local mobility matrix.  ...  We note that this is an important finding, as for N Ͼ 1 pathways that cross energy barriers ͑maxima of the function ⌽͒ can be avoided in favor of saddle points.  ... 
doi:10.1063/1.2943313 pmid:18601320 fatcat:wdy5zj5zqzgnbg3xy233a2ujwa

Distributed Hessian-Free Optimization for Deep Neural Network [article]

Xi He and Dheevatsa Mudigere and Mikhail Smelyanskiy and Martin Takáč
2017 arXiv   pre-print
However, due to non-covexity nature of the problem, it was observed that SGD slows down near saddle point.  ...  Recent empirical work claim that by detecting and escaping saddle point efficiently, it's more likely to improve training performance.  ...  Then the gradients are reduced and applied to update weights. We then have to make sure that after each iteration of SGD all weights are again synchronized.  ... 
arXiv:1606.00511v2 fatcat:7rhtl7merbgfzagu65fi3ljapi

Advances in Optimisation Algorithms and Techniques for Deep Learning

Chigozie Enyinna Nwankpa
2020 Advances in Science, Technology and Engineering Systems  
(saddle point), thereby making it difficult to continue alongside it's poor convergence speed.  ...  This causes the gradients to almost vanish at some point for very deep architectures [12] , thereby requiring some optimisation to achieve an adequate learning, without dead signals.  ... 
doi:10.25046/aj050570 fatcat:oxpzllc5ujdgtdhkwmpiyjatgm

Critical Point-Finding Methods Reveal Gradient-Flat Regions of Deep Network Losses [article]

Charles G. Frye, James Simon, Neha S. Wadia, Andrew Ligeralde, Michael R. DeWeese, Kristofer E. Bouchard
2020 arXiv   pre-print
network losses enjoy a no-bad-local-minima property and an abundance of saddle points.  ...  a stationary point.  ...  Looking back at Figure 4 of [11] , we see that their non-convex Newton method, a second-order optimization algorithm designed to avoid saddle points by reversing the Newton update along directions  ... 
arXiv:2003.10397v1 fatcat:27lhlhhn6jc2jm5bwx6gvq77uu

Fast X-Ray CT Image Reconstruction Using a Linearized Augmented Lagrangian Method With Ordered Subsets

Hung Nien, Jeffrey A. Fessler
2015 IEEE Transactions on Medical Imaging  
To further accelerate the proposed algorithm, we use a second-order recursive system analysis to design a deterministic downward continuation approach that avoids tedious parameter tuning and provides  ...  image when using many subsets for OS acceleration.  ...  The sequence of updates generated by the CP iterates (20) converges to a saddle-point of (18) , and the non-negative primal-dual gap converges to zero with rate [26, Th. 1] provided that , where and  ... 
doi:10.1109/tmi.2014.2358499 pmid:25248178 pmcid:PMC4315772 fatcat:7j4vat6wrfajbfkkyr6xh6yqr4

Large-Scale Phase Retrieval via Stochastic Reweighted Amplitude Flow

2020 KSII Transactions on Internet and Information Systems  
The second stage is the gradient refinement stage, which uses continuous updating of the amplitude-based stochastic weighted gradient algorithm to improve the initial estimate.  ...  Therefore, there are many saddle points in its search area, and the gradient iteration will most likely fall into the saddle point.  ...  In the gradient refinement stage, we use the weighted gradient method to update the estimated initial value, and the stochastic gradient method is used to accelerate the process.  ... 
doi:10.3837/tiis.2020.11.006 fatcat:coi53kkap5e5dpocyuph2nasai

Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces [article]

Sridhar Mahadevan, Bo Liu, Philip Thomas, Will Dabney, Steve Giguere, Nicholas Jacek, Ian Gemp, Ji Liu
2014 arXiv   pre-print
Our work builds extensively on recent work on the convergence of saddle-point algorithms, and on the theory of monotone operators.  ...  This allows temporal difference updates to occur in dual spaces, allowing a variety of important technical advantages.  ...  Convex-concave Saddle-Point First Order Algorithms A key novel contribution of our paper is a convex-concave saddle-point formulation for reinforcement learning.  ... 
arXiv:1405.6757v1 fatcat:u77kqc6iyncy7fixlnrfcnqrmy

Efficient Global Optimization of Non-differentiable, Symmetric Objectives for Multi Camera Placement [article]

Maria L. Hänel, Carola-B. Schönlieb
2021 arXiv   pre-print
We propose a novel iterative method for optimally placing and orienting multiple cameras in a 3D scene.  ...  We show how to accelerate the optimization by exploiting special properties of the objective function, such as symmetry.  ...  Is the saddle point the suboptimal stationary point we are looking for?  ... 
arXiv:2103.11210v1 fatcat:32wj3tf4prah5ilcxnrzyhmbz4

Avoidance of Convex and Concave Obstacles With Convergence Ensured Through Contraction

Lukas Huber, Aude Billard, Jean-Jacques Slotine
2019 IEEE Robotics and Automation Letters  
This paper presents a closed-form approach to obstacle avoidance for multiple moving convex and star-shaped concave obstacles. The method takes inspiration in harmonicpotential fields.  ...  We validate the approach in a simulated co-worker industrial environment, with one KUKA arm engaged in a pick and place grocery task, avoiding in real-time humans moving in its vicinity and in simulation  ...  MULTIPLE OBSTACLES In the presence of multiple obstacles, the nominal DS is modified by taking the weighted mean of the modulated DṠ the effect of each obstacle and to ensure that one the boundary of each  ... 
doi:10.1109/lra.2019.2893676 fatcat:x342gyhfajgnpiq775ozby3rc4

An overview of gradient descent optimization algorithms [article]

Sebastian Ruder
2017 arXiv   pre-print
[5] argue that the difficulty arises in fact not from local minima but from saddle points, i.e. points where one dimension slopes up and another slopes down.  ...  Figure 4b shows the behaviour of the algorithms at a saddle point, i.e. a point where one dimension has a positive slope, while the other dimension has a negative slope, which pose a difficulty for SGD  ... 
arXiv:1609.04747v2 fatcat:xobv3n2ljvfw5lrmn4ivlus6aa

Joint Synthesis of Safety Certificate and Safe Control Policy using Constrained Reinforcement Learning [article]

Haitong Ma, Changliu Liu, Shengbo Eben Li, Sifa Zheng, Jianyu Chen
2022 arXiv   pre-print
We evaluate our algorithms on multiple safety-critical benchmark environments. The results show that the proposed algorithm learns provably safe policies with no constraint violation.  ...  A valid safety certificate is an energy function indicating that safe states are with low energy, and there exists a corresponding safe control policy that allows the energy function to always dissipate  ...  First we show that each update of the multi-time scale discrete stochastic approximation algorithm (θ k , ξ k , ζ k ) converges almost surely, but at different speeds, to the stationary point (θ * , ξ  ... 
arXiv:2111.07695v3 fatcat:s5cctimxvvhfdpishnmzjmi2iq
« Previous Showing results 1 — 15 out of 2,935 results