27,993 Hits in 4.2 sec

On the Maximum Hessian Eigenvalue and Generalization [article]

Simran Kaur, Jeremy Cohen, Zachary C. Lipton
2022 arXiv   pre-print
Other works question the link between λ_max and generalization. In this paper, we present findings that call λ_max's influence on generalization further into question.  ...  of the Hessian of the loss); and algorithms, such as Sharpness-Aware Minimization (SAM) [1], that directly optimize for flatness.  ...  In this paper, we focus exclusively on the leading Hessian eigenvalue λ max and its relationship to generalization. λ max as a metric for flatness The maximum Hessian eigenvalue λ max is commonly viewed  ... 
arXiv:2206.10654v1 fatcat:bd4l3cpfgrc63c3nbqxzlxadtu

On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length [article]

Stanisław Jastrzębski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, Amos Storkey
2019 arXiv   pre-print
by small eigenvalues of the Hessian of the training loss.  ...  in wider regions visited), the overall training speed, and the generalization ability of the final model.  ...  10 eigenvalues of the Hessian for SimpleCNN and Resnet-32 trained on the CIFAR-10 dataset with η = 0.1 and S = 128.  ... 
arXiv:1807.05031v6 fatcat:vyhiz37wh5de7mlbekuhtnjvba

Adaptable Center Detection of a Laser Line with a Normalization Approach using Hessian-matrix Eigenvalues

Guan Xu, Lina Sun, Xiaotao Li, Jian Su, Zhaobing Hao, Xue Lu
2014 Journal of the Optical Society of Korea  
The Gaussian recognition function estimates the characteristic that one eigenvalue approaches zero, and enhances the sensitivity of the decision function to that characteristic, which corresponds to the  ...  Second, the feature points in an integral pixel level are selected as the initiating light line centers by the eigenvalues of the Hessian matrix.  ...  Then the eigenvalues and eigenvectors of the Hessian matrix are calculated after obtaining the Hessian matrix of each pixel on the laser line.  ... 
doi:10.3807/josk.2014.18.4.317 fatcat:e5onyin5frgkbls4s7he2msw7e

On the saddle point problem for non-convex optimization [article]

Razvan Pascanu, Yann N. Dauphin, Surya Ganguli, Yoshua Bengio
2014 arXiv   pre-print
Here we argue, based on results from statistical physics, random matrix theory, and neural network theory, that a deeper and more profound difficulty originates from the proliferation of saddle points,  ...  Such saddle points are surrounded by high error plateaus that can dramatically slow down learning, and give the illusory impression of the existence of a local minimum.  ...  and CIFAR.  ... 
arXiv:1405.4604v2 fatcat:xnjcqly4mvbj5cn544fldwk5n4

Maximizing the hyperpolarizability poorly determines the potential [article]

T. J. Atherton and J. Lesnefsky and G. A. Wiggers and R. G. Petschek
2011 arXiv   pre-print
The Hessian of \beta at the maximum has in each case only two large eigenvalues; the other eigenvalues diminish seemingly exponentially quickly, demonstrating a very wide range of nearby nearly optimal  ...  potentials, and that there are only two important parameters for optimizing \beta.  ...  In each case, we optimized β int and calculated the eigenvalues and eigenvectors of the Hessian at the maximum.  ... 
arXiv:1010.4919v2 fatcat:k6ewoilwcfdjbndtxhkfutmizu

Experimental observation of saddle points over the quantum control landscape of a two-spin system

Qiuyang Sun, István Pelczer, Gregory Riviello, Re-Bing Wu, Herschel Rabitz
2015 Physical Review A. Atomic, Molecular, and Optical Physics  
We address the saddles with a combined theoretical and experimental approach, measure the Hessian at each identified saddle point, and study how their presence can influence the search effort utilizing  ...  Theoretical analyses have predicted the existence of critical points over the landscapes, including saddle points with indefinite Hessians.  ...  Another important conclusion is the low rank of the Hessian at critical points, i.e., there exists a specific maximum number of positive and negative eigenvalues dependent on the nature of the quantum  ... 
doi:10.1103/physreva.91.043412 fatcat:uvxcgdikyrf7jhefgqyf6kthvi

Matrix conditioning and adaptive simultaneous perturbation stochastic approximation method

Xun Zhu
2001 Proceedings of the 2001 American Control Conference. (Cat. No.01CH37148)  
This paper proposes a modification to the simultaneous perturbation stochastic approximation (SPSA) methods based on the comparisons made between the first order and the second order SPSA (1SPSA and 2SPSA  ...  At finite iterations, the convergence rate depends on the matrix conditioning of the loss function Hessian.  ...  For a symmetric Hessian matrix H with all positive eigenvalues, its condition number with respect to the spectral norm (~2) is the ratio of the maximum eigenvalue to the minimum one (Horn and Johnson,  ... 
doi:10.1109/acc.2001.945918 fatcat:fl3gzjj6qjeb3l7b2fw4svubzu

Strategies for walking on potential energy surfaces using local quadratic approximations

Jack Simons, Jeff Nichols
1990 International Journal of Quantum Chemistry  
This method utilizes local gradient and Hessian (Le., fust and second energy derivative) information to generate a series of "steps" that are folIowed to the desired stationary point.  ...  By stepping uphill along one local eigenmode of the Hessian wIDIe minimizing the energy along all other modes, one )ocates transition states.  ...  Acknowledgments This wark was supported in part by the Office of Naval Research and by NSF Grant No. 8814765;  ... 
doi:10.1002/qua.560382427 fatcat:yo5ctkl46restj5vqwxrpx3rru

Search for stationary points on surfaces

Ajit Banerjee, Noah Adams, Jack Simons, Ron Shepard
1985 The Journal of Physical Chemistry  
The data indicate that the S2ion of Na2S or H2S is strongly adsorbed on the CdS-Nafion surface and is oxidized to sulfate (Figures 4 and 6) without irradiation. Conclusions conclusions: is small.  ...  with cation exchange sites. (4) Adsorbed sulfide ions on Nafion and CdS-Nafion are oxidized to sulfate ions at 300 K in the presence of oxygen. (5) The gray-blue deposit formed on cubic CdS-Nafion surfaces  ...  We also acknowledge the financial support of the National Science Foundation (Grant 8206845) and the donors of the Petroleum Research Fund, administered by the American Chemical Society.  ... 
doi:10.1021/j100247a015 fatcat:y6zo3o6r6zc7tbwfqifo44t6oa

Characterizing the loss landscape of variational quantum circuits [article]

Patrick Huembeli, Alexandre Dauphin
2020 arXiv   pre-print
The eigenvalues of the Hessian give information on the local curvature and we discuss how this information can be interpreted and compared to classical neural networks.  ...  We benchmark our result on several examples, starting with a simple analytic toy model to provide some intuition about the behavior of the Hessian, then going to bigger circuits, and also train VQCs on  ...  II C, we study the Hessian and its eigenvalues on an analytical example of a VQC. We show how to calculate the Hessian on an actual quantum hardware in Sec. IV, apply it to a general example in Sec.  ... 
arXiv:2008.02785v1 fatcat:z3gow6fenjblpp27tkfn2eozfy

Molecular Embedding via a Second Order Dissimilarity Parameterized Approach

Ian G. Grooms, Robert Michael Lewis, Michael W. Trosset
2009 SIAM Journal on Scientific Computing  
The nonconvexity arises due to matrix rank constraints in the problem, and we focus on their efficient computational treatment.  ...  We present numerical results for a number of synthetic and real protein data sets and comment on features of real experimental data that can cause computational difficulties.  ...  The authors thank the reviewers of this paper; their comments have greatly improved the paper.  ... 
doi:10.1137/070688547 fatcat:7wq4civvlzez7a4crtjkvi55fy

Selecting the Metric in Hamiltonian Monte Carlo [article]

Ben Bales, Arya Pourzanjani, Aki Vehtari, Linda Petzold
2019 arXiv   pre-print
The effectiveness of the selection criterion and adaptation are demonstrated on a number of applied problems. An implementation for the Stan probabilistic programming language is provided.  ...  and the availability of warmup draws.  ...  For a normal posterior, Eq. 9 and Eq. 13 are equivalent and the smallest Hessian eigenvalue, λ min , corresponds to the largest covariance eigenvalue.  ... 
arXiv:1905.11916v3 fatcat:ovmikfyb35bedf3eexegk6w3ii

The Information Content of the Elastic Reflection Matrix

Bjørn Ursin, Egil Tjåland
1996 Geophysical Journal International  
These are the parameters associated with the eigenvectors of the Hessian matrix (or the normalized Fisher information matrix) corresponding to the smallest eigenvalues.  ...  S U M M A R Y The P-SV reflection matrix for a plane interface between two elastic media depends on the density, P-wave velocity, and S-wave velocity of the two media.  ...  The Hessian matrix depends on maximum slowness, the type of reflection coefficients, and the earth model used.  ... 
doi:10.1111/j.1365-246x.1996.tb06547.x fatcat:h7oqoe6wmjaija2ajeauhytbli

Multiscale vessel enhancement filtering [chapter]

Alejandro F. Frangi, Wiro J. Niessen, Koen L. Vincken, Max A. Viergever
1998 Lecture Notes in Computer Science  
A vesselness measure is obtained on the basis of all eigenvalues of the Hessian. This measure is tested on two dimensional DSA and three dimensional aortoiliac and cerebral MRA data.  ...  Its clinical utility is shown by the simultaneous noise and background suppression and vessel enhancement in maximum intensity projections and volumetric displays.  ...  Theo van Walsum, Onno Wink and Joes Staal for fruitful discussions and various contributions to the paper.  ... 
doi:10.1007/bfb0056195 fatcat:tqtoxxal7jebzi4b3rxyxsfhyq

A Strain Energy Filter for 3D Vessel Enhancement [chapter]

Changyan Xiao, Marius Staring, Denis Shamonin, Johan H. C. Reiber, Jan Stolk, Berend C. Stoel
2010 Lecture Notes in Computer Science  
Moreover, a mathematical description of Hessian eigenvalues for general vessel shapes is obtained, based on an intensity continuity assumption, and a relative Hessian strength term is presented to ensure  ...  The proposed method is validated in experiments with a digital phantom and non-contrast-enhanced pulmonary CT data.  ...  Acknowledgments This research was funded by STW (Grant LPG.07998) of the Netherlands and by the National Natural Science Foundation of China (Nos. 60835004 and 60871096).  ... 
doi:10.1007/978-3-642-15711-0_46 fatcat:nk2nl7t7vzg6zkavwfpnrghiba
« Previous Showing results 1 — 15 out of 27,993 results