Filters








227,846 Hits in 4.0 sec

Gradient Methods Never Overfit On Separable Data [article]

Ohad Shamir
2020 arXiv   pre-print
In this paper, we formally show that standard gradient methods (in particular, gradient flow, gradient descent and stochastic gradient descent) never overfit on separable data: If we run these methods  ...  As a consequence, the predictors asymptotically do not overfit.  ...  For realistically small values of γ, this bound on T is unacceptably large. Could it be that gradient methods do not overfit only after so many iterations?  ... 
arXiv:2007.00028v2 fatcat:3svpzc7wvjapna4372bslkvc6u

Covariance among independent variables determines the overfitting and underfitting problems in variation partitioning methods: with a special focus on the mixed co-variation [article]

Youhua Chen
2014 arXiv   pre-print
Therefore, we analyzed the role of slight covariance on influencing species variation partitioning.  ...  The effectiveness and validity of applying variation partitioning methods in community ecology has been questioned.  ...  Because most of variation partitioning methods are similar, we only considered the simplest method-redundancy analysis (in our model of course, there is only one response variable, thus the method was  ... 
arXiv:1402.3324v1 fatcat:himzhckmsjbexcb66yvijvxgqy

VECA: A Method for Detecting Overfitting in Neural Networks (Student Abstract)

Liangzhu Ge, Yuexian Hou, Yaju Jiang, Shuai Yao, Chao Yang
2020 PROCEEDINGS OF THE THIRTIETH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE AND THE TWENTY-EIGHTH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE  
Despite their widespread applications, deep neural networks often tend to overfit the training data.  ...  Experiments performed on fully-connected networks and convolutional neural networks trained on benchmark image datasets show a strong correlation between test loss and VECA, which suggest that we can calculate  ...  Experimental Results To evaluate the effectiveness of VECA for detecting overfitting, we test our method on two kinds of networks: fully connected networks and convolutional networks.  ... 
doi:10.1609/aaai.v34i10.7167 fatcat:3nayylbhwbbqrbgju4dxzbx7bm

Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks [article]

Claudio Filipi Gonçalves dos Santos, João Paulo Papa
2022 pre-print
A critical factor in training concerns the network's regularization, which prevents the structure from overfitting.  ...  This work analyzes several regularization methods developed in the last few years, showing significant improvements for different CNN models.  ...  One key aspect of the regularization methods, independent of the training phase it works, is to prevent the model from overfitting the training data.  ... 
doi:10.1145/3510413 arXiv:2201.03299v1 fatcat:koik7yi4qjdqpp4hjk2jcc4vtq

Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Models

Robert Wolfe, Aylin Caliskan
2021 Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing   unpublished
Representations of infrequent names undergo more processing, but are more self-similar, indicating that models rely on less context-informed representations of uncommon and minority names which are overfit  ...  We use a dataset of U.S. first names with labels based on predominant gender and racial group to examine the effect of training corpus frequency on tokenization, contextualization, similarity to initial  ...  We use the ValNorm method of Toney-Wails and to select an intermediate layer from which to extract contextualized word embeddings. ValNorm evaluates semantic quality based on the single-value WEAT.  ... 
doi:10.18653/v1/2021.emnlp-main.41 fatcat:5razblwlwffcdmiaglfk5oajhm

Sequence Length is a Domain: Length-based Overfitting in Transformer Models

Dusan Varis, Ondřej Bojar
2021 Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing   unpublished
ing the length overfitting argument.  ...  suffer from overfitting during training.  ... 
doi:10.18653/v1/2021.emnlp-main.650 fatcat:whrt7lno2fcdhietfvzq2v5wxa

Covariance among independent variables determines the overfitting and underfitting in variation partitioning methods: with a focus on the mixed co-variation

Youhua Chen
2014 Computational Ecology and Software   unpublished
Therefore, we analyzed the role of slight covariance on influencing species variation partitioning.  ...  The effectiveness and validity of applying variation partitioning methods in community ecology has been questioned.  ...  Appendix 1 Mathematical deduction of overfitting and underfitting problems in three-step variation partitioning methods The full model when both spatial and environmental descriptors are necessary predictors  ... 
fatcat:2smvlkmzdnff3blbwchskyezqy

Moving away from semantic overfitting in disambiguation datasets

Marten Postma, Filip Ilievski, Piek Vossen, Marieke van Erp
2016 Proceedings of the Workshop on Uphill Battles in Language Processing: Scaling Early Achievements to Robust Methods   unpublished
To perform well, systems would have to show deep understanding on the linguistic tail.  ...  By following our four-step acquisition strategy and by using the active learning method, we expect to obtain a high accuracy on EvC.  ...  This has encouraged systems to overfit on the head and largely ignore the linguistic tail.  ... 
doi:10.18653/v1/w16-6004 fatcat:6je6eholijhv5muvokropifesa

Effect on model performance of regularization methods

Cafer BUDAK, Vasfiye MENÇİK, Mehmet Emin ASKER
2021 DÜMF Mühendislik Dergisi  
normalization, L1 and L2 regularization methods and the change in loss function the combination with these methods.  ...  Nonetheless, overfitting is a crucial problem in such networks.  ...  Conclusion In this study, different normalization methods and the effect of combinations of these methods on model performance were investigated to prevent overfitting.  ... 
doi:10.24012/dumf.1051352 fatcat:5mifrckeezewfhmlehrsmo73qm

Does overfitting affect performance in estimation of distribution algorithms

Hao Wu, Jonathan L. Shapiro
2006 Proceedings of the 8th annual conference on Genetic and evolutionary computation - GECCO '06  
What is found is: overfitting does occur in EDAs; overfitting correlates to EDAs performance; reduction of overfitting using early stopping can improve EDAs performance.  ...  The purpose of this paper is to investigate whether overfitting happens in EDAs, and to discover its consequences.  ...  REDUCING OVERFITTING TO IMPROVE EDAS PERFORMANCE In this section, one of most widely used methods, early stopping, is chosen to reduce overfitting.  ... 
doi:10.1145/1143997.1144078 dblp:conf/gecco/WuS06 fatcat:ltmsl5rx5zfxpk43ad7atxnuqa

Overfitting detection and adaptive covariant parsimony pressure for symbolic regression

Gabriel Kronberger, Michael Kommenda, Michael Affenzeller
2011 Proceedings of the 13th annual conference companion on Genetic and evolutionary computation - GECCO '11  
The method is based on the assumption that overfitting can be reduced by controlling the evolution of program length.  ...  Additionally, we propose an overfitting detection criterion that is based on the correlation of the fitness values on the training set and a validation set of all models in the population.  ...  Other approaches that are often used to control overfitting and improve generalization in statistical learning methods are based either on the estimation of the expected generalization error or on penalization  ... 
doi:10.1145/2001858.2002060 dblp:conf/gecco/KronbergerKA11 fatcat:zyfo2h6owrcmblzi357q5i2hcq

Identifying Incorrect Patches in Program Repair Based on Meaning of Source Code

Quang-Ngoc Phung, Misoo Kim, Eunseok Lee
2022 IEEE Access  
This paper proposes MIPI, a novel approach to reducing the number of overfitting patches generated in the APR.  ...  However, they often generate many overfitting patches which pass only a specific test-suite but do not fix the bugs correctly.  ...  patches (by the APCA method) that are actually overfitting.  ... 
doi:10.1109/access.2022.3145983 fatcat:7eoeodpet5cf3czgjcf563hpsa

Adversarial Examples on Segmentation Models Can be Easy to Transfer [article]

Jindong Gu, Hengshuang Zhao, Volker Tresp, Philip Torr
2021 arXiv   pre-print
First, we explore the overfitting phenomenon of adversarial examples on classification and segmentation models.  ...  overfit the source models.  ...  When Dynamic-Scale attack is applied, no overfitting is observed on AEs even on the source model with ResNet backbone. Figure 8 8 Figure 8. Ablation study of scaling ratio of DS attack method.  ... 
arXiv:2111.11368v1 fatcat:kjb4daqyq5ec5bajwjg3rmqdky

Understanding Catastrophic Overfitting in Single-step Adversarial Training

Hoki Kim, Woojin Lee, Jaewook Lee
2021 PROCEEDINGS OF THE THIRTIETH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE AND THE TWENTY-EIGHTH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE  
Based on this observation, we propose a simple method that not only prevents catastrophic overfitting, but also overrides the belief that it is difficult to prevent multi-step adversarial attacks with  ...  Although fast adversarial training has demonstrated both robustness and efficiency, the problem of "catastrophic overfitting" has been observed.  ...  The Thirty-Fifth AAAI Conference on Artificial Intelligence In this regard, few attempts have been made to discover the underlying reason for catastrophic overfitting and methods proposed to prevent this  ... 
doi:10.1609/aaai.v35i9.16989 fatcat:q4kjio5gm5gtpmrhnugef4oejq

Measuring Overfitting in Convolutional Neural Networks using Adversarial Perturbations and Label Noise [article]

Svetlana Pavlitskaya, Joël Oswald, J.Marius Zöllner
2022 arXiv   pre-print
Although numerous methods to reduce the overfitting of convolutional neural networks (CNNs) exist, it is still not clear how to confidently measure the degree of overfitting.  ...  Based on this, we define two new metrics that can confidently distinguish between correct and overfitted models.  ...  The authors show, that their method helps to separate correct models from overfitted ones -the detailed evaluation on neural networks is, however, missing. Second, Werpachowski et al.  ... 
arXiv:2209.13382v1 fatcat:fil2si5klzfslcd6tmachzhhpq
« Previous Showing results 1 — 15 out of 227,846 results