Filters








138 Hits in 11.0 sec

No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets [article]

Fahim Tajwar, Ananya Kumar, Sang Michael Xie, Percy Liang
2021 arXiv   pre-print
Prior literature has proposed various methods (e.g., MSP (Hendrycks & Gimpel, 2017), ODIN (Liang et al., 2018), Mahalanobis (Lee et al., 2018)), claiming they are state-of-the-art by showing they outperform  ...  In this work, we show that none of these methods are inherently better at OOD detection than others on a standardized set of 16 (ID, OOD) pairs.  ...  No method does consistently better than any other method across the 16 pairs (none of the numbers are 0 or 16).  ... 
arXiv:2109.05554v1 fatcat:6uwo3vaj7ngjjjt3ilf2xeoqbq

DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling [article]

Akis Linardos, Matthias Kümmerer, Ori Press, Matthias Bethge
2021 arXiv   pre-print
This new model, "DeepGaze IIE", yields a significant leap in benchmark performance in and out-of-domain with a 15 percent point improvement over DeepGaze II to 93% on MIT1003, marking a new state of the  ...  art on the MIT/Tuebingen Saliency Benchmark in all available metrics (AUC: 88.3%, sAUC: 79.4%, CC: 82.4%).  ...  Our pairwise combination of models is already enough to beat the state of the art, while our final combination of four models with three instances each leads to an even higher leap on the state of the  ... 
arXiv:2105.12441v3 fatcat:srjm65isxnbt3imgri2vvcfdjm

Are all outliers alike? On Understanding the Diversity of Outliers for Detecting OODs [article]

Ramneet Kaur, Susmit Jha, Anirban Roy, Oleg Sokolsky, Insup Lee
2021 arXiv   pre-print
Deep neural networks (DNNs) are known to produce incorrect predictions with very high confidence on out-of-distribution (OOD) inputs.  ...  Our results include experiments on CIFAR10, SVHN and MNIST as in-distribution data and Imagenet, LSUN, SVHN (for CIFAR10), CIFAR10 (for SVHN), KMNIST, and F-MNIST as OOD data across different DNN architectures  ...  A.2.2 COMPARISON WITH THE STATE-OF-THE-ART OOD DETECTION METHODS IN SUPERVISED SETTINGS ON PRE-TRAINED CLASSIFIERS We compare our results with the state-of-the-art methods in supervised settings, as reported  ... 
arXiv:2103.12628v1 fatcat:ej4w6j2q4bae7mnm27aycc2mui

A Simple Framework for Robust Out-of-Distribution Detection

Youngbum Hur, Eunho Yang, Sung Ju Hwang
2022 IEEE Access  
The existing state-of-the-art methods of OOD detection tackle this issue by utilizing the internal feature of the classification network.  ...  Our experiments demonstrate the superiority of our simple method under various OOD detection scenarios.  ...  TOWARDS BETTER OOD BENCHMARKS This section points out that existing state-of-the-art methods [20] , [28] in out-of-distribution (OOD) detection do not generalize well on detecting hard OOD datasets  ... 
doi:10.1109/access.2022.3153723 fatcat:e6bodhr7fnblpejcyr42zoht2m

Norm-Scaling for Out-of-Distribution Detection [article]

Deepak Ravikumar, Kaushik Roy
2022 arXiv   pre-print
Out-of-Distribution (OoD) inputs are examples that do not belong to the true underlying distribution of the dataset.  ...  We show that norm-scaling, when used with maximum softmax probability detector, achieves 9.78% improvement in AUROC, 5.99% improvement in AUPR and 33.19% reduction in FPR95 metrics over previous state-of-the-art  ...  On average, norm-scaling on MSP achieves 9.78% improvement in AU-ROC, 5.99% improvement in AUPR and 33.19% reduction in FPR95 over previous state-of-the-art methods.  ... 
arXiv:2205.03493v1 fatcat:3l2mbtiqtfgsnoofwgabzdpxka

Confidence-based Out-of-Distribution Detection: A Comparative Study and Analysis [article]

Christoph Berger, Magdalini Paschali, Ben Glocker, Konstantinos Kamnitsas
2021 arXiv   pre-print
In this work, we assess the capability of various state-of-the-art approaches for confidence-based OOD detection through a comparative study and in-depth analysis.  ...  First, we leverage a computer vision benchmark to reproduce and compare multiple OOD detection methods.  ...  Conclusion This work presented an analysis of various state-of-the-art methods for confidencebased OOD detection on a computer vision and a medical imaging task.  ... 
arXiv:2107.02568v1 fatcat:26z53wikwjezvfkznxezke2g2i

Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer [article]

James Smith, Jonathan Balloch, Yen-Chang Hsu, Zsolt Kira
2021 arXiv   pre-print
Our approach, DistillMatch, increases performance over the state-of-the-art by no less than 8.7% average task accuracy and up to 54.5% average task accuracy in SSCL CIFAR-100 experiments.  ...  We show that a strategy built on pseudo-labeling, consistency regularization, Out-of-Distribution (OoD) detection, and knowledge distillation reduces forgetting in this setting.  ...  We show that state-of-the-art continual learning methods perform inconsistently in the SSCL setting (i.e. no baseline method is "best" across all settings). 2.  ... 
arXiv:2101.09536v2 fatcat:inyfzbclgfbw7mrej4cxh5q5ze

Estimating Predictive Uncertainty Under Program Data Distribution Shift [article]

Yufei Li, Simin Chen, Wei Yang
2021 arXiv   pre-print
We also propose a large-scale benchmark of existing state-of-the-art predictive uncertainty on programming tasks and investigate their effectiveness under data distribution shift.  ...  on program dataset.  ...  However, to our knowledge all the existing state-of-the-art work evaluates the effectiveness of their uncertainty methods on CV and NLP tasks.  ... 
arXiv:2107.10989v1 fatcat:vxjuenmwhngefioz7ja23o4gc4

Scaling Out-of-Distribution Detection for Real-World Settings [article]

Dan Hendrycks and Steven Basart and Mantas Mazeika and Andy Zou and Joe Kwon and Mohammadreza Mostajabi and Jacob Steinhardt and Dawn Song
2022 arXiv   pre-print
We conduct extensive experiments in these more realistic settings for out-of-distribution detection and find that a surprisingly simple detector based on the maximum logit outperforms prior methods in  ...  To test ImageNet multiclass anomaly detectors, we introduce the Species dataset containing over 700,000 images and over a thousand anomalous species.  ...  Assuming single-model evaluation and no access to other anomalies or test-time adaptation, the MSP attains state-of-the-art anomaly detection performance in small-scale settings.  ... 
arXiv:1911.11132v4 fatcat:6d235jsui5cjbciufvbxn65vcy

Detecting Out-of-distribution Samples via Variational Auto-encoder with Reliable Uncertainty Estimation [article]

Xuming Ran, Mingkun Xu, Lingrui Mei, Qi Xu, Quanying Liu
2021 arXiv   pre-print
Experiments on various datasets demonstrate that compared to the standard VAEs, our model is superior in uncertainty estimation for the OOD data and is robust in anomaly detection tasks.  ...  Variational autoencoders (VAEs) are influential generative models with rich representation capabilities from the deep neural network architecture and Bayesian method.  ...  Through a number of experiments on the challenging OOD cases, we demonstrate that INCPVAE can learn the true characterization of OOD inputs, and achieves state-of-the-art (SOTA) performance in OOD detection  ... 
arXiv:2007.08128v3 fatcat:ovh6tdtwwbgo7a7qbtv4z3sggq

Confidence from Invariance to Image Transformations [article]

Yuval Bahat, Gregory Shakhnarovich
2018 arXiv   pre-print
In addition, we apply our technique to novelty detection scenarios, where we also demonstrate state of the art results.  ...  In experiments with multiple data sets (STL-10,CIFAR-100,ImageNet) and classifiers, we demonstrate new state of the art for the error detection task.  ...  In addition, we apply our technique to novelty detection scenarios, where we also demonstrate state of the art results.  ... 
arXiv:1804.00657v1 fatcat:o3xdnjq4ljazjebxmq22hlilwm

Data-SUITE: Data-centric identification of in-distribution incongruous examples [article]

Nabeel Seedat, Jonathan Crabbé, Mihaela van der Schaar
2022 arXiv   pre-print
We also illustrate how these identified regions can provide insights into datasets and highlight their limitations.  ...  B.1.3 Deep Ensembles (ENSEMBLE) Deep Ensembles [Lakshminarayanan et al., 2017] is widely regarded as the state-of-the-art non-Bayesian uncertainty estimation method.  ...  For each of the detection methods, we compute the overlap between the predicted OOD/Outlier instances and the uncertain and inconsistent instances as identified by Data-SUITE.  ... 
arXiv:2202.08836v2 fatcat:vh7tzoy6jzcprc643osxo6qtce

Relevant-features based Auxiliary Cells for Energy Efficient Detection of Natural Errors [article]

Sai Aparna Aketi and Priyadarshini Panda and Kaushik Roy
2020 arXiv   pre-print
Deep neural networks have demonstrated state-of-the-art performance on many classification tasks. However, they have no inherent capability to recognize when their predictions are wrong.  ...  We demonstrate the effectiveness of our technique on various image classification datasets such as CIFAR-10, CIFAR-100 and Tiny-ImageNet.  ...  We demonstrate the effectiveness of our methodology to detect natural errors on state-of-the-art networks, such as VGG [18] and ResNet [5] for image classification tasks on CIFAR [1] and Tiny-ImageNet  ... 
arXiv:2002.11052v2 fatcat:ukmefarv55cwdpwducpbazdti4

Detecting Out-of-Distribution Examples with In-distribution Examples and Gram Matrices [article]

Chandramouli Shama Sastry, Sageev Oore
2020 arXiv   pre-print
performs better than or equal to state-of-the-art OOD detection methods (including those that do assume access to OOD examples).  ...  The method is applicable across a variety of architectures and vision datasets and, for the important and surprisingly hard task of detecting far-from-distribution out-of-distribution examples, it generally  ...  We have proposed and reported on a relatively simple OOD detection method based on pairwise feature correlations that gives new state of the art detection results without requiring access to anything other  ... 
arXiv:1912.12510v2 fatcat:7y34yyxenbfjxcblxlap2h7hze

Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration [article]

Yu Wang and Jingyang Lin and Jingjing Zou and Yingwei Pan and Ting Yao and Tao Mei
2021 arXiv   pre-print
We empirically show UOTA's advantage over the state-of-the-art self-supervised paradigms with evident margin, which well justifies the existence of the OOD sample issue embedded in the existing approaches  ...  Our work reveals a structured shortcoming of the existing mainstream self-supervised learning methods.  ...  Acknowledgments and Disclosure of Funding Funding in direct support of this work: the National Key R&D Program of China under Grant No. 2020AAA0108600.  ... 
arXiv:2112.08132v1 fatcat:swknm76jwfe43cofmr2zknyusu
« Previous Showing results 1 — 15 out of 138 results