14,565 Hits in 3.4 sec

Contrastive Out-of-Distribution Detection for Pretrained Transformers [article]

Wenxuan Zhou, Fangyu Liu, Muhao Chen
2021 arXiv   pre-print
Pretrained Transformers achieve remarkable performance when training and test data are from the same distribution.  ...  However, in real-world scenarios, the model often faces out-of-distribution (OOD) instances that can cause severe semantic shift problems at inference time.  ...  Acknowledgment We appreciate the anonymous reviewers for their insightful comments and suggestions. This material is supported by the National Science Foundation of United States Grant IIS 2105329.  ... 
arXiv:2104.08812v2 fatcat:gyh5a64psvaydekm6tu2xo6squ

Contrastive Out-of-Distribution Detection for Pretrained Transformers

Wenxuan Zhou, Fangyu Liu, Muhao Chen
2021 Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing   unpublished
Pretrained Transformers achieve remarkable performance when training and test data are from the same distribution.  ...  We propose to fine-tune the Transformers with a contrastive loss, which improves the compactness of representations, such that OOD instances can be better differentiated from ID ones.  ...  Acknowledgment We appreciate the anonymous reviewers for their insightful comments and suggestions. This material is supported by the National Science Foundation of United States Grant IIS 2105329.  ... 
doi:10.18653/v1/2021.emnlp-main.84 fatcat:rmiosrfvkvcbpnvda4c575faau

Pretrained Transformers Improve Out-of-Distribution Robustness [article]

Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, Dawn Song
2020 arXiv   pre-print
We systematically measure out-of-distribution (OOD) generalization for seven NLP datasets by constructing a new robustness benchmark with realistic distribution shifts.  ...  Although pretrained Transformers such as BERT achieve high accuracy on in-distribution examples, do they generalize to new distributions?  ...  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.  ... 
arXiv:2004.06100v2 fatcat:2636tz4hcndxvmyfy6dcp2xtri

Self-Supervised Anomaly Detection by Self-Distillation and Negative Sampling [article]

Nima Rafiee, Rahil Gholamipoorfard, Nikolas Adaloglou, Simon Jaxy, Julius Ramakers, Markus Kollmann
2022 arXiv   pre-print
Detecting whether examples belong to a given in-distribution or are Out-Of-Distribution (OOD) requires identifying features specific to the in-distribution.  ...  In this work, we show that self-distillation of the in-distribution training set together with contrasting against negative examples derived from shifting transformation of auxiliary data strongly improves  ...  These kinds of large-scale pretrained models heavily rely on the classes of the pretraining dataset, which often include classes from both the in and out distribution.  ... 
arXiv:2201.06378v1 fatcat:e23qpsd3kjcslmcqfr2l7yomae

Temperature as Uncertainty in Contrastive Learning [article]

Oliver Zhang, Mike Wu, Jasmine Bayrooti, Noah Goodman
2021 arXiv   pre-print
Through experiments, we demonstrate that TaU is useful for out-of-distribution detection, while remaining competitive with benchmarks on linear evaluation.  ...  Moreover, we show that TaU can be learned on top of pretrained models, enabling uncertainty scores to be generated post-hoc with popular off-the-shelf models.  ...  Out-of-distribution AUROC is reported.  ... 
arXiv:2110.04403v1 fatcat:zs3r6wjnwbfxlgrpgp75ahb6k4

Revisiting Pretraining for Semi-Supervised Learning in the Low-Label Regime [article]

Xun Xu, Jingyi Liao, Lile Cai, Manh Cuong Nguyen, Kangkang Lu, Wanyue Zhang, Yasin Yazici, Chuan Sheng Foo
2022 arXiv   pre-print
The promising results validate the efficacy of target pretraining for SSL, in particular in the low-label regime.  ...  In this work, we first show that the better pretrained weights brought in by FT account for the state-of-the-art performance, and importantly that they are universally helpful to off-the-shelf semi-supervised  ...  For a dense prediction task, [34] developed a contrastive learning paradigm by comparing pixel-wise features and it turns out to be effective for detection and segmentation tasks.  ... 
arXiv:2205.03001v1 fatcat:2gw4pt6sirg3rngki7rcuas5ka

Toward Transformer-Based Object Detection [article]

Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew Zhai, Dmitry Kislyuk
2020 arXiv   pre-print
We also investigate improvements over a standard detection backbone, including superior performance on out-of-domain images, better performance on large objects, and a lessened reliance on non-maximum  ...  We view ViT-FRCNN as an important stepping stone toward a pure-transformer solution of complex vision tasks such as object detection.  ...  Acknowledgements We thank Kofi Boakye, Vahid Kazemi, and Chuck Rosenberg for valuable discussions regarding the paper.  ... 
arXiv:2012.09958v1 fatcat:rhdftpryhfbrdo6npq7e4vfuse

Masked Discrimination for Self-Supervised Learning on Point Clouds [article]

Haotian Liu, Mu Cai, Yong Jae Lee
2022 arXiv   pre-print
a significant pretraining speedup (e.g., 4.1x on ScanNet) compared to the prior state-of-the-art Transformer baseline.  ...  In this paper, we bridge this gap by proposing a discriminative mask pretraining Transformer framework, MaskPoint}, for point clouds.  ...  We thank Xumin Yu for the helpful discussion in reproducing the Point-BERT baselines.  ... 
arXiv:2203.11183v2 fatcat:s3s5niuea5b2rbe4vqjaeivnee

An Empirical Investigation of Contextualized Number Prediction [article]

Daniel Spokoyny, Taylor Berg-Kirkpatrick
2020 arXiv   pre-print
Specifically, we introduce a suite of output distribution parameterizations that incorporate latent variables to add expressivity and better fit the natural distribution of numeric values in running text  ...  We experiment with novel combinations of contextual encoders and output distributions over the real number line.  ...  Findings and observations do not necessarily reflect the views of funding agencies.  ... 
arXiv:2011.07961v1 fatcat:vc2s5k7ed5ghpn4yupwdyttdhe

EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answering [article]

Violetta Shevchenko, Ehsan Abbasnejad, Anthony Dick, Anton van den Hengel, Damien Teney
2022 arXiv   pre-print
Although EBMs prove useful for OOD detection, other results on supervised energy-based training and uncertainty calibration are largely negative.  ...  The availability of clean and diverse labeled data is a major roadblock for training models on complex tasks such as visual question answering (VQA).  ...  Amount of pretraining data To better understand the benefits of pretraining in lowdata regimes, we repeat our experiments while decreasing Out-of-distribution detection OOD Detection is relevant to VQA  ... 
arXiv:2206.14355v1 fatcat:e3bhiwdjbrdgbn22w2ul6r6cca

A Study on Self-Supervised Object Detection Pretraining [article]

Trung Dang, Simon Kornblith, Huy Thong Nguyen, Peter Chin, Maryam Khademi
2022 arXiv   pre-print
In this work, we study different approaches to self-supervised pretraining of object detection models.  ...  using a transformer, which potentially benefits downstream object detection tasks.  ...  At the end of this section, we compare the proposed framework with a number of existing pretraining techniques for object detection.  ... 
arXiv:2207.04186v1 fatcat:sdog5lz2lbdlji4vtekv2xhhhq

Fully Convolutional Cross-Scale-Flows for Image-based Defect Detection [article]

Marco Rudolph, Tom Wehrbein, Bodo Rosenhahn, Bastian Wandt
2021 arXiv   pre-print
Our work sets a new state-of-the-art in image-level defect detection on the benchmark datasets Magnetic Tile Defects and MVTec AD showing a 100% AUROC on 4 out of 15 classes.  ...  Using normalizing flows to assign meaningful likelihoods to input samples allows for efficient defect detection on image-level.  ...  The problem is interpreted in terms of whether a data sample lies out of the distribution p X of the set of normal images X, also named out-of-distribution (OOD) detection.  ... 
arXiv:2110.02855v1 fatcat:zb5nmwcpwjdgtnisshxsjrdk4m

Metadata-enhanced contrastive learning from retinal optical coherence tomography images [article]

Robbie Holland, Oliver Leingang, Hrvoje Bogunović, Sophie Riedl, Lars Fritsche, Toby Prevost, Hendrik P. N. Scholl, Ursula Schmidt-Erfurth, Sobha Sivaprasad, Andrew J. Lotery, Daniel Rueckert, Martin J. Menten
2022 arXiv   pre-print
By leveraging this often neglected information our metadata-enhanced contrastive pretraining leads to further benefits and outperforms conventional contrastive methods in five out of seven downstream tasks  ...  Several of the image transformations used to create positive contrastive pairs are not applicable to greyscale medical scans.  ...  Metadata-enhanced pretraining further boosts performance In six out of seven tasks our metadata-enhanced pretraining strategies outperformed standard contrastive pretraining on the Southampton dataset  ... 
arXiv:2208.02529v1 fatcat:75bf3zabinh3fkgur7qxlvjm5q

Same Same But DifferNet: Semi-Supervised Defect Detection with Normalizing Flows [article]

Marco Rudolph and Bastian Wandt and Bodo Rosenhahn
2020 arXiv   pre-print
The detection of manufacturing errors is crucial in fabrication processes to ensure product quality and safety standards.  ...  In contrast to most other methods, ours does not require a large number of training samples and performs well with as low as 16 images.  ...  Acknowledgements This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy within the Cluster of Excellence PhoenixD (EXC 2122).  ... 
arXiv:2008.12577v1 fatcat:qbfwnkpjwfhchmuwufs52dptte

How Transferable Are Self-supervised Features in Medical Image Classification Tasks? [article]

Tuan Truong, Sadegh Mohammadi, Matthias Lenga
2021 arXiv   pre-print
tumor classification task, 7.03% AUC in the pneumonia detection, and 9.4% in AUC in the detection of pathological conditions in chest X-ray.  ...  to any combination of pretrained models.  ...  = self . dropout ( out ) 37 out = self . classifier ( out ) 38 return out 2  ... 
arXiv:2108.10048v3 fatcat:jucst3yv5ndn7n36eaoj3x6j7y
« Previous Showing results 1 — 15 out of 14,565 results