A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2021; you can also visit the original URL.
The file type is application/pdf.
Filters
Contrastive Out-of-Distribution Detection for Pretrained Transformers
[article]
2021
arXiv
pre-print
Preserved Fulltext
Other Versions
Pretrained Transformers achieve remarkable performance when training and test data are from the same distribution. ...
However, in real-world scenarios, the model often faces out-of-distribution (OOD) instances that can cause severe semantic shift problems at inference time. ...
Acknowledgment We appreciate the anonymous reviewers for their insightful comments and suggestions. This material is supported by the National Science Foundation of United States Grant IIS 2105329. ...
arXiv:2104.08812v2
fatcat:gyh5a64psvaydekm6tu2xo6squ
Contrastive Out-of-Distribution Detection for Pretrained Transformers
2021
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
unpublished
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2022; you can also visit the original URL.
The file type is application/pdf.
Pretrained Transformers achieve remarkable performance when training and test data are from the same distribution. ...
We propose to fine-tune the Transformers with a contrastive loss, which improves the compactness of representations, such that OOD instances can be better differentiated from ID ones. ...
Acknowledgment We appreciate the anonymous reviewers for their insightful comments and suggestions. This material is supported by the National Science Foundation of United States Grant IIS 2105329. ...
doi:10.18653/v1/2021.emnlp-main.84
fatcat:rmiosrfvkvcbpnvda4c575faau
Pretrained Transformers Improve Out-of-Distribution Robustness
[article]
2020
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
Other Versions
We systematically measure out-of-distribution (OOD) generalization for seven NLP datasets by constructing a new robustness benchmark with realistic distribution shifts. ...
Although pretrained Transformers such as BERT achieve high accuracy on in-distribution examples, do they generalize to new distributions? ...
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. ...
arXiv:2004.06100v2
fatcat:2636tz4hcndxvmyfy6dcp2xtri
Self-Supervised Anomaly Detection by Self-Distillation and Negative Sampling
[article]
2022
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2022; you can also visit the original URL.
The file type is application/pdf.
Detecting whether examples belong to a given in-distribution or are Out-Of-Distribution (OOD) requires identifying features specific to the in-distribution. ...
In this work, we show that self-distillation of the in-distribution training set together with contrasting against negative examples derived from shifting transformation of auxiliary data strongly improves ...
These kinds of large-scale pretrained models heavily rely on the classes of the pretraining dataset, which often include classes from both the in and out distribution. ...
arXiv:2201.06378v1
fatcat:e23qpsd3kjcslmcqfr2l7yomae
Temperature as Uncertainty in Contrastive Learning
[article]
2021
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2021; you can also visit the original URL.
The file type is application/pdf.
Through experiments, we demonstrate that TaU is useful for out-of-distribution detection, while remaining competitive with benchmarks on linear evaluation. ...
Moreover, we show that TaU can be learned on top of pretrained models, enabling uncertainty scores to be generated post-hoc with popular off-the-shelf models. ...
Out-of-distribution AUROC is reported. ...
arXiv:2110.04403v1
fatcat:zs3r6wjnwbfxlgrpgp75ahb6k4
Revisiting Pretraining for Semi-Supervised Learning in the Low-Label Regime
[article]
2022
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2022; you can also visit the original URL.
The file type is application/pdf.
The promising results validate the efficacy of target pretraining for SSL, in particular in the low-label regime. ...
In this work, we first show that the better pretrained weights brought in by FT account for the state-of-the-art performance, and importantly that they are universally helpful to off-the-shelf semi-supervised ...
For a dense prediction task, [34] developed a contrastive learning paradigm by comparing pixel-wise features and it turns out to be effective for detection and segmentation tasks. ...
arXiv:2205.03001v1
fatcat:2gw4pt6sirg3rngki7rcuas5ka
Toward Transformer-Based Object Detection
[article]
2020
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
We also investigate improvements over a standard detection backbone, including superior performance on out-of-domain images, better performance on large objects, and a lessened reliance on non-maximum ...
We view ViT-FRCNN as an important stepping stone toward a pure-transformer solution of complex vision tasks such as object detection. ...
Acknowledgements We thank Kofi Boakye, Vahid Kazemi, and Chuck Rosenberg for valuable discussions regarding the paper. ...
arXiv:2012.09958v1
fatcat:rhdftpryhfbrdo6npq7e4vfuse
Masked Discrimination for Self-Supervised Learning on Point Clouds
[article]
2022
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2022; you can also visit the original URL.
The file type is application/pdf.
Other Versions
a significant pretraining speedup (e.g., 4.1x on ScanNet) compared to the prior state-of-the-art Transformer baseline. ...
In this paper, we bridge this gap by proposing a discriminative mask pretraining Transformer framework, MaskPoint}, for point clouds. ...
We thank Xumin Yu for the helpful discussion in reproducing the Point-BERT baselines. ...
arXiv:2203.11183v2
fatcat:s3s5niuea5b2rbe4vqjaeivnee
An Empirical Investigation of Contextualized Number Prediction
[article]
2020
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
Specifically, we introduce a suite of output distribution parameterizations that incorporate latent variables to add expressivity and better fit the natural distribution of numeric values in running text ...
We experiment with novel combinations of contextual encoders and output distributions over the real number line. ...
Findings and observations do not necessarily reflect the views of funding agencies. ...
arXiv:2011.07961v1
fatcat:vc2s5k7ed5ghpn4yupwdyttdhe
EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answering
[article]
2022
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2022; you can also visit the original URL.
The file type is application/pdf.
Although EBMs prove useful for OOD detection, other results on supervised energy-based training and uncertainty calibration are largely negative. ...
The availability of clean and diverse labeled data is a major roadblock for training models on complex tasks such as visual question answering (VQA). ...
Amount of pretraining data To better understand the benefits of pretraining in lowdata regimes, we repeat our experiments while decreasing
Out-of-distribution detection OOD Detection is relevant to VQA ...
arXiv:2206.14355v1
fatcat:e3bhiwdjbrdgbn22w2ul6r6cca
A Study on Self-Supervised Object Detection Pretraining
[article]
2022
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2022; you can also visit the original URL.
The file type is application/pdf.
In this work, we study different approaches to self-supervised pretraining of object detection models. ...
using a transformer, which potentially benefits downstream object detection tasks. ...
At the end of this section, we compare the proposed framework with a number of existing pretraining techniques for object detection. ...
arXiv:2207.04186v1
fatcat:sdog5lz2lbdlji4vtekv2xhhhq
Fully Convolutional Cross-Scale-Flows for Image-based Defect Detection
[article]
2021
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2021; you can also visit the original URL.
The file type is application/pdf.
Our work sets a new state-of-the-art in image-level defect detection on the benchmark datasets Magnetic Tile Defects and MVTec AD showing a 100% AUROC on 4 out of 15 classes. ...
Using normalizing flows to assign meaningful likelihoods to input samples allows for efficient defect detection on image-level. ...
The problem is interpreted in terms of whether a data sample lies out of the distribution p X of the set of normal images X, also named out-of-distribution (OOD) detection. ...
arXiv:2110.02855v1
fatcat:zb5nmwcpwjdgtnisshxsjrdk4m
Metadata-enhanced contrastive learning from retinal optical coherence tomography images
[article]
2022
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2022; you can also visit the original URL.
The file type is application/pdf.
By leveraging this often neglected information our metadata-enhanced contrastive pretraining leads to further benefits and outperforms conventional contrastive methods in five out of seven downstream tasks ...
Several of the image transformations used to create positive contrastive pairs are not applicable to greyscale medical scans. ...
Metadata-enhanced pretraining further boosts performance In six out of seven tasks our metadata-enhanced pretraining strategies outperformed standard contrastive pretraining on the Southampton dataset ...
arXiv:2208.02529v1
fatcat:75bf3zabinh3fkgur7qxlvjm5q
Same Same But DifferNet: Semi-Supervised Defect Detection with Normalizing Flows
[article]
2020
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
The detection of manufacturing errors is crucial in fabrication processes to ensure product quality and safety standards. ...
In contrast to most other methods, ours does not require a large number of training samples and performs well with as low as 16 images. ...
Acknowledgements This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy within the Cluster of Excellence PhoenixD (EXC 2122). ...
arXiv:2008.12577v1
fatcat:qbfwnkpjwfhchmuwufs52dptte
How Transferable Are Self-supervised Features in Medical Image Classification Tasks?
[article]
2021
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2021; you can also visit the original URL.
The file type is application/pdf.
Note that this fulltext copy is not of the "primary" version of this work. The version it corresponds to is:
2021 pre-print arXiv:2108.10048v1 fatcat:qirlco45gvhyfhjqldxdn27ilyOther Versions
tumor classification task, 7.03% AUC in the pneumonia detection, and 9.4% in AUC in the detection of pathological conditions in chest X-ray. ...
to any combination of pretrained models. ...
= self . dropout ( out )
37
out = self . classifier ( out )
38
return out
https://www.kaggle.com/c/histopathologic-cancer-detection 2 https://www.kaggle.com/c/aptos2019-blindness-detection ...
arXiv:2108.10048v3
fatcat:jucst3yv5ndn7n36eaoj3x6j7y
« Previous
Showing results 1 — 15 out of 14,565 results