Filters








129 Hits in 7.5 sec

Composed Fine-Tuning: Freezing Pre-Trained Denoising Autoencoders for Improved Generalization [article]

Sang Michael Xie, Tengyu Ma, Percy Liang
2021 arXiv   pre-print
We first show that standard fine-tuning after pre-training destroys some of this structure. We then propose composed fine-tuning, which trains a predictor composed with the pre-trained denoiser.  ...  We prove for two-layer ReLU networks that composed fine-tuning significantly reduces the complexity of the predictor, thus improving generalization.  ...  Effect of pre-training objectives Naively, better pre-trained denoisers should improve the gains with composed fine-tuning.  ... 
arXiv:2006.16205v3 fatcat:6cg3pl4bv5gxdgeiqrwe3fmpku

User-specific Adaptive Fine-tuning for Cross-domain Recommendations [article]

Lei Chen, Fajie Yuan, Jiaxi Yang, Xiangnan He, Chengming Li, Min Yang
2021 arXiv   pre-print
However, current methods are mainly based on the global fine-tuning strategy: the decision of which layers of the pre-trained model to freeze or fine-tune is taken for all users in the target domain.  ...  As such, we propose a novel User-specific Adaptive Fine-tuning method (UAF), selecting which layers of the pre-trained network to fine-tune, on a per-user basis.  ...  Similar to [40] , [41] , we generate all freezing/fine-tuning policies for all residual blocks at once for the trade-off of efficiency and accuracy.  ... 
arXiv:2106.07864v2 fatcat:2juhc4smfbcf5b5ogrl2ovkum4

Learning Transferable Features for Speech Emotion Recognition

Alison Marczewski, Adriano Veloso, Nivio Ziviani
2017 Proceedings of the on Thematic Workshops of ACM Multimedia 2017 - Thematic Workshops '17  
In this paper, we propose a deep architecture that jointly exploits a convolutional network for extracting domain-shared features and a long short-term memory network for classifying emotions using domain-specific  ...  We evaluate several domain adaptation approaches, and we perform an ablation study to understand which source domains add the most to the overall recognition effectiveness for a given target domain.  ...  ACKNOWLEDGEMENTS We thank the partial support given by the Brazilian National Institute of Science and Technology for the Web (grant MCT-CNPq 573871/2008-6), Project: Models, Algorithms and Systems for  ... 
doi:10.1145/3126686.3126735 dblp:conf/mm/MarczewskiVZ17 fatcat:l6b3h6yxp5eddoi6ist42xwewy

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension [article]

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer
2019 arXiv   pre-print
BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks.  ...  We present BART, a denoising autoencoder for pretraining sequence-to-sequence models.  ...  Future work should explore new methods for corrupting documents for pre-training, perhaps tailoring them to specific end tasks.  ... 
arXiv:1910.13461v1 fatcat:gxhwv2jwlbayhhonjdfksrcpou

Masked Autoencoders Are Scalable Vision Learners [article]

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick
2021 arXiv   pre-print
This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision.  ...  Coupling these two designs enables us to train large models efficiently and effectively: we accelerate training (by 3x or more) and improve accuracy.  ...  Our MAE pre-training does not use it. For any specialized fine-tuning.  ... 
arXiv:2111.06377v3 fatcat:4d7762easfdcniz4jvqedqizqy

Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction

Richard Zhang, Phillip Isola, Alexei A. Efros
2017 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)  
We propose split-brain autoencoders, a straightforward modification of the traditional autoencoder architecture, for unsupervised representation learning.  ...  Each sub-network is trained to perform a difficult taskpredicting one subset of the data channels from another. Together, the sub-networks extract features from the entire input signal.  ...  Acknowledgements We thank members of the Berkeley Artificial Intelligence Research Lab (BAIR), in particular Andrew Owens, for helpful discussions, as well as Saurabh Gupta for help with RGB-D experiments  ... 
doi:10.1109/cvpr.2017.76 dblp:conf/cvpr/ZhangIE17 fatcat:boaqjncxxrcklavpco4b5mrku4

Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction [article]

Richard Zhang, Phillip Isola, Alexei A. Efros
2017 arXiv   pre-print
We propose split-brain autoencoders, a straightforward modification of the traditional autoencoder architecture, for unsupervised representation learning.  ...  Each sub-network is trained to perform a difficult task -- predicting one subset of the data channels from another. Together, the sub-networks extract features from the entire input signal.  ...  Acknowledgements We thank members of the Berkeley Artificial Intelligence Research Lab (BAIR), in particular Andrew Owens, for helpful discussions, as well as Saurabh Gupta for help with RGB-D experiments  ... 
arXiv:1611.09842v3 fatcat:lja7ousv2zgo3hqtxafz7xsyvm

SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Training [article]

Wenyong Huang, Zhenhe Zhang, Yu Ting Yeung, Xin Jiang, Qun Liu
2022 arXiv   pre-print
training solely in the fine-tuning stage.  ...  We introduce a new approach for speech pre-training named SPIRAL which works by learning denoising representation of perturbed data in a teacher-student framework.  ...  pre-training is more effective than solely applying multi-condition training in the fine-tuning stage.We presume SPIRAL as a general pre-training method, which can apply to other modalities such as images  ... 
arXiv:2201.10207v3 fatcat:qyalg2exynbjflttbmggdh4qlm

Reusing Monolingual Pre-Trained Models by Cross-Connecting Seq2seq Models for Machine Translation

Jiun Oh, Yong-Suk Choi
2021 Applied Sciences  
This work uses sequence-to-sequence (seq2seq) models pre-trained on monolingual corpora for machine translation.  ...  We pre-train two seq2seq models with monolingual corpora for the source and target languages, then combine the encoder of the source language model and the decoder of the target language model, i.e., the  ...  The fluency of each pre-trained model is preserved intact. Cross-Connection and Fine-Tuning We compose the pre-trained models for fine-tuning.  ... 
doi:10.3390/app11188737 fatcat:oepw35gjp5fubcotgnqjxvofqm

Laplacian Denoising Autoencoder [article]

Jianbo Jiao, Linchao Bao, Yunchao Wei, Shengfeng He, Honghui Shi, Rynson Lau, Thomas S. Huang
2020 arXiv   pre-print
In this paper, we propose to learn data representations with a novel type of denoising autoencoder, where the noisy input data is generated by corrupting latent clean data in the gradient domain.  ...  While deep neural networks have been shown to perform remarkably well in many machine learning tasks, labeling a large amount of ground truth data for supervised training is usually very costly to scale  ...  The learned weights of our model trained on ImageNet are transferred to a standard AlexNet for the evaluation. We then fine-tune the model on the PASCAL VOC trainval set and test on the test set.  ... 
arXiv:2003.13623v1 fatcat:uel6btlupzbarey7rvr6i6ya7i

Online Learning of Deep Hybrid Architectures for Semi-supervised Categorization [chapter]

Alexander G. Ororbia, David Reitter, Jian Wu, C. Lee Giles
2015 Lecture Notes in Computer Science  
We (1) compare this architecture against a hybrid denoising autoencoder version of itself as well as several other models and (2) investigate training in the context of an incremental learning procedure  ...  The model's training algorithm is built on principles developed from hybrid discriminative Boltzmann machines and composes deep architectures in a greedy fashion.  ...  We thank Hugo Larochelle and Roberto Calandra for their correspondence and advice, although all shortcomings of the presented work are ours.  ... 
doi:10.1007/978-3-319-23528-8_32 fatcat:znn2gsczwzhd3kypvnfg2meayu

Deep Learning Approaches for Detecting Freezing of Gait in Parkinson's Disease Patients through On-Body Acceleration Sensors

Luis Sigcha, Nélson Costa, Ignacio Pavón, Susana Costa, Pedro Arezes, Juan Manuel López, Guillermo De Arcas
2020 Sensors  
Also, several machine and deep learning approaches for FOG detection are evaluated using a leave-one-subject-out (LOSO) cross-validation.  ...  Freezing of gait (FOG) is one of the most incapacitating motor symptoms in Parkinson's disease (PD). The occurrence of FOG reduces the patients' quality of live and leads to falls.  ...  Architecture for ML Denoiser Autoencoder For the novelty detection based in DL, a denoiser autoencoder composed of seven fully connected layers, according to the scheme presented in Figure 2 , was tested  ... 
doi:10.3390/s20071895 pmid:32235373 fatcat:7hfljuaik5fljfuod5ubesbsci

Test-Time Adaptable Neural Networks for Robust Medical Image Segmentation [article]

Neerav Karani, Ertunc Erdil, Krishna Chaitanya, Ender Konukoglu
2020 arXiv   pre-print
We employ an independently trained denoising autoencoder (DAE) in order to model such an implicit prior on plausible anatomical segmentation labels.  ...  The proposed test-time adaptation consistently provides performance improvement, demonstrating the promise and generality of the approach.  ...  We also thank NVIDIA corporation for their GPU donation.  ... 
arXiv:2004.04668v3 fatcat:lxh3hsoeifhffmx6pgwtrdwwkq

IPMiner: hidden ncRNA-protein interaction sequential pattern mining with stacked autoencoder for accurate computational prediction

Xiaoyong Pan, Yong-Xian Fan, Junchi Yan, Hong-Bin Shen
2016 BMC Genomics  
Finally, stacked ensembling is used to integrate different predictors to further improve the prediction performance.  ...  In addition, we further apply IPMiner for large-scale prediction of ncRNA-protein network, that achieves promising prediction performance.  ...  Acknowledgements We thanks for the two anonymous reviewers's comments on this study.  ... 
doi:10.1186/s12864-016-2931-8 pmid:27506469 pmcid:PMC4979166 fatcat:ibwcovlqcndo7jlbxjp6xphpvi

Representation Learning Through Latent Canonicalizations [article]

Or Litany, Ari Morcos, Srinath Sridhar, Leonidas Guibas, Judy Hoffman
2020 arXiv   pre-print
We dub these transformations latent canonicalizers, as they aim to modify the value of a factor to a pre-determined (but arbitrary) canonical value (e.g., recoloring the image foreground to black).  ...  We seek to learn a representation on a large annotated data source that generalizes to a target domain using limited new supervision.  ...  Transfer Learning and Few-shot Learning: Alternatively, synthetic data can be used for pre-training followed by fine-tuning on real data (Richter et al., 2016 )-a form of transfer learning.  ... 
arXiv:2002.11829v1 fatcat:sqvupdu2bvgl7azfjtsfw5kdxe
« Previous Showing results 1 — 15 out of 129 results