Filters








3,509 Hits in 4.3 sec

Transferring Knowledge from a RNN to a DNN [article]

William Chan and Nan Rosemary Ke and Ian Lane
2015 arXiv   pre-print
In this paper, we utilize a state-of-the-art RNN to transfer knowledge to small DNN.  ...  Traditionally, the approach for embedded platforms is to either train a small DNN directly, or to train a small DNN that learns the output distribution of a large DNN.  ...  Methodology Our goal is to transfer knowledge from the RNN expert to a small DNN. We follow an approach similar to [14] .  ... 
arXiv:1504.01483v1 fatcat:coilp3jfvbghtmofuw7uuk5yr4

Transferring knowledge from a RNN to a DNN

William Chan, Nan Rosemary Ke, Ian Lane
2015 Interspeech 2015   unpublished
In this paper, we utilize a state-of-the-art RNN to transfer knowledge to small DNN.  ...  Traditionally, the approach for embedded platforms is to either train a small DNN directly, or to train a small DNN that learns the output distribution of a large DNN.  ...  Methodology Our goal is to transfer knowledge from the RNN expert to a small DNN. We follow an approach similar to [14] .  ... 
doi:10.21437/interspeech.2015-657 fatcat:giqhl6xomvddvgyfwygldn42um

Knowledge Transfer Pre-training [article]

Zhiyuan Tang, Dong Wang, Yiqiao Pan, Zhiyong Zhang
2015 arXiv   pre-print
This paper presents a new pre-training approach based on knowledge transfer learning.  ...  Experiments on a speech recognition task demonstrated that with this approach, complex RNNs can be well trained with a weaker deep neural network (DNN) model.  ...  In the first experiment, the knowledge transfer pre-training is used to train RNNs with a DNN as the teacher model.  ... 
arXiv:1506.02256v1 fatcat:22iwweuolnfsrilwk4f3epk6v4

Recurrent neural network training with dark knowledge transfer

Zhiyuan Tang, Dong Wang, Zhiyong Zhang
2016 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)  
In this paper, we employ the knowledge transfer learning approach to train RNNs (precisely LSTM) using a deep neural network (DNN) model as the teacher.  ...  This is different from most of the existing research on knowledge transfer learning, since the teacher (DNN) is assumed to be weaker than the child (RNN); however, our experiments on an ASR task showed  ...  Another related work is the knowledge transfer between DNNs and RNNs, as proposed in [17] . However, it employs knowledge transfer to train DNNs with RNNs.  ... 
doi:10.1109/icassp.2016.7472809 dblp:conf/icassp/TangWZ16 fatcat:pingjchf2nhejaeufoh37awkee

Adaptation and contextualization of deep neural network models

Dimitrios Kollias, Miao Yu, Athanasios Tagaris, Georgios Leontidis, Andreas Stafylopatis, Stefanos Kollias
2017 2017 IEEE Symposium Series on Computational Intelligence (SSCI)  
Moreover, DNNs should use prior information on data classes, or object categories, so as to provide efficient classification of new data, or objects, without forgetting their previous knowledge.  ...  In this paper, we propose a novel class of systems that are able to adapt and contextualize the structure of trained DNNs, providing ways for handling the abovementioned problems.  ...  Our approach to CNN-RNN network design has been through transfer learning, i.e. transfering the weights of the convolutional and pooling layers of a pretrained CNN, to the generated network and training  ... 
doi:10.1109/ssci.2017.8280975 dblp:conf/ssci/KolliasYTLSK17 fatcat:ckdelnwsm5cgxh5u652nkzyvxa

Phonetic Temporal Neural Model for Language Identification [article]

Zhiyuan Tang, Dong Wang, Yixiang Chen, Lantian Li, Andrew Abel
2017 arXiv   pre-print
We present a phonetic temporal neural model for LID, which is an LSTM-RNN LID system that accepts phonetic features produced by a phone-discriminative DNN as the input, rather than raw acoustic features  ...  This new model is similar to traditional phonetic LID methods, but the phonetic knowledge here is much richer: it is at the frame level and involves compacted information of all phones.  ...  Transfer learning perspective: The second perspective to understand the PTN approach is from the transfer learning perspective [38] .  ... 
arXiv:1705.03151v3 fatcat:ep52qpfurzcobfawhf6soeafie

Analysis of Multilingual Sequence-to-Sequence Speech Recognition Systems

Martin Karafiát, Murali Karthick Baskar, Shinji Watanabe, Takaaki Hori, Matthew Wiesner, Jan Černocký
2019 Interspeech 2019  
Although both approaches significantly improved the performance from a monolingual seq2seq baseline, interestingly, we found the multilingual bottle-neck features to be superior to multilingual models  ...  In this experiment, we also explore various architectures and training strategies of the multilingual seq2seq model by making use of knowledge obtained in the DNN-HMM based transfer-learning.  ...  Multilingual features Multilingual features are trained separately from seq2seq model according to a setup from our previous RNN/DNN-HMM work [23] .  ... 
doi:10.21437/interspeech.2019-2355 dblp:conf/interspeech/KarafiatBWHWC19 fatcat:22cw452vrnggdio4wqaqx6jc5e

Parameter Transfer Unit for Deep Neural Networks [article]

Yinghua Zhang, Yu Zhang, Qiang Yang
2018 arXiv   pre-print
To address the gap, a parameter transfer unit (PTU) is proposed in this paper.  ...  In the PTU, the transferability is controlled by two gates which are artificial neurons and can be learned from data. The PTU is a general and flexible module which can be used in both CNNs and RNNs.  ...  For the update gate z, it controls how much knowledge is flowed from the source domain to the target domain. A larger z indicates more knowledge transfer.  ... 
arXiv:1804.08613v1 fatcat:4vv5t5f4lbej3krntxe53d23r4

Short-term Traffic Prediction with Deep Neural Networks: A Survey

Kyungeun Lee, Moonjung Eo, Euna Jung, Yoonjin Yoon, Wonjong Rhee
2021 IEEE Access  
Reference [77] is aimed at effectively transferring knowledge from a data-rich source city to a data-scarce target city.  ...  Learning Transfer learning is a machine learning technique in which knowledge is obtained by solving a problem, and the acquired knowledge is used for solving a related but different problem.  ... 
doi:10.1109/access.2021.3071174 fatcat:qzasxqazwraehojhqahx4zesjy

Deep neural architectures for prediction in healthcare

Dimitrios Kollias, Athanasios Tagaris, Andreas Stafylopatis, Stefanos Kollias, Georgios Tagaris
2017 Complex & Intelligent Systems  
Their application in healthcare for prediction and diagnosis purposes can produce high accuracy results and can be combined with medical knowledge to improve effectiveness, adaptation and transparency  ...  This paper presents a novel class of systems assisting diagnosis and personalised assessment of diseases in healthcare.  ...  the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.  ... 
doi:10.1007/s40747-017-0064-6 fatcat:ougakudravdptjv6ukdm3bv64q

Empirical Analysis of Score Fusion Application to Combined Neural Networks for Open Vocabulary Spoken Term Detection

Shi-wook Lee, Kazuyo Tanaka, Yoshiaki Itoh
2018 Interspeech 2018  
recent knowledge transfer learning.  ...  In this study, to describe how to extract effective knowledge from an ensemble of neural networks, we first examine several score fusions from an ensemble of neural networks tasked with open vocabulary  ...  Acknowledgements This research was partially supported by a Grant-in-Aid for Scientific Research (C), KAKENHI Project Nos. 15K00241 and 15K00262.  ... 
doi:10.21437/interspeech.2018-1776 dblp:conf/interspeech/LeeT018 fatcat:bypyvkshv5fq5ar7alxhg746uu

Towards end-to-end speech recognition with transfer learning

Chu-Xiong Qin, Dan Qu, Lian-Hai Zhang
2018 EURASIP Journal on Audio, Speech, and Music Processing  
A transfer learning-based end-to-end speech recognition approach is presented in two levels in our framework.  ...  Firstly, a feature extraction approach combining multilingual deep neural network (DNN) training with matrix factorization algorithm is introduced to extract high-level features.  ...  Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.  ... 
doi:10.1186/s13636-018-0141-9 fatcat:gm3v56s37rf5rgsqxpdeozyyk4

Short-term Traffic Prediction with Deep Neural Networks: A Survey [article]

Kyungeun Lee, Moonjung Eo, Euna Jung, Yoonjin Yoon, Wonjong Rhee
2020 arXiv   pre-print
involved. 2) We briefly explain a wide range of DNN techniques from the earliest networks, including Restricted Boltzmann Machines, to the most recent, including graph-based and meta-learning networks  ...  In this study, we survey recent STTP studies applying deep networks from four perspectives. 1) We summarize input data representation methods according to the number and type of spatial and temporal dependencies  ...  Reference [64] is aimed at effectively transferring knowledge from a data-rich source city to a data-scarce target city.  ... 
arXiv:2009.00712v1 fatcat:rvcz235ugbahhjglkjgvwgxks4

Deep learning techniques for prediction of losses in data transmitted in wireless sensor networks

2020 Turkish Journal of Electrical Engineering and Computer Sciences  
It enables the data obtained from the sensors to be transferred between nodes with the help of end-to-end 7 wireless protocols.  ...  In this study, the loss rate in the transferred data packets was 14 tried to be estimated with the highest accuracy by using Deep Belief Network (DBN), Recurrent Neural Network (RNN) 15 and Deep Neural  ...  Transfer of sensed data from sensor nodes to storage unit via sink node in WSN Data transfer process in WSN consists of 4 stages [40] .  ... 
doi:10.3906/elk-2001-145 fatcat:ov72dtcyarfvpkmyg4mqsnuh3m

Driver Behavior Profiling and Recognition Using Deep-Learning Methods: In Accordance with Traffic Regulations and Experts Guidelines

Ward Ahmed Al-Hussein, Lip Yee Por, Miss Laiha Mat Kiah, Bilal Bahaa Zaidan
2022 International Journal of Environmental Research and Public Health  
Following that, the study adopts three deep-learning-based algorithms, namely, Deep Neural Network (DNN), Recurrent Neural Network (RNN), and Convolutional Neural Network (CNN), to classify recorded driving  ...  Various techniques were used to prevent the classification algorithms from overfitting.  ...  Acknowledgments: The authors are thankful and appreciative to Azhar Hamzah from the Road User Behavioural Change Research Center, at the Malaysian Institute of Road Safety Research (MIROS) for his support  ... 
doi:10.3390/ijerph19031470 pmid:35162493 pmcid:PMC8835443 fatcat:cgwvuzve7jhrnmoiclj44yub64
« Previous Showing results 1 — 15 out of 3,509 results