Filters








42,685 Hits in 5.0 sec

Linguistic Knowledge and Transferability of Contextual Representations [article]

Nelson F. Liu and Matt Gardner and Yonatan Belinkov and Matthew E. Peters and Noah A. Smith
2019 arXiv   pre-print
To shed light on the linguistic knowledge they capture, we study the representations produced by several recent pretrained contextualizers (variants of ELMo, the OpenAI transformer language model, and  ...  Contextual word representations derived from large-scale neural language models are successful across a diverse set of NLP tasks, suggesting that they encode useful and transferable features of language  ...  We also thank the members of the Noah's ARK group at the University of Washington, the researchers at the Allen Institute for Artificial Intelligence, and the anonymous reviewers for their valuable feedback  ... 
arXiv:1903.08855v5 fatcat:74y4i5pf3resnj34mlkl2jrium

Linguistic Knowledge and Transferability of Contextual Representations

Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, Noah A. Smith
2019 Proceedings of the 2019 Conference of the North  
To shed light on the linguistic knowledge they capture, we study the representations produced by several recent pretrained contextualizers (variants of ELMo, the OpenAI transformer language model, and  ...  Contextual word representations derived from large-scale neural language models are successful across a diverse set of NLP tasks, suggesting that they encode useful and transferable features of language  ...  We also thank the members of the Noah's ARK group at the University of Washington, the researchers at the Allen Institute for Artificial Intelligence, and the anonymous reviewers for their valuable feedback  ... 
doi:10.18653/v1/n19-1112 dblp:conf/naacl/Liu0BPS19 fatcat:uxpjqb7v3vgtjpxyz3ansjwubm

Improving CTC-based speech recognition via knowledge transferring from pre-trained language models [article]

Keqi Deng, Songjun Cao, Yike Zhang, Long Ma, Gaofeng Cheng, Ji Xu, Pengyuan Zhang
2022 arXiv   pre-print
To solve this issue, we propose two knowledge transferring methods that leverage pre-trained LMs, such as BERT and GPT2, to improve CTC-based models.  ...  When compared to the vanilla CTC-based models fine-tuned from the wav2vec2.0 models, our knowledge transferring method reduces CER by 16.1% relatively without external LMs.  ...  contextual knowledge of pre-trained LMs into the ASR systems and we propose two kinds of knowledge transferring methods.  ... 
arXiv:2203.03582v1 fatcat:tyra453gkjfethxt6ysaomi2fi

Six Steps from Visual Metaphors to Knowledge Transfer

Ekaterina Isaeva, Svetlana Mishlanova
2015 Mediterranean Journal of Social Sciences  
The paper presents evidence from research into the problem of knowledge transfer modalities and cognitive powers of the visual metaphor.  ...  The Procedure has been applied for the extraction of knowledge, transferred via visual metaphors in the World War II propaganda posters by KUKRYNIKSI.  ...  which the projects on the study of knowledge transfer and conceptual modelling were completed.  ... 
doi:10.5901/mjss.2015.v6n6s5p228 fatcat:uyyyy7hm3ze5rbtaq2q7tntu64

Inferring symmetry in natural language [article]

Chelsea Tanchip, Lei Yu, Aotao Xu, Yang Xu
2020 arXiv   pre-print
Our results show that a hybrid transfer learning model that integrates linguistic features with contextualized language models most faithfully predicts the empirical data.  ...  We develop methods that formalize these approaches and evaluate them against a novel symmetry inference sentence (SIS) dataset comprised of 400 naturalistic usages of literature-informed verbs spanning  ...  YX is funded through a Connaught New Researcher Award, a NSERC Discovery Grant RGPIN-2018-05872, and a SSHRC Insight Grant #435190272.  ... 
arXiv:2010.08090v1 fatcat:bt7jbsyp4nhvvizqysjnzohvkm

Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages [article]

Ehsan Aghazadeh, Mohsen Fayyaz, Yadollah Yaghoobzadeh
2022 arXiv   pre-print
Our extensive experiments suggest that contextual representations in PLMs do encode metaphorical knowledge, and mostly in their middle layers.  ...  The knowledge is transferable between languages and datasets, especially when the annotation is consistent across training and testing sets.  ...  Acknowledgements We would like to thank the anonymous reviewers and action editors who helped us greatly in improving our work with their comments.  ... 
arXiv:2203.14139v1 fatcat:tsrjljkzbvds3awwpms7msty4q

Cross-lingual Structure Transfer for Relation and Event Extraction

Ananya Subburathinam, Di Lu, Heng Ji, Jonathan May, Shih-Fu Chang, Avirup Sil, Clare Voss
2019 Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)  
We exploit relation-and event-relevant language-universal features, leveraging both symbolic (including part-of-speech and dependency path) and distributional (including type representation and contextualized  ...  We thus find that language-universal symbolic and distributional representations are complementary for cross-lingual structure transfer.  ...  Linguistic knowledge and transferability of contextual representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics.  ... 
doi:10.18653/v1/d19-1030 dblp:conf/emnlp/SubburathinamLJ19 fatcat:3vwt72efyrfn7ojd6pkn64q7w4

Translation-Based Implicit Annotation Projection for Zero-Shot Cross-Lingual Event Argument Extraction

Chenwei Lou, Jun Gao, Changlong Yu, Wei Wang, Huan Zhao, Weiwei Tu, Ruifeng Xu
2022 Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval  
With the use of translation-based parallel corpora, no additional linguistic features are required during training and inference.  ...  Moreover, our implicit annotation projection approach introduces less noises and hence is more effective and robust than explicit ones.  ...  Science and Technology Program JSGG20210802154400001, and the Joint Lab of HITSZ and China Merchants Securities.  ... 
doi:10.1145/3477495.3531808 fatcat:73jwfzmwz5fihcvnuiv4ln3efm

Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing

Tal Schuster, Ori Ram, Regina Barzilay, Amir Globerson
2019 Proceedings of the 2019 Conference of the North  
While contextual embeddings have been shown to yield richer representations of meaning compared to their static counterparts, aligning them poses a challenge due to their dynamic nature.  ...  We introduce a novel method for multilingual transfer that utilizes deep contextual embeddings, pretrained in an unsupervised fashion.  ...  The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the  ... 
doi:10.18653/v1/n19-1162 dblp:conf/naacl/SchusterRBG19 fatcat:ns2bxzatkjdovnyqzxegtw53i4

Effect of Post-processing on Contextualized Word Representations [article]

Hassan Sajjad and Firoj Alam and Fahim Dalvi and Nadir Durrani
2021 arXiv   pre-print
On a diverse set of pre-trained models, we show that post-processing unwraps vital information present in the representations for both lexical tasks (such as word similarity and analogy)and sequence classification  ...  Our findings raise interesting points in relation to theresearch studies that use contextualized representations, and suggest z-score normalization as an essential step to consider when using them in an  ...  Lin- guistic knowledge and transferability of contextual representations.  ... 
arXiv:2104.07456v1 fatcat:zc7ngcfy7nhcpchcj7dgw5kkxu

Pre-trained Models for Natural Language Processing: A Survey [article]

Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, Xuanjing Huang
2020 arXiv   pre-print
Next, we describe how to adapt the knowledge of PTMs to the downstream tasks. Finally, we outline some potential directions of PTMs for future research.  ...  We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy with four perspectives.  ...  This work was supported by the National Natural Science Foundation of China (No. 61751201 and 61672162), Shanghai Municipal Science and Technology Major Project (No. 2018SHZDZX01) and ZJLab.  ... 
arXiv:2003.08271v3 fatcat:ze64wcfecfgs7bguq4vajpsgpu

Quantifying the Contextualization of Word Representations with Semantic Class Probing [article]

Mengjie Zhao, Philipp Dufter, Yadollah Yaghoobzadeh, Hinrich Schütze
2020 arXiv   pre-print
for semantic class inference; and that top layer representations are more task-specific after finetuning while lower layer representations are more transferable.  ...  We investigate the contextualization of words in BERT.  ...  Acknowledgments We thank the anonymous reviewers for the insightful comments and suggestions.  ... 
arXiv:2004.12198v2 fatcat:crnslqh4jjb5bhizczlpdmtgay

On the Hierarchical Information in a Single Contextualised Word Representation (Student Abstract)

Dean L. Slack, Mariann Hardey, Noura Al Moubayed
2020 PROCEEDINGS OF THE THIRTIETH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE AND THE TWENTY-EIGHTH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE  
linguistic features encoded in their representations.  ...  This work aims to investigate to what extent any linguistic hierarchical information is encoded into a single contextual embedding.  ...  However, due to a poor understanding of the linguistic information contained in contextual representations -a thriving new area of research looks to employ novel analysis techniques to improve the interpretability  ... 
doi:10.1609/aaai.v34i10.7231 fatcat:itjoahgt5zgzdk7hnfcsshctuq

Learning Robust, Transferable Sentence Representations for Text Classification [article]

Wasi Uddin Ahmad, Xueying Bai, Nanyun Peng, Kai-Wei Chang
2018 arXiv   pre-print
Extensive experiments and analyses using a wide range of transfer and linguistic tasks endorse the effectiveness of our approach.  ...  sentence representations that are useful for transfer learning.  ...  The unified sentence encoder that combines both the sentence and contextual word representations captures most of the linguistic properties. Impact of training data.  ... 
arXiv:1810.00681v1 fatcat:6pkwhuzjrrh7bklpd7lxunf3nq

Translation by Quasi Logical Form transfer

Hiyan Alshawi, David Carter, Manny Rayner, Björn Gambäck
1991 Proceedings of the 29th annual meeting on Association for Computational Linguistics -  
Transfer takes place at the level of Quasi Logical Form (QLF), a contextually sensitive logical form representation which is deep enough for dealing with cross-linguistic differences.  ...  Theoretical arguments and experimental results are presented to support the claim that this framework has good properties in terms of modularity, compositionality, reversibility and monotonicity.  ...  ACKNOWLEDGMENTS The work reported here was funded by the Swedish Institute of Computer Science, and the greater part of it was carried out while the third author was employed there.  ... 
doi:10.3115/981344.981365 dblp:conf/acl/AlshawiCR91 fatcat:b73pt7ynbjbvtou6bfqgcci3eu
« Previous Showing results 1 — 15 out of 42,685 results