Filters








1,185 Hits in 3.2 sec

Are Multilingual Neural Machine Translation Models Better at Capturing Linguistic Features?

David Mareček, Hande Celikkanat, Miikka Silfverberg, Vinit Ravishankar, Jörg Tiedemann
2020 Prague Bulletin of Mathematical Linguistics  
attention structures regarding the inherent phrase and dependency information and a structural probe on contextualized word representations.  ...  The latter also applies to phrase structure and syntactic dependencies that do not seem to be developing in sentence representations when increasing the linguistic diversity in training to translate.  ...  The structural probe presented by Hewitt and Manning (2019) investigates the relation between the syntax tree of a sentence and its contextualized word embeddings derived from a model.  ... 
doi:10.14712/00326585.009 fatcat:yerr7ankqbbwjg5zpcu4yulwwy

Syntax Representation in Word Embeddings and Neural Networks – A Survey [article]

Tomasz Limisiewicz, David Mareček
2020 arXiv   pre-print
We mainly summarize re-search on English monolingual data on language modeling tasks and multilingual data for neural machine translation systems and multilingual language models.  ...  Neural networks trained on natural language processing tasks capture syntax even though it is not provided as a supervision signal.  ...  Therefore, the experiment is not strictly comparable with probing as the most of syntactic information is captured by the parser, and not by the embeddings.  ... 
arXiv:2010.01063v1 fatcat:a6lna2623ngfld27eilirbmfsu

How Language-Neutral is Multilingual BERT? [article]

Jindřich Libovický and Rudolf Rosa and Alexander Fraser
2019 arXiv   pre-print
Previous work probed the cross-linguality of mBERT using zero-shot transfer learning on morphological and syntactic tasks. We instead focus on the semantic properties of mBERT.  ...  semantics to allow high-accuracy word-alignment and sentence retrieval but is not yet good enough for the more difficult task of MT quality estimation.  ...  We then analyze the semantic properties of both the original and the centered representations using a range of probing tasks. For all tasks, we test all layers of the model.  ... 
arXiv:1911.03310v1 fatcat:ti3oyahm45ggtjiiwdsiy47s7m

Empirical Linguistic Study of Sentence Embeddings

Katarzyna Krasnowska-Kieraś, Alina Wróblewska
2019 Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics  
Aside from dependency parser-based embeddings, linguistic information is retained best in the recently proposed LASER sentence embeddings.  ...  We introduce a method of analysing the content of sentence embeddings based on universal probing tasks, along with the classification datasets for two contrasting languages.  ...  tests can be generated for any language with a UD treebank on which a parser can be trained. 1 The Universal Dependencies initiative aims at developing a cross-linguistically consistent morphosyntactic  ... 
doi:10.18653/v1/p19-1573 dblp:conf/acl/Krasnowska-Kieras19 fatcat:5kwxe5f3urfehhmlk7u6d3tbpe

Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networks [article]

Prajit Dhar, Arianna Bisazza
2021 arXiv   pre-print
We find that exposing our LMs to a related language does not always increase grammatical knowledge in the target language, and that optimal conditions for lexical-semantic transfer may not be optimal for  ...  In this paper we dissect different forms of cross-lingual transfer and look for its most determining factors, using a variety of models and probing tasks.  ...  Acknowledgements Arianna Bisazza was partly funded by the Netherlands Organization for Scientific Research (NWO) under project number 639.021.646.  ... 
arXiv:2003.14056v3 fatcat:bvuocwt3xrbzdmoiezj5iqkxxa

Infusing Finetuning with Semantic Dependencies [article]

Zhaofeng Wu, Hao Peng, Noah A. Smith
2021 arXiv   pre-print
We apply novel probes to recent language models -- specifically focusing on predicate-argument structure as operationalized by semantic dependencies (Ivanova et al., 2012) -- and find that, unlike syntax  ...  , semantics is not brought to the surface by today's pretrained models.  ...  This research was supported in part by a Google Fellowship to HP and NSF grant 1562364.  ... 
arXiv:2012.05395v3 fatcat:rvy45vhmavfahk6um55u7r2vzy

Low-Resource Parsing with Crosslingual Contextualized Representations [article]

Phoebe Mulcaire, Jungo Kasai, Noah A. Smith
2019 arXiv   pre-print
We assess recent approaches to multilingual contextual word representations (CWRs), and compare them for crosslingual transfer from a language with a large treebank to a language with a small or nonexistent  ...  treebank, by sharing parameters between languages in the parser itself.  ...  Acknowledgments The authors thank Nikolaos Pappas and Tal Schuster as well as the anonymous reviewers for their helpful feedback.  ... 
arXiv:1909.08744v1 fatcat:hojp7ddnwjgnrk7lorh5ocvhdi

From General Language Understanding to Noisy Text Comprehension

Buddhika Kasthuriarachchy, Madhu Chetty, Adrian Shatte, Darren Walls
2021 Applied Sciences  
Five new probing tasks are developed for Tweets, which can serve as benchmark probing tasks to study noisy text comprehension.  ...  For this, we propose a new generic methodology to derive a diverse set of sentence vectors combining and extracting various linguistic characteristics from latent representations of multi-layer, pre-trained  ...  [31] introduced "edge probing" tasks, covering syntax, semantic meaning and dependency relations phenomena to study how contextual representations encode sentence structures.  ... 
doi:10.3390/app11177814 fatcat:tdcafintwvfl3cyyuxlnrxpeau

What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties

Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, Marco Baroni
2018 Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)  
Although much effort has recently been devoted to training high-quality sentence embeddings, we still have a poor understanding of what they are capturing.  ...  We introduce here 10 probing tasks designed to capture simple linguistic features of sentences, and we use them to study embeddings generated by three different encoders trained in eight distinct ways,  ...  Acknowledgments We thank David Lopez-Paz, Holger Schwenk, Hervé Jégou, Marc'Aurelio Ranzato and Douwe Kiela for useful comments and discussions.  ... 
doi:10.18653/v1/p18-1198 dblp:conf/acl/BaroniBLKC18 fatcat:tqjfd266snfyngrftmmvse2qce

QuASE: Question-Answer Driven Sentence Encoding [article]

Hangfeng He, Qiang Ning, Dan Roth
2020 arXiv   pre-print
This paper studies a natural question: Can we get supervision from QA data for other tasks (typically, non-QA ones)?  ...  In particular, we observe the need to distinguish between two types of sentence encodings, depending on whether the target task is a single- or multi-sentence input; in both cases, the resulting encoding  ...  The following probing analysis, based on the Xinhua subset in the AMR dataset, shows that s-QUASE QAM R embeddings encode more semantics related to AMR than BERT embeddings.  ... 
arXiv:1909.00333v3 fatcat:vyylkxrlkneatj5o5mks2n2zyy

On the Evolution of Syntactic Information Encoded by BERT's Contextualized Representations [article]

Laura Pérez-Mayos, Roberto Carlini, Miguel Ballesteros, Leo Wanner
2021 arXiv   pre-print
Experimental results show that the encoded syntactic information is forgotten (PoS tagging), reinforced (dependency and constituency parsing) or preserved (semantics-related tasks) in different ways along  ...  In this paper, we analyze the evolution of the embedded syntax trees along the fine-tuning process of BERT for six different tasks, covering all levels of the linguistic structure.  ...  What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties.  ... 
arXiv:2101.11492v2 fatcat:hqakazppuzfjtmdhpxkrypjjpi

SG-Net: Syntax Guided Transformer for Language Representation [article]

Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, Rui Wang
2021 arXiv   pre-print
In detail, for self-attention network (SAN) sponsored Transformer-based encoder, we introduce syntactic dependency of interest (SDOI) design into the SAN to form an SDOI-SAN with syntax-guided self-attention  ...  Syntax-guided network (SG-Net) is then composed of this extra SDOI-SAN and the SAN from the original Transformer encoder through a dual contextual architecture for better linguistics inspired representation  ...  A related work is from Strubell et al. (2018) [55] , which proposed to incorporate syntax with multi-task learning for semantic role labeling.  ... 
arXiv:2012.13915v2 fatcat:2zyyd4s6ibcuvjal3k7t2e4v44

Improving Natural Language Inference with a Pretrained Parser [article]

Deric Pang, Lucy H. Lin, Noah A. Smith
2019 arXiv   pre-print
We introduce a novel approach to incorporate syntax into natural language inference (NLI) models. Our method uses contextual token-level vector representations from a pretrained dependency parser.  ...  A.1 Word Embeddings The parser uses 100-dimensional GloVe word embeddings (Pennington et al., 2014) trained on Wikipedia/Gigaword while the rest of our models use 300-dimensional GloVe word embeddings  ...  sp sh trained dependency parser.  ... 
arXiv:1909.08217v1 fatcat:2iicjhzwzjg47o7kd5ciqvsvyq

Visualizing and Measuring the Geometry of BERT [article]

Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, Martin Wattenberg
2019 arXiv   pre-print
At a high level, linguistic features seem to be represented in separate semantic and syntactic subspaces. We find evidence of a fine-grained geometric representation of word senses.  ...  Transformer architectures show significant promise for natural language processing.  ...  Acknowledgments: We would like to thank David Belanger, Tolga Bolukbasi, Jasper Snoek, and Ian Tenney for helpful feedback and discussions.  ... 
arXiv:1906.02715v2 fatcat:aydti652tfgildot2x52kc2itu

Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Case [article]

Adam Dahlgren Lindström, Suna Bensch, Johanna Björklund, Frank Drewes
2021 arXiv   pre-print
Semantic embeddings have advanced the state of the art for countless natural language processing tasks, and various extensions to multimodal domains, such as visual-semantic embeddings, have been proposed  ...  for those properties, and (iv) compare various state-of-the-art embeddings under the lens of the proposed probing tasks.  ...  We thank the anonymous reviewers for their valuable feedback, which has had a substantial influence on the final version of the paper.  ... 
arXiv:2102.11115v1 fatcat:f6l5gzk7hjdgjayvzgnw7ux4ha
« Previous Showing results 1 — 15 out of 1,185 results