Filters








153 Hits in 7.3 sec

Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks [article]

R. Thomas McCoy and Robert Frank and Tal Linzen
2018 arXiv   pre-print
We examine this proposal using recurrent neural networks (RNNs), which are not constrained in such a way.  ...  We simulate the acquisition of question formation, a hierarchical transformation, in a fragment of English.  ...  Acknowledgments Our experiments were conducted using the resources of the Maryland Advanced Research Computing Center (MARCC).  ... 
arXiv:1802.09091v3 fatcat:vaxujd3jynaaxmnjas2jveycv4

Can RNNs trained on harder subject-verb agreement instances still perform well on easier ones? [article]

Hritik Bansal, Gantavya Bhatt, Sumeet Agarwal
2021 arXiv   pre-print
However, we observe that several RNN types, including the ONLSTM which has a soft structural inductive bias, surprisingly fail to perform well on sentences without attractors when trained solely on sentences  ...  Our findings suggest that RNNs trained on our hard agreement instances still do not capture the underlying syntax of agreement, but rather tend to overfit the training distribution in a way which leads  ...  Revisiting the poverty of the stimulus: Hi- erarchical generalization without a hierarchical bias in recurrent neural networks.  ... 
arXiv:2010.04976v2 fatcat:ivbyjydq6re5lcea3iufpgm6b4

Cross-Linguistic Syntactic Evaluation of Word Prediction Models [article]

Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina, Tal Linzen
2020 arXiv   pre-print
A range of studies have concluded that neural word prediction models can distinguish grammatical from ungrammatical sentences with high accuracy.  ...  On other constructions, agreement accuracy was generally higher in languages with richer morphology. Multilingual models generally underperformed monolingual models.  ...  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the other supporting  ... 
arXiv:2005.00187v2 fatcat:zhlwi3wcwnb2lg5n3xcnmwb5de

Analysis Methods in Neural Language Processing: A Survey

Yonatan Belinkov, James Glass
2019 Transactions of the Association for Computational Linguistics  
The field of natural language processing has seen impressive progress in recent years, with neural network models replacing many of the traditional systems.  ...  This has led researchers to analyze, interpret, and evaluate neural networks in novel and more fine-grained ways.  ...  Acknowledgments We would like to thank the anonymous reviewers and the action editor for their very helpful comments. This work was supported by the Qatar Computing Research Institute.  ... 
doi:10.1162/tacl_a_00254 fatcat:unfqn4wpmvbofbsuv46djwlm7e

The learnability of abstract syntactic principles

Amy Perfors, Joshua B. Tenenbaum, Terry Regier
2011 Cognition  
These generalizations must be guided by some inductive bias -some abstract knowledge -that leads them to prefer the correct hypotheses even in the absence of directly supporting evidence.  ...  could recognize the hierarchical phrase structure of language without having this knowledge innately specified as part of the language faculty.  ...  A hierarchical Bayesian model for assessing Poverty of Stimulus arguments.  ... 
doi:10.1016/j.cognition.2010.11.001 pmid:21186021 fatcat:c4c6r6degjenviogz76oapkv5y

SyGNS: A Systematic Generalization Testbed Based on Natural Language Semantics [article]

Hitomi Yanaka, Koji Mineshima, Kentaro Inui
2021 arXiv   pre-print
Recently, deep neural networks (DNNs) have achieved great success in semantically challenging NLP tasks, yet it remains unclear whether DNN models can capture compositional meanings, those aspects of meaning  ...  We also find that the generalization performance to unseen combinations is better when the form of meaning representations is simpler.  ...  Acknowledgement We thank the three anonymous reviewers for their helpful comments and suggestions. This work was partially supported by JSPS KAKENHI Grant Number JP20K19868.  ... 
arXiv:2106.01077v1 fatcat:g3mjlg6s7fdunipvbqb3kl6ep4

The neural basis of cognitive development: a constructivist manifesto

S R Quartz, T J Sejnowski
1997 Behavioral and Brain Sciences  
Neural constructivism suggests that the evolutionary emergence of neocortex in mammals is a progression toward more flexible representational structures, in contrast to the popular view of cortical evolution  ...  The interaction between the environment and neural growth results in a flexible type of learning: "constructive learning" minimizes the need for prespecification in accordance with recent neurobiological  ...  The poverty may not have been in the stimulus but in the minds of nativists.  ... 
pmid:10097006 fatcat:ammov422u5dy7aouvbm5ezhg2e

The neural basis of cognitive development: A constructivist manifesto

Steven R. Quartz, Terrence J. Sejnowski
1997 Behavioral and Brain Sciences  
Neural constructivism suggests that the evolutionary emergence of neocortex in mammals is a progression toward more flexible representational structures, in contrast to the popular view of cortical evolution  ...  The interaction between the environment and neural growth results in a flexible type of learning: "constructive learning" minimizes the need for prespecification in accordance with recent neurobiological  ...  The poverty may not have been in the stimulus but in the minds of nativists.  ... 
doi:10.1017/s0140525x97001581 fatcat:2x57otwpxrcwpklsixe3sukshu

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference [article]

R. Thomas McCoy and Ellie Pavlick and Tal Linzen
2019 arXiv   pre-print
We find that models trained on MNLI, including BERT, a state-of-the-art model, perform very poorly on HANS, suggesting that they have indeed adopted these heuristics.  ...  A machine learning system can score well on a given test set by relying on heuristics that are effective for frequent example types but break down in more challenging cases.  ...  Acknowledgments We are grateful to Adam Poliak, Benjamin Van Durme, Samuel Bowman, the members of the JSALT General-Purpose Sentence Representation Learning team, and the members of the Johns Hopkins Computation  ... 
arXiv:1902.01007v4 fatcat:d5tgj7g6cnbvlhwcmumtpp64d4

The conscious and the unconscious: A package deal

Martin Kurthen
2002 Behavioral and Brain Sciences  
We propose that the isomorphism generally observed between the representations composing our momentary phenomenal experience and the structure of the world is the end-product of a progressive organization  ...  that emerges thanks to elementary associative processes that take our conscious representations themselves as the stuff on which they operate, a thesis that we summarize in the concept of Self-Organizing  ...  I am most grateful to Rachel Murray for her help in preparing this manuscript as a plain text HTML file, and to Christina Bandomir, Geert-Jan Boudewijnse, Sylvie Hebért, Andrea Kilgour, Kyunghwa Kwak,  ... 
doi:10.1017/s0140525x02360060 fatcat:yfjqvbt2avcrlj7yfs7i3rszvq

Open Sesame: Getting inside BERT's Linguistic Knowledge

Yongjie Lin, Yi Chern Tan, Robert Frank
2019 Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP   unpublished
We present here two studies which aim to provide a better understanding of the nature of BERT's representations.  ...  In both cases, we find that BERT encodes positional information about word tokens well on its lower layers, but switches to a hierarchically-oriented encoding on higher layers.  ...  Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks.  ... 
doi:10.18653/v1/w19-4825 fatcat:j4ciubms6fatfg4aynigpm3b7y

Language as Shaped by the Brain [chapter]

Morten H. Christiansen, Nick Chater
2016 Creating Language  
This suggests that apparently arbitrary aspects of linguistic structure may result from general learning and processing biases, independent of language.  ...  This relationship is frequently suggested to be rooted in a language-specific biological endowment, which encodes universal, but arbitrary, principles of language structure (a universal grammar or UG).  ...  Ryskamp Fellowship from the American Council of Learned Societies and by the Santa Fe Institute; NC was supported by a Senior Research Fellowship from the Leverhulme Trust.  ... 
doi:10.7551/mitpress/9780262034319.003.0002 fatcat:caqssxuqkne2nd7kcyshax7p4m

The self-organizing consciousness

Pierre Perruchet, Annie Vinter
2002 Behavioral and Brain Sciences  
We propose that the isomorphism generally observed between the representations composing our momentary phenomenal experience and the structure of the world is the end-product of a progressive organization  ...  that emerges thanks to elementary associative processes that take our conscious representations themselves as the stuff on which they operate, a thesis that we summarize in the concept of Self-Organizing  ...  This work was supported by a grant from the Université Libre de Bruxelles in support of IUAP program P4/19 and by grant HPRN-CT-2000-00065 from the European Commission.  ... 
doi:10.1017/s0140525x02000067 fatcat:xwzlrlhvnfd3bozxoriqtiyeq4

Language as shaped by the brain

Morten H. Christiansen, Nick Chater
2008 Behavioral and Brain Sciences  
This suggests that apparently arbitrary aspects of linguistic structure may result from general learning and processing biases, independent of language.  ...  This relationship is frequently suggested to be rooted in a language-specific biological endowment, which encodes universal, but arbitrary, principles of language structure (a universal grammar or UG).  ...  Ryskamp Fellowship from the American Council of Learned Societies and by the Santa Fe Institute; NC was supported by a Senior Research Fellowship from the Leverhulme Trust.  ... 
doi:10.1017/s0140525x08004998 pmid:18826669 fatcat:2jlqnbtcwnanpborlom4uy5ffe

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference

Tom McCoy, Ellie Pavlick, Tal Linzen
2019 Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics  
We find that models trained on MNLI, including BERT, a state-of-the-art model, perform very poorly on HANS, suggesting that they have indeed adopted these heuristics.  ...  A machine learning system can score well on a given test set by relying on heuristics that are effective for frequent example types but break down in more challenging cases.  ...  Acknowledgments We are grateful to Adam Poliak, Benjamin Van Durme, Samuel Bowman, the members of the JSALT General-Purpose Sentence Representation Learning team, and the members of the Johns Hopkins Computation  ... 
doi:10.18653/v1/p19-1334 dblp:conf/acl/McCoyPL19 fatcat:mgyqntzj6bf2lnz7wyqjdbr4ca
« Previous Showing results 1 — 15 out of 153 results