Filters








5 Hits in 2.0 sec

StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling [article]

Yikang Shen, Yi Tay, Che Zheng, Dara Bahri, Donald Metzler, Aaron Courville
2021 arXiv   pre-print
Experimental results show that our model can achieve strong results on unsupervised constituency parsing, unsupervised dependency parsing, and masked language modeling at the same time.  ...  While previous unsupervised parsing methods mostly focus on only inducing one class of grammars, we introduce a novel model, StructFormer, that can simultaneously induce dependency and constituency structure  ...  the proposed model on three tasks: Masked Language Modeling, Unsupervised Constituency Parsing and Unsupervised Dependency Parsing.  ... 
arXiv:2012.00857v3 fatcat:ph3pvht7gvenzdnr7lcftvlegm

StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling

Yikang Shen, Yi Tay, Che Zheng, Dara Bahri, Donald Metzler, Aaron Courville
2021 Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)   unpublished
Experimental results show that our model can achieve strong results on unsupervised constituency parsing, unsupervised dependency parsing, and masked language modeling at the same time.  ...  While previous unsupervised parsing methods mostly focus on only inducing one class of grammars, we introduce a novel model, StructFormer, that can simultaneously induce dependency and constituency structure  ...  the proposed model on three tasks: Masked Language Modeling, Unsupervised Constituency Parsing and Unsupervised Dependency Parsing.  ... 
doi:10.18653/v1/2021.acl-long.559 fatcat:kduvwkoszfeb7kbboy34bqo3ji

Phrase-aware Unsupervised Constituency Parsing

Xiaotao Gu, Yikang Shen, Jiaming Shen, Jingbo Shang, Jiawei Han
2022 Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)   unpublished
Recent studies have achieved inspiring success in unsupervised grammar induction using masked language modeling (MLM) as the proxy task.  ...  We show that the initial phrase regularization serves as an effective bootstrap, and phrase-guided masking improves the identification of high-level structures.  ...  StructFormer (Shen et al., 2020) propose a joint dependency and constituency parser, then uses the dependency adja-cency matrix to constraint the self-attention heads in transformer models. 2) Using  ... 
doi:10.18653/v1/2022.acl-long.444 fatcat:jxmhdz3nobditbmxytxiqroeii

Unsupervised Dependency Graph Network

Yikang Shen, Shawn Tan, Alessandro Sordoni, Peng Li, Jie Zhou, Aaron Courville
2022 Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)   unpublished
We introduce a new model, the Unsupervised Dependency Graph Network (UDGN), that can induce dependency structures from raw corpora and the masked language modeling task.  ...  Recent work has identified properties of pretrained self-attention models that mirror those of dependency parse structures.  ...  This joint parser-language model framework can induce grammar from masked language modeling tasks.  ... 
doi:10.18653/v1/2022.acl-long.327 fatcat:cephsaawxndpbawvjagwpvw54y

Neural Bi-Lexicalized PCFG Induction

Songlin Yang, Yanpeng Zhao, Kewei Tu
2021 Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)   unpublished
Our approach directly models bilexical dependencies and meanwhile reduces both learning and representation complexities of L-PCFGs.  ...  Experimental results on the English WSJ dataset confirm the effectiveness of our approach in improving both running speed and unsupervised parsing performance.  ...  This work was supported by the National Natural Science Foundation of China (61976139).  ... 
doi:10.18653/v1/2021.acl-long.209 fatcat:sprn4xbydvhz3aa2fqtniwnnpa