When Are Tree Structures Necessary for Deep Learning of Representations?

Jiwei Li, Thang Luong, Dan Jurafsky, Eduard Hovy
2015 Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing  
Recursive neural models, which use syntactic parse trees to recursively generate representations bottom-up, are a popular architecture. However there have not been rigorous evaluations showing for exactly which tasks this syntax-based method is appropriate. In this paper, we benchmark recursive neural models against sequential recurrent neural models, enforcing applesto-apples comparison as much as possible. We investigate 4 tasks: (1) sentiment classification at the sentence level and phrase
more » ... vel; (2) matching questions to answerphrases; (3) discourse parsing; (4) semantic relation extraction. Our goal is to understand better when, and why, recursive models can outperform simpler models. We find that recursive models help mainly on tasks (like semantic relation extraction) that require longdistance connection modeling, particularly on very long sequences. We then introduce a method for allowing recurrent models to achieve similar performance: breaking long sentences into clause-like units at punctuation and processing them separately before combining. Our results thus help understand the limitations of both classes of models, and suggest directions for improving recurrent models.
doi:10.18653/v1/d15-1278 dblp:conf/emnlp/LiLJH15 fatcat:bcvvcfcvenbubl6qdaz66bua2q