Closing Brackets with Recurrent Neural Networks

Natalia Skachkova, Thomas Trost, Dietrich Klakow
2018 Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP  
Many natural and formal languages contain words or symbols that require a matching counterpart for making an expression wellformed. The combination of opening and closing brackets is a typical example of such a construction. Due to their commonness, the ability to follow such rules is important for language modeling. Currently, recurrent neural networks (RNNs) are extensively used for this task. We investigate whether they are capable of learning the rules of opening and closing brackets by
more » ... ying them to synthetic Dyck languages that consist of different types of brackets. We provide an analysis of the statistical properties of these languages as a baseline and show strengths and limits of Elman-RNNs, GRUs and LSTMs in experiments on random samples of these languages. In terms of perplexity and prediction accuracy, the RNNs get close to the theoretical baseline in most cases.
doi:10.18653/v1/w18-5425 dblp:conf/emnlp/SkachkovaTK18 fatcat:2p2vs2bpmzbcpasdwamtyr35zu