Sanskrit Word Segmentation Using Character-level Recurrent and Convolutional Neural Networks

Oliver Hellwig, Sebastian Nehrdich
2018 Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing  
The paper introduces end-to-end neural network models that tokenize Sanskrit by jointly splitting compounds and resolving phonetic merges (Sandhi). Tokenization of Sanskrit depends on local phonetic and distant semantic features that are incorporated using convolutional and recurrent elements. Contrary to most previous systems, our models do not require feature engineering or extern linguistic resources, but operate solely on parallel versions of raw and segmented text. The models discussed in
more » ... his paper clearly improve over previous approaches to Sanskrit word segmentation. As they are language agnostic, we will demonstrate that they also outperform the state of the art for the related task of German compound splitting. 2 The compound in Ex. 1 is split asāśraya-bhūta-kha-ādikathanena, and kha +ādi = khādi is a Sandhi phenomenon.
doi:10.18653/v1/d18-1295 dblp:conf/emnlp/HellwigN18 fatcat:ukccvdaedvdh5fqedhdi4jm6hm