HeLI, a Word-Based Backoff Method for Language Identification

Tommi Jauhiainen, Krister Lindén, Heidi Jauhiainen
2016 Workshop on NLP for Similar Languages, Varieties and Dialects  
In this paper we describe the Helsinki language identification method, HeLI, and the resources we created for and used in the 3rd edition of the Discriminating between Similar Languages (DSL) shared task, which was organized as part of the VarDial 2016 workshop. The shared task comprised of a total of 8 tracks, of which we participated in 7. The shared task had a record number of participants, with 17 teams providing results for the closed track of the test set A. Our system reached the 2nd
more » ... tion in 4 tracks (A closed and open, B1 open and B2 open) and in this paper we are focusing on the methods and data used for those tracks. We describe our wordbased backoff method in mathematical notation. We also describe how we selected the corpus we used in the open tracks.
dblp:conf/vardial/JauhiainenLJ16 fatcat:veybzhq7ffedldtz2qltcyyx6y