Open-Lexicon Language Modeling Combining Word and Character Levels

Michal Kozielski, Martin Matysiak, Patrick Doetsch, Ralf Schloter, Hermann Ney
2014 2014 14th International Conference on Frontiers in Handwriting Recognition  
In this paper we investigate different n-gram language models that are defined over an open lexicon. We introduce a character-level language model and combine it with a standard word-level language model in a backoff fashion. The character-level language model is redefined and renormalized to assign zero probability to words from a fixed vocabulary. Furthermore we present a way to interpolate language models created at the word and character levels. The computation of character-level
more » ... es incorporates the across-word context. We compare perplexities on all words from the test set and on in-lexicon and OOV words separately on corpora of English and Arabic text.
doi:10.1109/icfhr.2014.64 dblp:conf/icfhr/KozielskiMDSN14 fatcat:v2bi2jrtv5h7rcq5jdr4nvwqou