Sentence-Level Fluency Evaluation: References Help, But Can Be Spared!

Katharina Kann, Sascha Rothe, Katja Filippova
2018 Proceedings of the 22nd Conference on Computational Natural Language Learning  
Motivated by recent findings on the probabilistic modeling of acceptability judgments, we propose syntactic log-odds ratio (SLOR), a normalized language model score, as a metric for referenceless fluency evaluation of natural language generation output at the sentence level. We further introduce WPSLOR, a novel WordPiece-based version, which harnesses a more compact language model. Even though word-overlap metrics like ROUGE are computed with the help of hand-written references, our
more » ... es, our referenceless methods obtain a significantly higher correlation with human fluency scores on a benchmark dataset of compressed sentences. Finally, we present ROUGE-LM, a reference-based metric which is a natural extension of WPSLOR to the case of available references. We show that ROUGE-LM yields a significantly higher correlation with human judgments than all baseline metrics, including WPSLOR on its own.
doi:10.18653/v1/k18-1031 dblp:conf/conll/KannRF18 fatcat:vp2fxiceqbghjnl6fx4jdlet54