Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability

Jonathan H. Clark, Chris Dyer, Alon Lavie, Noah A. Smith
2011 Annual Meeting of the Association for Computational Linguistics  
In statistical machine translation, a researcher seeks to determine whether some innovation (e.g., a new feature, model, or inference algorithm) improves translation quality in comparison to a baseline system. To answer this question, he runs an experiment to evaluate the behavior of the two systems on held-out data. In this paper, we consider how to make such experiments more statistically reliable. We provide a systematic analysis of the effects of optimizer instability-an extraneous variable
more » ... that is seldom controlled for-on experimental outcomes, and make recommendations for reporting results more accurately.
dblp:conf/acl/ClarkDLS11 fatcat:5jgr7xk5pbfujazl5r7b7iu4xq