Deriving lexical and syntactic expectation-based measures for psycholinguistic modeling via incremental top-down parsing

Brian Roark, Asaf Bachrach, Carlos Cardenas, Christophe Pallier
2009 Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing Volume 1 - EMNLP '09   unpublished
A number of recent publications have made use of the incremental output of stochastic parsers to derive measures of high utility for psycholinguistic modeling, following the work of Hale (2001; . In this paper, we present novel methods for calculating separate lexical and syntactic surprisal measures from a single incremental parser using a lexicalized PCFG. We also present an approximation to entropy measures that would otherwise be intractable to calculate for a grammar of that size.
more » ... results demonstrate the utility of our methods in predicting human reading times.
doi:10.3115/1699510.1699553 fatcat:iqang7qgpvat5jlzazhnpa5uaq