We present a targeted, scaled-up comparison of incremental processing in humans and neural language models by collecting by-word reaction time data for sixteen different syntactic test suites across a range of structural phenomena. Human reaction time data comes from a novel online experimental paradigm called the Interpolated Maze task. We compare human reaction times to by-word probabilities for four contemporary language models, with different architectures and trained on a range of data set

more »

... sizes. We find that across many phenomena, both humans and language models show increased processing difficulty in ungrammatical sentence regions with human and model 'accuracy' scores (à la Marvin and Linzen (2018) ) about equal. However, although language model outputs match humans in direction, we show that models systematically under-predict the difference in magnitude of incremental processing difficulty between grammatical and ungrammatical sentences. Specifically, when models encounter syntactic violations they fail to accurately predict the longer reaction times observed in the human data. These results call into question whether contemporary language models are approaching human-like performance for sensitivity to syntactic violations. NPI Licensing, any, Subj RC Modifier NPL-any-src No/The senator that no/the journalist likes has gotten any votes. NPI Licensing, any, Obj RC Modifier NPL-any-orc No/The senator that likes no/the journalists has gotten any votes. NPI Licensing, ever, Subj RC Modifier NPL-ever-src No/The senator that no/the journalist likes has ever won. NPI Licensing, ever, Obj RC Modifier NPL-ever-orc No/The senator that likes no/the journalists has ever won. Subject-Verb Number Agr., Subj RC Modifier SVNA-src The lawyer/lawyers that helped the mayor is/are organized. Subject-Verb Number Agr., Obj RC Modifier SVNA-orc The lawyer/lawyers that the mayor hired is/are very organized. Subject-Verb Number Agr., PP Modifier SVNA-pp The lawyer/lawyers next to the mayor is/are very organized. Reflexive Anaphora, Masc., Subj RC Modifier RNA-m-src The dukes/duke that hunted the rabbits saw himself/themselves in the mirror. Reflexive Anaphora, Masc., Obj RC Modifier RNA-m-orc The dukes/duke that the knights distrust saw himself/themselves in the mirror. Reflexive Anaphora, Fem., Subj RC Modifier RNA-f-src The queens/queen that hunted the rabbits saw herself/themselves in the mirror. Reflexive Anaphora, Fem., Obj RC Modifier RNA-f-orc The queens/queen that the knights distrust saw herself/themselves in the mirror.