Benefits of Intermediate Annotations in Reading Comprehension

Dheeru Dua, Sameer Singh, Matt Gardner
2020 Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics   unpublished
Complex, compositional reading comprehension datasets require performing latent sequential decisions that are learned via supervision from the final answer. A large combinatorial space of possible decision paths that result in the same answer, compounded by the lack of intermediate supervision to help choose the right path, makes the learning particularly hard for this task. In this work, we study the benefits of collecting intermediate reasoning supervision along with the answer during data
more » ... lection. We find that these intermediate annotations can provide two-fold benefits. First, we observe that for any collection budget, spending a fraction of it on intermediate annotations results in improved model performance, for two complex compositional datasets: DROP and Quoref. Second, these annotations encourage the model to learn the correct latent reasoning steps, helping combat some of the biases introduced during the data collection process.
doi:10.18653/v1/2020.acl-main.497 fatcat:5hnb2cz6ubfvlbh5jwzwbzngea