Internet Archive Scholar

TA-MAMC at SemEval-2021 Task 4: Task-adaptive Pretraining and Multi-head Attention for Abstract Meaning Reading Comprehension

Jing Zhang, Yimeng Zhuang, Yinpei Su
2021 Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)   unpublished

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (1.7 MB)
https://web.archive.org/web/20210801175924/https://aclanthology.org/2021.semeval-1.5.pdf

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2021; you can also visit the original URL. The file type is application/pdf.

This paper describes our system used in the SemEval-2021 Task 4 Reading Comprehension of Abstract Meaning, achieving 1st for subtask 1 and 2nd for subtask 2 on the leaderboard. We propose an ensemble of ELECTRAbased models with task-adaptive pretraining and a multi-head attention multiple-choice classifier on top of the pre-trained model. The main contributions of our system are 1) revealing the performance discrepancy of different transformer-based pretraining models on the downstream task, 2)
more » ... presentation of an efficient method to generate large task-adaptive corpora for pretraining. We also investigated several pretraining strategies and contrastive learning objectives. Our system achieves a test accuracy of 95.11 and 94.89 on subtask 1 and subtask 2 respectively. * Contribution during Intership in Samsung Research China-Beijing.
doi:10.18653/v1/2021.semeval-1.5 fatcat:kunw2j4ecbd4zfqfsolzgkdk4y
Citation

Jing Zhang, Yimeng Zhuang, Yinpei Su. "TA-MAMC at SemEval-2021 Task 4: Task-adaptive Pretraining and Multi-head Attention for Abstract Meaning Reading Comprehension." Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021) (2021)