BERTtoCNN: Similarity-preserving enhanced knowledge distillation for stance detection

Yang Li, Yuqing Sun, Nana Zhu, Weinan Zhang
<span title="2021-09-10">2021</span> <i title="Public Library of Science (PLoS)"> <a target="_blank" rel="noopener" href="https://fatcat.wiki/container/s3gm7274mfe6fcs7e3jterqlri" style="color: black;">PLoS ONE</a> </i> &nbsp;
In recent years, text sentiment analysis has attracted wide attention, and promoted the rise and development of stance detection research. The purpose of stance detection is to determine the author's stance (favor or against) towards a specific target or proposition in the text. Pre-trained language models like BERT have been proven to perform well in this task. However, in many reality scenes, they are usually very expensive in computation, because such heavy models are difficult to implement
more &raquo; ... ith limited resources. To improve the efficiency while ensuring the performance, we propose a knowledge distillation model BERTtoCNN, which combines the classic distillation loss and similarity-preserving loss in a joint knowledge distillation framework. On the one hand, BERTtoCNN provides an efficient distillation process to train a novel 'student' CNN structure from a much larger 'teacher' language model BERT. On the other hand, based on the similarity-preserving loss function, BERTtoCNN guides the training of a student network, so that input pairs with similar (dissimilar) activation in the teacher network have similar (dissimilar) activation in the student network. We conduct experiments and test the proposed model on the open Chinese and English stance detection datasets. The experimental results show that our model outperforms the competitive baseline methods obviously.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1371/journal.pone.0257130">doi:10.1371/journal.pone.0257130</a> <a target="_blank" rel="external noopener" href="https://www.ncbi.nlm.nih.gov/pubmed/34506549">pmid:34506549</a> <a target="_blank" rel="external noopener" href="https://pubmed.ncbi.nlm.nih.gov/PMC8432858/">pmcid:PMC8432858</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/hnfbgotconefrjpgsx4pmbicu4">fatcat:hnfbgotconefrjpgsx4pmbicu4</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20210915044052/https://storage.googleapis.com/plos-corpus-prod/10.1371/journal.pone.0257130/1/pone.0257130.pdf?X-Goog-Algorithm=GOOG4-RSA-SHA256&amp;X-Goog-Credential=wombat-sa%40plos-prod.iam.gserviceaccount.com%2F20210915%2Fauto%2Fstorage%2Fgoog4_request&amp;X-Goog-Date=20210915T044052Z&amp;X-Goog-Expires=86400&amp;X-Goog-SignedHeaders=host&amp;X-Goog-Signature=272ba857053b4981b3b7f36bd4ad33fe1bd4a5735346a7928af8602465a6d695b8bc7b6b40d372701bddf3fa823169acc209dcc1f29252e2e7ee492dbf2179cd026f46bed5300dc631496a8e68e35f02268d61974dda493241c5d897f4ca228d8bb7c049e293ab964b82437dd22f3262ee7a56a0aedf74d813c398c517309469e498bf48fa219692fe4b7b0536a420accf080143196737b1aebb25eb9808929ea0a149f7d28a4b8c36ef9a113e4534cba543f076def7f480d1cc20894baae36a76650393ee7dfe41c3273fbf0ca13e926f7cbf3adf21fe11cc80d317c6be32c39e52f657f1a9bb043bba58d15733a31a3ee736fb250bdd982e59fbd6e246675b" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/49/23/4923b8349da6bb3bfa8366617abc25e730a47fac.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1371/journal.pone.0257130"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="unlock alternate icon" style="background-color: #fb971f;"></i> plos.org </button> </a> <a target="_blank" rel="external noopener" href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8432858" title="pubmed link"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> pubmed.gov </button> </a>