Deep Reinforced Self-Attention Masks for Abstractive Summarization (DR.SAS) [article]

Ankit Chadha, Mohamed Masoud
<span title="2019-12-30">2019</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
We present a novel architectural scheme to tackle the abstractive summarization problem based on the CNN/DMdataset which fuses Reinforcement Learning (RL) withUniLM, which is a pre-trained Deep Learning Model, to solve various natural language tasks. We have tested the limits of learning fine-grained attention in Transformers to improve the summarization quality. UniLM applies attention to the entire token space in a global fashion. We propose DR.SAS which applies the Actor-Critic (AC)
more &raquo; ... to learn a dynamic self-attention distribution over the tokens to reduce redundancy and generate factual and coherent summaries to improve the quality of summarization. After performing hyperparameter tuning, we achievedbetter ROUGE results compared to the baseline. Our model tends to be more extractive/factual yet coherent in detail because of optimization over ROUGE rewards. We present detailed error analysis with examples of the strengths and limitations of our model. Our codebase will be publicly available on our GitHub.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2001.00009v1">arXiv:2001.00009v1</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/v3jnpaybzbdxplyq5zf7wmpipq">fatcat:v3jnpaybzbdxplyq5zf7wmpipq</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200809070422/https://arxiv.org/pdf/2001.00009v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/2f/3e/2f3e6a2872cbce15c9af005cf716d0aab3328ebd.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2001.00009v1" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>