A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit <a rel="external noopener" href="https://arxiv.org/pdf/2001.00009v1.pdf">the original URL</a>. The file type is <code>application/pdf</code>.
Deep Reinforced Self-Attention Masks for Abstractive Summarization (DR.SAS)
[article]
<span title="2019-12-30">2019</span>
<i >
arXiv
</i>
<span class="release-stage" >pre-print</span>
Preserved Fulltext
We present a novel architectural scheme to tackle the abstractive summarization problem based on the CNN/DMdataset which fuses Reinforcement Learning (RL) withUniLM, which is a pre-trained Deep Learning Model, to solve various natural language tasks. We have tested the limits of learning fine-grained attention in Transformers to improve the summarization quality. UniLM applies attention to the entire token space in a global fashion. We propose DR.SAS which applies the Actor-Critic (AC)
<span class="external-identifiers">
<a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2001.00009v1">arXiv:2001.00009v1</a>
<a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/v3jnpaybzbdxplyq5zf7wmpipq">fatcat:v3jnpaybzbdxplyq5zf7wmpipq</a>
</span>
more »
... to learn a dynamic self-attention distribution over the tokens to reduce redundancy and generate factual and coherent summaries to improve the quality of summarization. After performing hyperparameter tuning, we achievedbetter ROUGE results compared to the baseline. Our model tends to be more extractive/factual yet coherent in detail because of optimization over ROUGE rewards. We present detailed error analysis with examples of the strengths and limitations of our model. Our codebase will be publicly available on our GitHub.
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200809070422/https://arxiv.org/pdf/2001.00009v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext">
<button class="ui simple right pointing dropdown compact black labeled icon button serp-button">
<i class="icon ia-icon"></i>
Web Archive
[PDF]
<div class="menu fulltext-thumbnail">
<img src="https://blobs.fatcat.wiki/thumbnail/pdf/2f/3e/2f3e6a2872cbce15c9af005cf716d0aab3328ebd.180px.jpg" alt="fulltext thumbnail" loading="lazy">
</div>
</button>
</a>
<a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2001.00009v1" title="arxiv.org access">
<button class="ui compact blue labeled icon button serp-button">
<i class="file alternate outline icon"></i>
arxiv.org
</button>
</a>