A Study of Realtime Summarization Metrics

Matthew Ekstrand-Abueg, Richard McCreadie, Virgil Pavlu, Fernando Diaz
<span title="">2016</span> <i title="ACM Press"> <a target="_blank" rel="noopener" href="https://fatcat.wiki/container/6g37zvjwwrhv3dizi6ffue642m" style="color: black;">Proceedings of the 25th ACM International on Conference on Information and Knowledge Management - CIKM &#39;16</a> </i> &nbsp;
Unexpected news events, such as natural disasters or other human tragedies, create a large volume of dynamic text data from official news media as well as less formal social media. Automatic real-time text summarization has become an important tool for quickly transforming this overabundance of text into clear, useful information for end-users including affected individuals, crisis responders, and interested third parties. Despite the importance of real-time summarization systems, their
more &raquo; ... on is not well understood as classic methods for text summarization are inappropriate for realtime and streaming conditions. The TREC 2013-2015 Temporal Summarization (TREC-TS) track was one of the first evaluation campaigns to tackle the challenges of real-time summarization evaluation, introducing new metrics, ground-truth generation methodology and dataset. In this paper, we present a study of TREC-TS track evaluation methodology, with the aim of documenting its design, analyzing its effectiveness, as well as identifying improvements and best practices for the evaluation of temporal summarization systems.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1145/2983323.2983653">doi:10.1145/2983323.2983653</a> <a target="_blank" rel="external noopener" href="https://dblp.org/rec/conf/cikm/Ekstrand-AbuegM16.html">dblp:conf/cikm/Ekstrand-AbuegM16</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/qb2tuidwk5g4tdkpbhx3xej6vm">fatcat:qb2tuidwk5g4tdkpbhx3xej6vm</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20180719200730/http://eprints.gla.ac.uk/148597/1/148597.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/bc/2c/bc2c5d8bac25a4ecd2738e73e5a437c39f3489a7.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1145/2983323.2983653"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="external alternate icon"></i> acm.org </button> </a>