Fault Tolerance for Stream Processing Engines [article]

Muhammad Anis Uddin Nasir
<span title="2020-05-05">2020</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
Distributed Stream Processing Engines (DSPEs) target applications related to continuous computation, online machine learning and real-time query processing. DSPEs operate on high volume of data by applying lightweight operations on real-time and continuous streams. Such systems require clusters of hundreds of machine for their deployment. Streaming applications come with various requirements, i.e., low-latency, high throughput, scalability and high availability. In this survey, we study the
more &raquo; ... t tolerance problem for DSPEs. We discuss fault tolerance techniques that are used in modern stream processing engines that are Storm, S4, Samza, SparkStreaming and MillWheel. Further, we give insight on fault tolerance approaches that we categorize as active replication, passive replication and upstream backup. Finally, we discuss implications of the fault tolerance techniques for different streaming application requirements.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1605.00928v3">arXiv:1605.00928v3</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/kvdgebicrfbktogtew77mv7ppy">fatcat:kvdgebicrfbktogtew77mv7ppy</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20191012194014/https://arxiv.org/pdf/1605.00928v1.pdf" title="fulltext PDF download [not primary version]" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <span style="color: #f43e3e;">&#10033;</span> <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/69/6e/696e7fb84096e5927eccee6607e31bab44a1daa9.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1605.00928v3" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>