Continuous Dataflow Update Strategies for Mission-Critical Applications

Charith Wickramaarachchi, Yogesh Simmhan
2013 2013 IEEE 9th International Conference on e-Science  
Continuous dataflows complement scientific workflows by allowing composition of realtime data ingest and analytics pipelines to process data streams from pervasive sensors and "always-on" scientific instruments. Such dataflows are missioncritical applications that cannot suffer downtime, need to operate consistent, and are long running, but may need to be updated to fix bug or add features. This poses the problem: How do we update the continuous dataflow application with minimal disruption? In
more » ... his paper, we formalize different types of dataflow update models for continuous dataflow applications, and identify the qualitative and quantitative metrics to be considered when choosing an update strategy. We propose five dataflow update strategies, and analytically characterize their performance trade-offs. We validate one of these consistent, low-latency update strategies using the F oε dataflow engine for an eEngineering application from the Smart Power Grid domain, and show its relative performance benefits against a naïve update strategy.
doi:10.1109/escience.2013.35 dblp:conf/eScience/WickramaarachchiS13 fatcat:q2yzienphjdn7mwc342sntsodu