Filters








254 Hits in 6.1 sec

Lightweight Asynchronous Snapshots for Distributed Dataflows [article]

Paris Carbone, Gyula Fóra, Stephan Ewen, Seif Haridi, Kostas Tzoumas
2015 arXiv   pre-print
In this work we propose Asynchronous Barrier Snapshotting (ABS), a lightweight algorithm suited for modern dataflow execution engines that minimises space requirements.  ...  Existing approaches rely on periodic global state snapshots that can be used for failure recovery. Those approaches suffer from two main drawbacks.  ...  In this work, we focus on providing lightweight snapshotting, specifically targeted at distributed stateful dataflow systems, with low impact on performance.  ... 
arXiv:1506.08603v1 fatcat:b7fidps35jg2hblp5eapccq4ee

Apache Flink™: Stream and Batch Processing in a Single Engine

Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, Kostas Tzoumas
2015 IEEE Data Engineering Bulletin  
, and a batch offline path for late accurate results.  ...  real-time analytics, continuous data pipelines, historic data processing (batch), and iterative algorithms (machine learning, graph analysis) can be expressed and executed as pipelined fault-tolerant dataflows  ...  ABS bears resemblances to the Chandy-Lamport algorithm for asynchronous distributed snapshots [11] .  ... 
dblp:journals/debu/CarboneKEMHT15 fatcat:xzgvdr6pljctzb75xecvg74m3q

AutoFlow: Hotspot-Aware, Dynamic Load Balancing for Distributed Stream Processing [article]

Pengqi Lu and Liang Yuan and Yunquan Zhang and Hang Cao and Kun Li
2021 arXiv   pre-print
We introduce AutoFlow, an automatic, hotspot-aware dynamic load balance system for streaming dataflows.  ...  The scheduler achieves these two tasks using a simple asynchronous distributed control message mechanism and a hotspot-diminishing algorithm.  ...  Asynchronous Barrier Snapshot [6] is another technique that has been successfully applied to Flink [7] as a checkpointing mechanism.  ... 
arXiv:2103.08888v1 fatcat:peu5wnkbbvaeljsfywryoiizja

Implementing Software Resiliency in HPX for Extreme Scale Computing [article]

Nikunj Gupta, Jackson R. Mayo, Adrian S. Lemoine, Hartmut Kaiser
2020 arXiv   pre-print
We test the APIs with both artificial workloads and a dataflow based stencil application.  ...  In this paper, we implement software resilience in HPX, an Asynchronous Many-Task Runtime system.  ...  HPX HPX [16] - [20] is a C++ standard library for distributed and parallel programming built on top of an asynchronous manytask (AMT) runtime system.  ... 
arXiv:2004.07203v1 fatcat:cyor2hx3qbg5fltj2afi24mwj4

ELF: Efficient Lightweight Fast Stream Processing at Scale

Liting Hu, Karsten Schwan, Hrishikesh Amur, Xin Chen
2014 USENIX Annual Technical Conference  
Stream processing has become a key means for gaining rapid insights from webserver-captured data.  ...  Job masters at the roots of SRTs can dynamically customize worker actions, obtain aggregated results for end user delivery and/or coordinate with other jobs.  ...  Proceeding in this fashion, a late-coming snapshot eventually blocks the entire upstream path to the root. All snapshots from distributed CBTs are thus sequentially aggregated.  ... 
dblp:conf/usenix/HuSAC14 fatcat:st5ewmw6bfaxre2bnyouo34v7i

Asynchronous Complex Analytics in a Distributed Dataflow Architecture [article]

Joseph E. Gonzalez, Peter Bailis, Michael I. Jordan, Michael J. Franklin, Joseph M. Hellerstein, Ali Ghodsi, Ion Stoica
2015 arXiv   pre-print
In this work, we attempt to bridge this gap by evaluating the use of lightweight, asynchronous state transfer within a commodity dataflow engine.  ...  Scalable distributed dataflow systems have recently experienced widespread adoption, with commodity dataflow engines such as Hadoop and Spark, and even commodity SQL engines routinely supporting increasingly  ...  Unlike a single machine, a distributed dataflow engine has no natural point of rendezvous for asynchronous, in-band solver communication.  ... 
arXiv:1510.07092v1 fatcat:32qafevpyjdfffjl64fvpkwnyi

A Model and Survey of Distributed Data-Intensive Systems [article]

Alessandro Margara, Gianpaolo Cugola, Nicolò Felicioni, Stefano Cilloni
2022 arXiv   pre-print
and implementation strategies to handle data characteristics such as volume, velocity, heterogeneity, and distribution.  ...  These challenges radically transformed all research fields that gravitate around data management and processing, with the introduction of distributed data-intensive systems that offer new programming models  ...  Recent versions also support distributed transactions using two-phase commit for atomicity and multi-version concurrency control for snapshot isolation. CouchDB.  ... 
arXiv:2203.10836v1 fatcat:xbg34nuzhndwjpv2asbbhtgvlu

Spatial computation

Mihai Budiu, Girish Venkataramani, Tiberiu Chelcea, Seth Copen Goldstein
2004 ACM SIGOPS Operating Systems Review  
SC program implementations are completely distributed, with no centralized control. SC circuits are optimized for wires at the expense of computation units.  ...  As a consequence, communication on the datapath never requires arbitration; the only arbitration required is for accessing memory.  ...  We thank Dan Vogel for help with scripting and benchmark management. Finally, we wish to thank the many reviewers for their helpful comments.  ... 
doi:10.1145/1037949.1024396 fatcat:gycsxj3ebfhazpstc2dbx6ebiq

Spatial computation

Mihai Budiu, Girish Venkataramani, Tiberiu Chelcea, Seth Copen Goldstein
2004 SIGARCH Computer Architecture News  
SC program implementations are completely distributed, with no centralized control. SC circuits are optimized for wires at the expense of computation units.  ...  As a consequence, communication on the datapath never requires arbitration; the only arbitration required is for accessing memory.  ...  We thank Dan Vogel for help with scripting and benchmark management. Finally, we wish to thank the many reviewers for their helpful comments.  ... 
doi:10.1145/1037947.1024396 fatcat:5jkfjbhrdzamrdmhosahxd6dzu

Spatial computation

Mihai Budiu, Girish Venkataramani, Tiberiu Chelcea, Seth Copen Goldstein
2004 SIGPLAN notices  
SC program implementations are completely distributed, with no centralized control. SC circuits are optimized for wires at the expense of computation units.  ...  As a consequence, communication on the datapath never requires arbitration; the only arbitration required is for accessing memory.  ...  We thank Dan Vogel for help with scripting and benchmark management. Finally, we wish to thank the many reviewers for their helpful comments.  ... 
doi:10.1145/1037187.1024396 fatcat:5jeulzqygbfnnkch33wohm3imi

Spatial computation

Mihai Budiu, Girish Venkataramani, Tiberiu Chelcea, Seth Copen Goldstein
2004 Proceedings of the 11th international conference on Architectural support for programming languages and operating systems - ASPLOS-XI  
SC program implementations are completely distributed, with no centralized control. SC circuits are optimized for wires at the expense of computation units.  ...  As a consequence, communication on the datapath never requires arbitration; the only arbitration required is for accessing memory.  ...  We thank Dan Vogel for help with scripting and benchmark management. Finally, we wish to thank the many reviewers for their helpful comments.  ... 
doi:10.1145/1024393.1024396 dblp:conf/asplos/BudiuVCG04 fatcat:ncnfj5flsrakpax7vhaf5io3ja

Efficient Migration of Very Large Distributed State for Scalable Stream Processing

Bonaventura Del Monte
2017 Very Large Data Bases Conference  
We propose an incremental migration mechanism for fine-grained state shards through periodic incremental checkpoints and replica groups.  ...  Therefore, how to guarantee those features in the presence of large distributed state with minimal impact on the performance is still an open issue.  ...  In particular, Apache Flink features a technique that asynchronously checkpoints the global states to minimize the latency of a snapshot [3] .  ... 
dblp:conf/vldb/Monte17 fatcat:xghm7nrosnd2nmzkoncigf4cdm

Naiad

Derek G. Murray, Frank McSherry, Rebecca Isaacs, Michael Isard, Paul Barham, Martín Abadi
2013 Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles - SOSP '13  
Naiad is a distributed system for executing data parallel, cyclic dataflow programs.  ...  This model enriches dataflow computation with timestamps that represent logical points in the computation and provide the basis for an efficient, lightweight coordination mechanism.  ...  We would also like to thank the anonymous SOSP reviewers for their comments, and Robert Morris for his shepherding of the paper.  ... 
doi:10.1145/2517349.2522738 dblp:conf/sosp/MurrayMIIBA13 fatcat:kirmhcn2ynaxzleflnsfnamyaa

Convergence in Language Design: A Case of Lightning Striking Four Times in the Same Place [chapter]

Peter Van Roy
2006 Lecture Notes in Computer Science  
distributed programming, and teaching programming as a unified discipline.  ...  I present four case studies of substantial research projects that tackle important problems in four quite different areas: fault-tolerant programming, secure distributed programming, network-transparent  ...  We would like to thank the members of the Programming Languages and Distributed Computing group at UCL for creating the environment in which the speculations of this article could arise.  ... 
doi:10.1007/11737414_2 fatcat:t4zv7hbbengrxggd2hen6757l4

Dynamic control flow in large-scale machine learning

Yuan Yu, Peter Hawkins, Michael Isard, Manjunath Kudlur, Rajat Monga, Derek Murray, Xiaoqiang Zheng, Martín Abadi, Paul Barham, Eugene Brevdo, Mike Burrows, Andy Davis (+3 others)
2018 Proceedings of the Thirteenth EuroSys Conference on - EuroSys '18  
This paper presents a programming model for distributed machine learning that supports dynamic control flow.  ...  For performance, scalability, and expressiveness, a machine learning system must support dynamic control flow in distributed and heterogeneous environments.  ...  We also thank our shepherd, Peter Pietzuch, for his guidance in improving the paper.  ... 
doi:10.1145/3190508.3190551 dblp:conf/eurosys/YuABBBDDGHHIKMM18 fatcat:5u4gcsi5fba33mv2nyni32h424
« Previous Showing results 1 — 15 out of 254 results