527 Hits in 9.1 sec

Three steps is all you need: fast, accurate, automatic scaling decisions for distributed streaming dataflows

Vasiliki Kalavri, John Liagouris, Moritz Hoffmann, Desislava C. Dimitrova, Matthew Forshaw, Timothy Roscoe
2018 USENIX Symposium on Operating Systems Design and Implementation  
We present DS2, an automatic scaling controller for such systems which combines a general performance model of streaming dataflows with lightweight instrumentation to estimate the true processing and output  ...  The process is cumbersome, slow and often inefficient. Where automatic scaling is supported, policies rely on coarse-grained metrics like observed throughput, backpressure, and CPU utilization.  ...  Vasiliki Kalavri is supported by an ETH Postdoctoral fellowship.  ... 
dblp:conf/osdi/KalavriLHDFR18 fatcat:5ddhkycprjcyzmfzdv5otzunki

$$\mathbb {ECHO}$$ : An Adaptive Orchestration Platform for Hybrid Dataflows across Cloud and Edge [chapter]

Pushkara Ravindra, Aakash Khochare, Siva Prakash Reddy, Sarthak Sharma, Prateeksha Varshney, Yogesh Simmhan
2017 Lecture Notes in Computer Science  
But a key limitation is the lack of a platform-as-a-service for applications spanning Edge and Cloud. Here, we propose ECHO, an orchestration platform for dataflows across distributed resources.  ...  The Internet of Things (IoT) is offering unprecedented observational data that are used for managing Smart City utilities.  ...  We would also like to thank Venkatesh Babu and Avishek from the VAL lab at IISc for inputs on the video analytics deep-learning model.  ... 
doi:10.1007/978-3-319-69035-3_28 fatcat:5uspil6byrc7par2i7gnylow7q

MacroBase: Prioritizing Attention in Fast Data [article]

Peter Bailis and Edward Gan and Samuel Madden and Deepak Narayanan and Kexin Rong and Sahaana Suri
2017 arXiv   pre-print
sketch specialized for fast data streams.  ...  MacroBase enables efficient, accurate, and modular analyses that highlight and aggregate important and unusual behavior, acting as a search engine for fast data.  ...  providing feedback on and inspiration for this work.  ... 
arXiv:1603.00567v4 fatcat:plqi22difnal3dzzd7jy6qwcxu


Peter Bailis, Edward Gan, Samuel Madden, Deepak Narayanan, Kexin Rong, Sahaana Suri
2017 Proceedings of the 2017 ACM International Conference on Management of Data - SIGMOD '17  
sketch specialized for fast data streams.  ...  MacroBase enables efficient, accurate, and modular analyses that highlight and aggregate important and unusual behavior, acting as a search engine for fast data.  ...  providing feedback on and inspiration for this work.  ... 
doi:10.1145/3035918.3035928 dblp:conf/sigmod/BailisGMNRS17 fatcat:ntyrcf2txffnxdcq22fhxyz234

StreamBlocks: A compiler for heterogeneous dataflow computing (technical report) [article]

Endri Bezati, Mahyar Emami, Jörn Janneck, James Larus
2021 arXiv   pre-print
A key challenge in designing these systems is partitioning computation between processors and an FPGA.  ...  Because of the dataflow model's semantics and the CAL language, StreamBlocks can exploit both thread parallelism in multi-core CPUs and the inherent parallelism of FPGAs.  ...  Janneck is funded by the ELLIIT program of the Swedish government.  ... 
arXiv:2107.09333v1 fatcat:kw2iszud6fhyxlczkesqqeymai

Design of a recommendation system based on collaborative filtering and machine learning considering personal needs of the user

Vasyl Lytvyn, Victoria Vysotska, Viktor Shatskykh, Ihor Kohut, Oksana Petruchenko, Lyudmyla Dzyubyk, Vitaliy Bobrivetc, Valentyna Panasyuk, Svitlana Sachenko, Myroslav Komar
2019 Eastern-European Journal of Enterprise Technologies  
Recommended movies are shown at the section "Recommended For You" at the home page of IMDb [14] .  ...  At the second stage, based on the processing of a flow of service data at steps <3-5> [32], the information is transmitted and stored as a preliminary processed data stream from the user, specified in  ... 
doi:10.15587/1729-4061.2019.175507 fatcat:lq2avvq455hkbkcpafyl25kunq

Predictive topology refinements in distributed stream processing system

Muhammad Hanif, Choonhwa Lee, Sumi Helal, Rashid Mehmood
2020 PLoS ONE  
Apache Flink distributed processing engine is used as a testbed in the paper. The result shows that the prediction scheme works well for both workloads, i.e., synthetic as well as real traces of data.  ...  Good quality of service (QoS) is a must for the enterprises, as they strive to survive in a competitive environment.  ...  To fulfill these analytical applications needs, several distributed stream processing systems have been developed, which can process large and fast continuous streams of data on the fly and respond to  ... 
doi:10.1371/journal.pone.0240424 pmid:33151974 fatcat:mfq6n4x2xbhn3bdggs3t6s6v2a


Michail Tsenos, Aristotelis Peri, Vana Kalogeraki
2022 Proceedings of the 16th ACM International Conference on Distributed and Event-Based Systems  
In this paper, we present AMESoS, our scalable and elastic framework for latency-sensitive streaming pipelines.  ...  function in the pipeline, and (iii) dynamically scales the number of replicas for each pipeline's functions in the presence of overloads.  ...  Apache Flink [2] is a distributed processing engine for stateful computations over unbounded and bounded data streams.  ... 
doi:10.1145/3524860.3539642 fatcat:nao5ibvs4jfe3mldpnqyjkrcta

Resilient Execution of Data-triggered Applications on Edge, Fog and Cloud Resources [article]

Prateeksha Varshney, Shriram Ramesh, Shayal Chhabra, Aakash Khochare, Yogesh Simmhan
2022 arXiv   pre-print
We propose an innovative application model to declaratively specify queries to match streams of micro-batch data from stream sources and trigger the distributed execution of data pipelines.  ...  There is a lack of intuitive means to deploy application pipelines to consume such diverse streams, and to execute them reliably on edge and fog resources.  ...  We thank the members of the DREAM:Lab at IISc, including Prashanthi S.K. and Deepsubhra Guha Roy, for their feedback on the CoFEE platform and the paper.  ... 
arXiv:2203.13324v1 fatcat:y4bocdz4snhhbcf6widi73etoq

A dataflow pattern catalog for sound and music computing

Pau Arumí, David García, Xavier Amatriain
2006 Proceedings of the 2006 conference on Pattern languages of programs - PLoP '06  
Contributions of this paper are: General Dataflow Patterns, that address problems about how to organize highlevel aspects of the dataflow architecture, by having different types of modules connections;  ...  Flow Implementation Patterns, that address how to physically transfer tokens from one module to another, according to the types of flow defined by the "general dataflow patterns".  ...  We couldn't have even imagined all the real-world experience that Ralph accumulates on the topic of dataflow systems!  ... 
doi:10.1145/1415472.1415503 fatcat:bqxhqlzxzjdvjfjlxjlw2f6fge

Efficient System-Level Hardware Synthesis of Dataflow Programs Using Shared Memory Based FIFO

Mariem Abid, Khaled Jerbi, Mickaël Raulet, Olivier Déforges, Mohamed Abid
2017 Journal of Signal Processing Systems  
The basic premise is to model the behavior of the entire system using high-level specifications, and to enable the automatic synthesis to low-level specifications for efficient implementation in Field-Programmable  ...  The design flow combines a dataflow compiler for generating C-based HLS descriptions from a dataflow description and a C-to-gate synthesizer for generating Register-Transfer Level (RTL) descriptions.  ...  The YUV splitting is applied to all FUs except the Algo Parser and the xIT FUs since the bit-stream of the three layers is merged in the input video stream and it would be difficult to separate it.  ... 
doi:10.1007/s11265-017-1226-x fatcat:ewevwhnbibao5hsh26fgjl2tga

Visual Programming in the Wild: A Survey of LabVIEW Programmers

2001 Journal of Visual Languages and Computing  
Acknowledgements Our gratitude goes to the survey respondents for their time. We thank Doug Fisher, Thomas Green and Laura Novick for their helpful feedback on our work.  ...  We thank National Instruments, particularly Lisa Wells, for providing prizes as an incentive to respondents.  ...  After the coding step was completed, the final step was to calculate three tallies for each theme: z Positive tally.  ... 
doi:10.1006/jvlc.2000.0198 fatcat:6pv3yyw2hfdldlon2n53sdazwi

VIoLET: A Large-Scale Virtual Environment for Internet of Things [chapter]

Shreyas Badiger, Shrey Baheti, Yogesh Simmhan
2018 Lecture Notes in Computer Science  
Here, we propose VIoLET, a virtual environment for defining and launching large-scale IoT deployments within cloud VMs.  ...  Despite the intense interest from researchers and practitioners, most do not have access to large-scale IoT testbeds for validation.  ...  The need comes from the availability of large volumes of data streams that need to be analyzed closer to the edge to conserve bandwidth (e.g., video surveillance), or of fast data streams that need to  ... 
doi:10.1007/978-3-319-96983-1_22 fatcat:bxgzvxxssjeeppbxd4c3tybp7u

A Survey on Automatic Parameter Tuning for Big Data Processing Systems

Herodotos Herodotou, Yuxing Chen, Jiaheng Lu
2020 ACM Computing Surveys  
may depend on the setting of a different parameter) [61, 66]. (2) System scale and complexity: As data analytics platforms have grown in scale and complexity, system administrators may need to configure  ...  The use of automated parameter tuning techniques is a promising, yet challenging approach for optimizing system performance.  ...  Spark, providing two different APIs for implementing large-scale parallel algorithms: (i) a Pregel abstraction and (ii) a general MapReduce style API. (4) Spark Streaming [7] , which employs Spark fast  ... 
doi:10.1145/3381027 fatcat:7aglimtuwze25boptuano4ufdy

BigDataGrapes D6.1 - Integrated Software Stack and APIs

Timos Lanitis, Giannis Stoitsis, Panagiotis Rousis, Ioanna Polychronou, Mihalis Papakonstadinou
2020 Zenodo  
The next chapters are organized according to the BigDataGrapes Software Stack Architecture and provide all the necessary steps for deploying the software components of the BigDataGrapes Software Stack.  ...  The document first introduces the chosen deployment platform in which all the current and the future software components will be hosted.  ...  The Apache Flink is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications.  ... 
doi:10.5281/zenodo.4546208 fatcat:pl3444azijbt5px3b7k3to4hmq
« Previous Showing results 1 — 15 out of 527 results