88 Hits in 8.8 sec

ELF: Efficient Lightweight Fast Stream Processing at Scale

Liting Hu, Karsten Schwan, Hrishikesh Amur, Xin Chen
2014 USENIX Annual Technical Conference  
Job masters at the roots of SRTs can dynamically customize worker actions, obtain aggregated results for end user delivery and/or coordinate with other jobs.  ...  The ELF stream processing system addresses these new challenges.  ...  The leaping straggler approach leverages the streaming nature of ELF, maintaining timeliness at reduced levels of result accuracy.  ... 
dblp:conf/usenix/HuSAC14 fatcat:st5ewmw6bfaxre2bnyouo34v7i

Three steps is all you need: fast, accurate, automatic scaling decisions for distributed streaming dataflows

Vasiliki Kalavri, John Liagouris, Moritz Hoffmann, Desislava C. Dimitrova, Matthew Forshaw, Timothy Roscoe
2018 USENIX Symposium on Operating Systems Design and Implementation  
Some modern large-scale stream processors allow dynamic scaling but typically leave the difficult task of deciding how much to scale to the user.  ...  We present DS2, an automatic scaling controller for such systems which combines a general performance model of streaming dataflows with lightweight instrumentation to estimate the true processing and output  ...  Acknowledgements We thank Nicolas Hafner for his help with the Nexmark queries implementation on Timely.  ... 
dblp:conf/osdi/KalavriLHDFR18 fatcat:5ddhkycprjcyzmfzdv5otzunki

Aggregation and Degradation in JetStream: Streaming Analytics in the Wide Area

Ariel Rabkin, Matvey Arye, Siddhartha Sen, Vivek S. Pai, Michael J. Freedman
2014 Symposium on Networked Systems Design and Implementation  
The system incorporates structured storage in the form of OLAP data cubes, so data can be stored for analysis near where it is generated.  ...  Our adaptive control mechanisms are responsive enough to keep end-to-end latency within a few seconds, even when available bandwidth drops by a factor of two, and are flexible enough to express practical  ...  Acknowledgments The authors appreciate the helpful advice and comments of Jinyang Li, Jennifer Rexford, Erik Nordström, Rob Kiefer, our shepherd Ramesh Govindan, and the anonymous reviewers.  ... 
dblp:conf/nsdi/RabkinASPF14 fatcat:j7bryvrplza2tfyeh4sqaispaa

Distributed real-time sentiment analysis for big data social streams

Amir Hossein Akhavan Rahnama
2014 2014 International Conference on Control, Decision and Information Technologies (CoDIT)  
Lastly, the learner needs to provide high analytical accuracy measures.  ...  The real challenge with real-time stream data processing is that it is impossible to store instances of data, and therefore online analytical algorithms are utilized.  ...  a cluster of low-end computers and they guarantee reliability by having no single point of failure.  ... 
doi:10.1109/codit.2014.6996998 dblp:conf/codit/Rahnama14 fatcat:qgrmbsuih5ednifehbihlmcrm4

Asynchronous Federated Learning on Heterogeneous Devices: A Survey [article]

Chenhao Xu, Youyang Qu, Yong Xiang, Longxiang Gao
2022 arXiv   pre-print
Furthermore, the disparity of data spread on devices (i.e. data heterogeneity) in real-world scenarios downgrades the accuracy of models.  ...  Federated learning (FL) is experiencing a fast booming with the wave of distributed machine learning.  ...  To consume real-time streaming data, a sliding training window is introduced to reduce computation and communication latency.  ... 
arXiv:2109.04269v3 fatcat:bcix56mg7zev7hzav4rahkycai

FusionRAID: Achieving Consistent Low Latency for Commodity SSD Arrays

Tianyang Jiang, Guangyan Zhang, Zican Huang, Xiaosong Ma, Junyu Wei, Zhiyue Li, Weimin Zheng
2021 USENIX Conference on File and Storage Technologies  
Our results also reveal that with SSD latency low and decreasing, the software overhead of RAID write creates long, complex write paths involving more drives, raising both average-case latency and risk  ...  Our evaluation with traces and applications shows that FusionRAID brings a 22%-98% reduction in median latency, and a 2.7×-62× reduction in tail latency, with a moderate and temporary space overhead.  ...  This work was supported by the National key R&D Program of China under Grant 2018YFB0203902, and the National Natural Science Foundation of China under Grants 61672315 and 62025203.  ... 
dblp:conf/fast/JiangZHMWLZ21 fatcat:3o7ojqzd2natxhij5zkqca5pha

Scalable progressive analytics on big data in the cloud

Badrish Chandramouli, Jonathan Goldstein, Abdul Quamar
2013 Proceedings of the VLDB Endowment  
works with "progress-aware reducers"-in particular, it works with streaming engines to support progressive SQL over big data.  ...  Extensive experiments on Windows Azure with real and synthetic workloads validate the scalability and benefits of Now! and its optimizations, over current solutions for progressive analytics.  ...  framework that does not target real-time data or low-latency queries.  ... 
doi:10.14778/2556549.2556557 fatcat:taqjyvpk7bggzov5aatoh5svrm

Adaptive Block and Batch Sizing for Batched Stream Processing System

Quan Zhang, Yang Song, Ramani R. Routray, Weisong Shi
2016 2016 IEEE International Conference on Autonomic Computing (ICAC)  
Therefore, we leave the work to support Window workload for the future.  ...  because of their ability to provide low latency analytics on streaming data, which also has led to the development of distributed stream processing systems (DSPS), that are designed to provide fast, scalable  ... 
doi:10.1109/icac.2016.27 dblp:conf/icac/ZhangSRS16 fatcat:3er6legizfeitab7cmrze7n4jq

Trading Timeliness and Accuracy in Geo-Distributed Streaming Analytics

Benjamin Heintz, Abhishek Chandra, Ramesh K. Sitaraman
2016 Proceedings of the Seventh ACM Symposium on Cloud Computing - SoCC '16  
In this paper, we focus on windowed grouped aggregation, an important and widely used primitive in streaming analytics, and we study the tradeoff between the key metrics of staleness and error.  ...  Whether the input streams represent sensor data from smart homes, user interaction logs from streaming video clients, or server logs from a content delivery network (CDN), it is common for such streams  ...  Updates (Random): The Random algorithm effectively combines batching with streaming using random sampling.  ... 
doi:10.1145/2987550.2987580 dblp:conf/cloud/HeintzCS16 fatcat:kkcjryk2v5gddpyxh2lipu5oai

Dynamic Asynchronous Anti Poisoning Federated Deep Learning with Blockchain-Based Reputation-Aware Solutions

Zunming Chen, Hongyan Cui, Ensen Wu, Xi Yu
2022 Sensors  
, which enables federated learning to adaptively remove the stragglers with low computing power, bad channel conditions, or anomalous parameters.  ...  The experiment results on three datasets illustrate that our design can reduce the training time by 30% and is robust to the representative poisoning attacks significantly, confirming the applicability  ...  Conflicts of Interest: The authors declare no conflict of interest.  ... 
doi:10.3390/s22020684 pmid:35062645 pmcid:PMC8777936 fatcat:jw72thgz2jgb3jxq2kusy5cq5e

A survey on data analysis on large-Scale wireless networks: online stream processing, trends, and challenges

Dianne S. V. Medeiros, Helio N. Cunha Neto, Martin Andreoni Lopez, Luiz Claudio S. Magalhães, Natalia C. Fernandes, Alex B. Vieira, Edelberto F. Silva, Diogo M. F. Mattos
2020 Journal of Internet Services and Applications  
We present the primary methods for sampling, data collection, and monitoring of wireless networks and we characterize knowledge extraction as a machine learning problem on big data stream processing.  ...  We show the main trends in big data stream processing frameworks.  ...  Acknowledgements We acknowledge colleagues of the Universidade Federal Fluminense (UFF), Samsung R&D Institute, and Universidade Federal de Juíz de Fora (UFJF) for their incentives and suggestions.  ... 
doi:10.1186/s13174-020-00127-2 fatcat:kpx2fyxxkreevga6r22nqdj6rq

Uncertainty Propagation in Data Processing Systems

Ioannis Manousakis, Íñigo Goiri, Ricardo Bianchini, Sandro Rigo, Thu D. Nguyen
2018 Proceedings of the ACM Symposium on Cloud Computing - SoCC '18  
Our evaluation shows that UP-MapReduce propagates uncertainties with high accuracy and, in many cases, low performance overheads.  ...  For example, a social network trend analysis application that combines data sampling with UP can reduce execution time by 2.3x when the user can tolerate a maximum relative error of 5% in the final answer  ...  We observe that UP-MapReduce estimates the means with very low bias, especially when the input relative errors are small (< 3%). We next study the accuracy of the estimated relative errors.  ... 
doi:10.1145/3267809.3267833 dblp:conf/cloud/ManousakisGBRN18 fatcat:ng32pkxpyfdqldatji4u6ku3yi

FLeet: Online Federated Learning via Staleness Awareness and Performance Prediction [article]

Georgios Damaskinos, Rachid Guerraoui, Anne-Marie Kermarrec, Vlad Nitu, Rhicheek Patra, Francois Taiani
2020 arXiv   pre-print
I-Prof can accurately control the impact of learning tasks by improving the prediction accuracy up to 3.6x (computation time) and up to 19x (energy).  ...  FLeet combines the privacy of Standard FL with the precision of online learning thanks to two core components: (i) I-Prof, a new lightweight profiler that predicts and controls the impact of learning tasks  ...  In the presence of stragglers with large delays (comparing to the mean latency), staleness can grow and drive Λ(τ ) close to 0, i.e., almost neglect the gradients of these stragglers.  ... 
arXiv:2006.07273v1 fatcat:qfic5skl6bhxfjq5ses2wedaxq

Multi-tenant mobile offloading systems for real-time computer vision applications

Zhou Fang, Jeng-Hau Lin, Mani B. Srivastava, Rajesh K. Gupta
2019 Proceedings of the 20th International Conference on Distributed Computing and Networking - ICDCN '19  
In pursuit of low latency and high throughput, Clipper adopts caching, adaptive batching, and straggler mitigation techniques.  ...  Time Series Linear Regression Model: In the auto-regressive model for time series prediction problems, the value y n at index n is assumed to be a weighted sum of previous samples in a moving window with  ...  A microservice must come with a method for creating jobs from queries. Listing A.3 gives an example of object detection queries.  ... 
doi:10.1145/3288599.3288634 dblp:conf/icdcn/FangLS019 fatcat:qpib2wkm7jdnfg7k64eh3hwje4

D5.1 Operator Cost Estimation and Workflow Optimisation Technology V1

Project Consortium Members
2020 Zenodo  
Moreover, WP5 interacts with the Synopses Data Engine C [...]  ...  WP5 interacts with WP4 since the Optimizer Component is a fundamental component of the overall INFORE architecture.  ...  and stragglers so as to improve it at runtime.  ... 
doi:10.5281/zenodo.4034108 fatcat:t22h4qqgjfbsporpl4zkf5c2qm
« Previous Showing results 1 — 15 out of 88 results