165,211 Hits in 4.9 sec

A Survey of Distributed Data Stream Processing Frameworks

Haruna Isah, Tariq Abughofa, Sazia Mahfuz, Dharmitha Ajerla, Farhana Zulkernine, Shahzad Khan
2019 IEEE Access  
With a view to addressing this issue, in this paper we present a taxonomy, a comparative study of distributed data stream processing and analytics frameworks, and a critical review of representative open  ...  INDEX TERMS Dataflow architectures, data stream architectures, distributed processing systems comparison, survey, taxonomy.  ...  We present a literature survey and a study of the components of data stream processing systems (DSPS).  ... 
doi:10.1109/access.2019.2946884 fatcat:lu6oknfpkraybmtuqxismmlqda

Challenges and Solutions for Processing Real-Time Big Data Stream: a systematic literature review

Erum Mehmood, Tayyaba Anees
2020 IEEE Access  
and can serve as a guide for the implementation of real-time stream processing framework for all shapes of data streams.  ...  The capability of organizing big data in efficient manner to reach a business decision empowers data warehousing in terms of real-time stream processing.  ...  Processing of geographically distributed data has been surveyed in a study [19] , without shifting whole datasets to a single location.  ... 
doi:10.1109/access.2020.3005268 fatcat:b2xlblvarrgenctrnpqrpvijau

A survey on bandwidth-aware geo-distributed frameworks for big-data analytics

Mohammed Bergui, Said Najah, Nikola S. Nikolov
2021 Journal of Big Data  
In this article, we discuss challenges and survey the latest geo-distributed big-data analytics frameworks and schedulers (based on MapReduce and Spark) with WAN-bandwidth awareness.  ...  AbstractIn the era of global-scale services, organisations produce huge volumes of data, often distributed across multiple data centres, separated by vast geographical distances.  ...  Thus, we survey geo-distributed big data processing frameworks with WAN-bandwidth awareness and provide pros and cons for most of the frameworks.  ... 
doi:10.1186/s40537-021-00427-9 fatcat:u2jx7x6hkfc47kn2iqpkcquhi4

Incremental Techniques for Large-Scale Dynamic Query Processing

Iman Elghandour, Ahmet Kara, Dan Olteanu, Stijn Vansummeren
2018 Proceedings of the 27th ACM International Conference on Information and Knowledge Management - CIKM '18  
In this tutorial, we briefly discuss legacy approaches for incremental query processing, and then give an overview of the new challenges introduced due to processing big data streams.  ...  Moreover, many of these algorithms are now leveraging distributed streaming platforms such as Spark Streaming and Flink.  ...  Part III: Dynamic Query Processing in Big Data Frameworks Incremental processing of queries has been studied for queries executed by MapReduce [5, 21, 23] and by other distributed streaming platforms  ... 
doi:10.1145/3269206.3274271 dblp:conf/cikm/Elghandour0OV18 fatcat:rwdtjicsibghzgvdajh5puobjm

Incremental Techniques for Large-Scale Dynamic Query Processing [article]

Iman Elghandour and Ahmet Kara and Dan Olteanu and Stijn Vansummeren
2019 arXiv   pre-print
Moreover, many of these algorithms are now leveraging distributed streaming platforms such as Spark Streaming and Flink.  ...  In this tutorial, we briefly discuss legacy approaches for incremental query processing, and then give an overview of the new challenges introduced due to processing big data streams.  ...  Part III: Dynamic Query Processing in Big Data Frameworks Incremental processing of queries has been studied for queries executed by MapReduce [5, 23, 21] and by other distributed streaming platforms  ... 
arXiv:1902.00585v1 fatcat:4dtwnxhiqjfqdbb63tartjo4fq

An experimental survey on big data frameworks

Wissem Inoubli, Sabeur Aridhi, Haithem Mezni, Mondher Maddouri, Engelbert Mephu Nguifo
2018 Future generations computer systems  
Recently proposed frameworks for Big Data applications help to store, analyze and process the data. In this paper, we discuss the challenges of Big Data and we survey existing Big Data frameworks.  ...  This survey is concluded with a presentation of best practices related to the use of the studied frameworks in several application domains such as machine learning, graph processing and real-world applications  ...  In our work, we compare the studied frameworks in the case of both batch processing and stream processing which is not studied in existing surveys.  ... 
doi:10.1016/j.future.2018.04.032 fatcat:dxl42yu54retblcgttysadacqu

Reference Architecture and Classification of Technologies, Products and Services for Big Data Systems

Pekka Pääkkönen, Daniel Pakkala
2015 Big Data Research  
Additionally, we acknowledge developers and publishers of the reviewed big data use cases, whose publications were used as empirical material in this study.  ...  Acknowledgements The authors acknowledge Markus Meier for design of the reference architecture for big data systems, which inspired the authors for replication and differentiation in this work.  ...  Qian et al. proposed TimeStream, which is a distributed system for processing of low latency streaming data on commodity machines [39] .  ... 
doi:10.1016/j.bdr.2015.01.001 fatcat:diyyx34wqzewbcwshq2yezsxm4

A Comparative Analysis of Big Data Frameworks: An Adoption Perspective

Madiha Khalid, Muhammad Murtaza Yousaf
2021 Applied Sciences  
These limitations have led to the development of new technologies to process and store very large datasets. As a result, several execution frameworks emerged for big data processing.  ...  Hadoop MapReduce, the pioneering framework, set the ground for forthcoming frameworks that improve the processing and development of large-scale data in many ways.  ...  Data Availability Statement: Not applicable. Conflicts of Interest: The authors declare no conflict of interest.  ... 
doi:10.3390/app112211033 fatcat:mfh3thwe5ngdnolkhc264rkbdi

Performance Assay of Big IoT Data Analytics Framework

2019 International journal of recent technology and engineering  
This paper, discusses the challenges and issues faced by distributed stream analytics frameworks at the data processing level and tries to recommend a possible a Scalable Framework to adapt with the volume  ...  and velocity of Big IoT Stream Data.  ...  There is a crucial need of a data analytics framework that can process stream data of millions of events in seconds with low latency and high throughput.  ... 
doi:10.35940/ijrte.d7383.118419 fatcat:r65sqxqy6bedpmsvt4sfdqg2ii

A Systematic Study for Big Data Stream Processing Frameworks

Ali Yazici, Ziya Karakaya, Mohammed Alayyoub
2016 Journal on Advances in Theoretical and Applied Informatics  
The choice of the most effective stream processing framework (SPF) for Big Data has been an important issue among the researchers and practioners.  ...  Each of the SPFs has different cutting edge technologies in their steps of processing the data in motion that gives them a better advantage over the others.  ...  The paper also presents a brief summary of comparison between Spark streaming and Storm. In [10] , a survey on modern approaches for Big Data stream processing is presented.  ... 
doi:10.26729/jadi.v2i2.1914 fatcat:glh5raazjbgpzju7iytqk7nxgq

Integrating Remote Sensing and 2D Hydraulic Modelling for Meso-habitat Modelling in the Aurino, a Gravel-bed Alpine River

2022 Publications of the Institute of Geophysics Polish Academy of Sciences Geophysical Data Bases Processing and Instrumentation  
We present an example of the application of a methodological framework for meso-scale habitat suitability modelling, on a reach of the gravel-bed Aurino River (NE Italy).  ...  allowing to survey longer river stretches.  ...  Comoglio (2014), Habitat modeling in high-gradient streams: the mesoscale approach and application, Ecol. Appl. 24, 4, 844-861, DOI: 10.1890/11-2066.1.  ... 
doi:10.25171/instgeoph_pas_publs-2022-007 fatcat:ewekzlue5ra7bahslrekxbs4tu

SIROM3 -- A Scalable Intelligent Roaming Multi-modal Multi-sensor Framework

Jiaxing Zhang, Hanjiao Qiu, Salar Shahini Shamsabadi, Ralf Birken, Gunar Schirner
2014 2014 IEEE 38th Annual Computer Software and Applications Conference  
A Heterogeneous Stream File-system Overlay (HSFO) and a flexible plugin system (PLEX) are embedded in SIROM 3 to simplify big data storage, processing, and correlation.  ...  SIROM 3 offers a scalable and expandable framework through orthogonally abstracting software / hardware structures in a layered Run-Time Environment (RTE), which facilities sensor fusion, distributed computing  ...  by a single processing element and thus requiring a distributed system solution [20] .  ... 
doi:10.1109/compsac.2014.97 dblp:conf/compsac/ZhangQSBS14 fatcat:jkrduvvuorhsphbcrcfqkxivxe

Survey of Real-time Processing Technologies of IoT Data Streams

Keiichi Yasumoto, Hirozumi Yamaguchi, Hiroshi Shigeno
2016 Journal of Information Processing  
Then we propose a new framework for IoT data streams called the Information Flow of Things (IFoT) that processes, analyzes, and curates massive IoT streams in real-time based on distributed processing  ...  In this paper, we survey the emerging technologies toward the real-time utilization of IoT data streams in terms of networking, processing, and content curation and clarify the open issues.  ...  In this section, we propose the Information Flow of Things (IFoT), a new framework for processing, analyzing, and curating IoT data streams in real-time and in a scalable manner based on distributed processing  ... 
doi:10.2197/ipsjjip.24.195 fatcat:g6i6zwzaqfbwraui7bochm6ksi

Astrophysics and Big Data: Challenges, Methods, and Tools

Mauro Garofalo, Alessio Botta, Giorgio Ventre
2016 Proceedings of the International Astronomical Union  
Therefore, new and advanced processing solutions will be needed to process this huge amount of data.  ...  Nowadays there is no field research which is not flooded with data. Among the sciences, astrophysics has always been driven by the analysis of massive amounts of data.  ...  For both frameworks, Fig. 1 shows a schematic representation of the architecture and the steps involved in the analytics process.  ... 
doi:10.1017/s1743921316012813 fatcat:ljwe6wqv4rgrzb2nqrolh6tu7q

A comprehensive survey of anomaly detection techniques for high dimensional big data

Srikanth Thudumu, Philip Branch, Jiong Jin, Jugdutt (Jack) Singh
2020 Journal of Big Data  
Acknowledgements The article processing charge is funded by Swinburne University of Technology, Australia.  ...  Apache Spark(Spark) [136] is another distributed framework based on MapReduce for processing large volumes of data on a distributed system, however, has a feature called in-memory computation [159,  ...  For both stream processing and batch processing, Apache Flink [137] another opensource framework was proposed combining the scalability and programming flexibility of other distributed paradigms such  ... 
doi:10.1186/s40537-020-00320-x fatcat:nrx7fnuzbvf65edoisv65by4s4
« Previous Showing results 1 — 15 out of 165,211 results