Filters








1,471 Hits in 2.4 sec

Towards a unified storage and ingestion architecture for stream processing

Ovidiu-Cristian Marcu, Alexandru Costan, Gabriel Antoniu, Maria S. Perez-Hernandez, Radu Tudoran, Stefano Bortoli, Bogdan Nicolae
2017 2017 IEEE International Conference on Big Data (Big Data)  
In this position paper, we argue for a unified ingestion and storage architecture for streaming data that addresses the aforementioned challenge.  ...  Current streaming-oriented runtimes and middlewares are not flexible enough to deal with this trend, as they address ingestion (collection and pre-processing of data streams) and persistent storage (archival  ...  To this end, we propose a series of design principles and architecture towards a unified stream ingestion and storage solution that addresses these limitations and missing features.  ... 
doi:10.1109/bigdata.2017.8258196 dblp:conf/bigdataconf/MarcuCAPTBN17 fatcat:w5a2fruranfflbwchhfuerqrpm

BigDataGrapes D2.3 - BigDataGrapes Software Stack Design

Panagis Katsivelis
2021 Zenodo  
It is expected that, as the use cases are refined and executed, the overall specification and the corresponding architecture will evolve towards the new requirements that arise.  ...  Deliverable D2.3 is the first submitted iteration of a living document that will describe the components and architecture of the BigDataGrapes software stack, providing design principles and technical  ...  Additionally, Kafka connects to external systems (for data import/export) via Kafka Connect and provides Kafka Streams, a Java stream processing library.  ... 
doi:10.5281/zenodo.4546026 fatcat:a35e4prb5ffqjmsammsac4e52q

Device Data Ingestion for Industrial Big Data Platforms with a Case Study

Cun Ji, Qingshi Shao, Jiao Sun, Shijun Liu, Li Pan, Lei Wu, Chenglei Yang
2016 Sensors  
Despite having played a significant role in the Industry 4.0 era, the Internet of Things is currently faced with the challenge of how to ingest large-scale heterogeneous and multi-type  ...  Dolui et al. discussed two types of sensor data processing architectures, namely, on-device and on-server data processing architectures [14] .  ...  Thus, we created the IBDP with a heterogeneous device data ingestion model for data from multiple sources. Using this model, we can ingest various device data and store them in a unified format.  ... 
doi:10.3390/s16030279 pmid:26927121 pmcid:PMC4813854 fatcat:zxg6gr2abra4pj5smatth6ikuy

BigDataGrapes D2.3 - BigDataGrapes Software Stack Design

Pythagoras Karampiperis, Antonis Koukourikos, Raffaele Perego, Franco Maria Nardini, Nicola Tonellotto, Milena Yankova, Panagiotis Zervas, Mihalis Papakonstadinou
2019 Zenodo  
It is expected that, as the use cases are refined and executed, the overall specification and the corresponding architecture will evolve towards the new requirements that arise.  ...  Deliverable D2.3 is the first submitted iteration of a living document that will describe the components and architecture of the BigDataGrapes software stack, providing design principles and technical  ...  Additionally, Kafka connects to external systems (for data import/export) via Kafka Connect and provides Kafka Streams, a Java stream processing library.  ... 
doi:10.5281/zenodo.3269261 fatcat:7czrc4pqb5hn5kuwh34ytuzmjq

A Unified Frame Work to Integrate Hadoop and IOT to Resolve the Issues of Storage, Processing with Leveraging Capacity of Analytics

Gudapati Syam Prasad, P Rajesh, Sk Wasim Akram
2018 International Journal of Engineering & Technology  
The outcome of the proposed idea is integration of Hadoop and IOT platforms with a unified frame work which accommodates the integration of Hadoop and IOT, storage provisions to handle bulk data, processing  ...  The proposed idea gives the usage of Hadoop platform to store the data and from that data performing analytics for the sake of better utilization of IOT communications.  ...  A big data problem is one that when data itself becomes a part of the problem in case of storage and processing.  ... 
doi:10.14419/ijet.v7i2.32.15390 fatcat:qpdsxtgbcrhvjbsnd2jzwjf4ey

Learning analytics for higher education: proposal of big data ingestion architecture

Meseret Yihun Amare, Stanislava Simonova, T. Kliestik
2021 SHS Web of Conferences  
The data ingestion layer is responsible for collecting data and transform for analysis.  ...  The lack of tailored big-data ingestion architectures for academics led to several implementation challenges.  ...  The architecture comprises of a data pipeline for automation of ingestion process, data ingestion techniques and storage for analytical data that could enhance the data ingestion process by making it seamless  ... 
doi:10.1051/shsconf/20219202002 fatcat:lgvhvqavqbaonjs4h2az3pxqaa

Scalable Architecture for Personalized Healthcare Service Recommendation Using Big Data Lake [chapter]

Sarathkumar Rangarajan, Huai Liu, Hua Wang, Chuan-Long Wang
2018 Lecture Notes in Business Information Processing  
This study proposes a novel data lake architecture to reduce the data ingestion time and improve the precision of healthcare analytics.  ...  The data lake architecture uses the Hadoop Distributed File System (HDFS) to provide the storage for both structured and unstructured data.  ...  Due to the pre-defined data schema for data warehouse architecture, it is impossible to store the data of different types in a unified storage location.  ... 
doi:10.1007/978-3-319-76587-7_5 fatcat:xz3srv6vibfzdh6c6xpb5a5qze

Cloud architecture for plant phenotyping research

Olivier Debauche, Sidi Ahmed Mahmoudi, Nicolas De Cock, Saïd Mahmoudi, Pierre Manneback, Frédéric Lebeau
2020 Concurrency and Computation  
However, the capacity of ingesting and requesting of the database is essential and can constitute a bottleneck for the architecture, in particular, for in terms of availability and response time of data  ...  BACKGROUND LAMBDA ARCHITECTURE The literature shows that a lambda architecture 29 is formed by three layers: (1) the batch layer that ingests and stores immutable large datasets and provides batch views  ...  The authors would also like to thank Mr Adriano Guttadauria for his technical support and for setting up all the electronic and informatic systems necessary for carrying out this cloud architecture.  ... 
doi:10.1002/cpe.5661 fatcat:d56lx4zndndbdmvlnajnoikdpa

I-BiDaaS: Industrial-Driven Big Data as a Self-Service Solution

Giorgos Vasiliadis, Dusan Jakovetic, Ilias Spais, Sotiris Ioannidis
2018 Zenodo  
while coping with the rate of data asset growth, and facilitates cross-domain data-ow towards a thriving data-driven EU economy.  ...  In this paper, we present I-BiDaaS, a European Union Horizon 2020 research and innovation project that proposes a self-service solution for Big Data analytics.  ...  Acknowledgments The authors thank all project partners for all the fruitful discussions on several aspects of the architecture.  ... 
doi:10.5281/zenodo.3355416 fatcat:ssxpe5cmxveh3k7ev34y2gz45q

Big Data and Personalisation for Non-Intrusive Smart Home Automation

Suriya Priya R. Asaithambi, Sitalakshmi Venkatraman, Ramanathan Venkatraman
2021 Big Data and Cognitive Computing  
This paper proposes a non-intrusive approach of integrating and collecting data from open standard IoT devices for personalised smart home automation using big data analytics and machine learning.  ...  We demonstrate the implementation of our proposed novel technology instantiation approach for achieving non-intrusive IoT based big data analytics with a use case of a smart home environment.  ...  Acknowledgments: This work contains processed information from static and dynamic datasets from a smart home setting with smart hub running on a docker container with Raspberry Pi, OpenHab and Apache NiFi  ... 
doi:10.3390/bdcc5010006 fatcat:6dv4b3dvbrdkjei2vvccr5pstq

MOBDA: Microservice-Oriented Big Data Architecture for Smart City Transport Systems

Suriya Priya R. Asaithambi, Ramanathan Venkatraman, Sitalakshmi Venkatraman
2020 Big Data and Cognitive Computing  
A novel hybrid architecture is proposed to combine stream processing and batch processing of big data for a smart computation of microservice-oriented transportation metrics that can serve the different  ...  This paper proposes a Microservice-Oriented Big Data Architecture (MOBDA) incorporating data processing techniques, such as predictive modelling for achieving smart transportation and analytics microservices  ...  The authors also wish to acknowledge the reviewers and editor for their invaluable inputs and support rendered for this paper. Conflicts of Interest: The authors declare no conflict of interest.  ... 
doi:10.3390/bdcc4030017 fatcat:7vnhe5u3affznn4cwxfxjjc4qy

A Distributed Stream Processing Middleware Framework for Real-Time Analysis of Heterogeneous Data on Big Data Platform: Case of Environmental Monitoring

Adeyinka Akanbi, Muthoni Masinde
2020 Sensors  
Herein, as a solution, a distributed stream processing middleware framework for real-time analysis of heterogeneous environmental monitoring and management data is presented and tested on a cluster using  ...  The stream processing engine executes the predictive numerical models and algorithms represented in event processing (EP) languages for real-time analysis of the data streams.  ...  Acknowledgments: The authors would like to thank all the anonymous reviewers and the editor for their valuable input and comments. Conflicts of Interest: The authors declare no conflict of interest.  ... 
doi:10.3390/s20113166 pmid:32503145 pmcid:PMC7308861 fatcat:cxk3ipbluvbhtaeibssamyg344

AI based HealthCare Platform for Real Time, Predictive and Prescriptive Analytics using Reactive Programming

Jagreet Kaur, Dr. Kulwinder Singh Mann
2018 Journal of Physics, Conference Series  
AI in Healthcare needed to bring real, actionable insights and Individualized insights in real time for patients and Doctors to support treatment decisions., We need a Patient Centred Platform for integrating  ...  This paper proposes a generic architecture for enabling AI based healthcare analytics Platform by using open sources Technologies Apache beam, Apache Flink Apache Spark, Apache NiFi, Kafka, Tachyon, Gluster  ...  The proposed architecture can support for Artificial intelligence bssed healthcare analytics by providing batch and stream computing, extendable storage solution and query management.  ... 
doi:10.1088/1742-6596/933/1/012010 fatcat:bfcdkoaqkjec5j4m43pi737lhu

D2.2 CONCEPTUAL MODEL & REFERENCE ARCHITECTURE

Kiourtis Thanos, Mavrogiorgou Argyro, Manias George, Kyriazis Dimosthenis, La Rocca Giuseppe, Biran Ofer, Feder Oshrit, Sanguino Gonzalez Maria Angeles, Montero Gomez Jorge, Pariente Lobo Tomas, Duzha Armend, Achilleopoulos Nikos (+18 others)
2020 Zenodo  
A number of scenarios developed for the Use Cases of PolicyCLOUD are also included in the document, in order to serve as end-to-end examples for the demonstration of the data ingest flow and data exploitation  ...  and for the analysis of the processing and data transformations along the complete data path.  ...  Due to the scalable transactional processing provided by the LXS datastore and its additional interface that allows directly accessing its storage layer, it can support data ingestion coming from streams  ... 
doi:10.5281/zenodo.4081542 fatcat:rykyoljatragxfyw7gi4fs3i6a

A software reference architecture for semantic-aware Big Data systems

Sergi Nadal, Victor Herrero, Oscar Romero, Alberto Abelló, Xavier Franch, Stijn Vansummeren, Danilo Valerio
2017 Information and Software Technology  
Context: Big Data systems are a class of software systems that ingest, store, process and serve massive amounts of heterogeneous data, from multiple sources.  ...  Objective: In this paper, we follow software engineering principles to refine the λ-architecture, a reference model for Big Data systems, and use it as seed to create Bolster, a software reference architecture  ...  Acknowledgements 1063 We thank Gerhard Engelbrecht for his assistance in setting up the validation 1064 process, and Silverio Martínez for his comments and insights that helped to 1065 improve this  ... 
doi:10.1016/j.infsof.2017.06.001 fatcat:iizyd3uupvhftent27lk45d46m
« Previous Showing results 1 — 15 out of 1,471 results