Filters








2,608 Hits in 3.3 sec

Big Data Workflows: Locality-Aware Orchestration Using Software Containers

Andrei-Alin Corodescu, Nikolay Nikolov, Akif Quddus Khan, Ahmet Soylu, Mihhail Matskin, Amir H. Payberah, Dumitru Roman
2021 Sensors  
This article proposes a novel architecture and a proof-of-concept implementation for software container-centric big data workflow orchestration that puts data locality at the forefront.  ...  Existing big data processing solutions provide limited support for handling data locality and are inefficient in processing small and frequent events specific to the edge environments.  ...  Locality-Aware Workflow Orchestration for Big Data.  ... 
doi:10.3390/s21248212 pmid:34960302 pmcid:PMC8706844 fatcat:3nc2j4pvdfdynn573zzq7ympca

Toward Next Generation Interfaces for Exploiting Workflows

Julian Kunkel
2020 Zenodo  
Optimizing data placement strategy in ESDM/workflow scheduler Utilizing hints for IME to pin data to cache Storing data locally between depending tasks (using modified Slurm) Optimizing initial data allocation  ...  : Advantage for Data Placement Scenario Consider three file systems: local, scratch, and work Local is a compute-node local storage system Data can be stored on any of these storage systems Scheduler  ... 
doi:10.5281/zenodo.3985173 fatcat:g5lgrasivff5jmy2jl3ppgqexa

Potential of I/O-Aware Workflows in Climate and Weather

Julian Kunkel, Luciana Pedro, Bryan Lawrence, Glenn Greed, David Matthews, Hua Huang
2020 Zenodo  
orchestration containers with all software data management plan with data lifecycle time constraints and budget Vision Design Summary Vision: Exploit Workflow Knowledge Enhance workflow description  ...  Ensure proper execution Provoking: Big data technology is ahead of HPC in such an agenda Julian M.  ... 
doi:10.5281/zenodo.3985161 fatcat:3rvdiuykxjbjvfdnced3ufnkqm

A note on new trends in data-aware scheduling and resource provisioning in modern HPC systems

Jie Tao, Joanna Kolodziej, Rajiv Ranjan, Prem Prakash Jayaraman, Rajkumar Buyya
2015 Future generations computer systems  
ParSA introduces application-aware data distribution in HDFS and locality-aware task scheduling to accelerate wellknown analysis methods for climate data.  ...  (RS) big data, challenges, current techniques, and existing works for processing big data.  ... 
doi:10.1016/j.future.2015.04.016 fatcat:z2254f74bjdhtgoxus44ogqkyq

Location, Location, Location: Data-Intensive Distributed Computing in the Cloud

Michael Luckeneder, Adam Barker
2013 2013 IEEE 5th International Conference on Cloud Computing Technology and Science  
When orchestrating highly distributed and dataintensive Web service workflows the geographical placement of the orchestration engine can greatly affect the overall performance of a workflow.  ...  Our experimental results show that our proposed optimisation strategy, depending on the particular workflow, can speed up execution time on average by 82.25% compared to local execution.  ...  Data-Aware and Location-Aware Scheduling Amazon have recently added Latency-Based Routing (LBR) [1] to the Route 53 service.  ... 
doi:10.1109/cloudcom.2013.91 dblp:conf/cloudcom/LuckenederB13 fatcat:2c4qi52ivnet7e2ybpiokjqmu4

Location, Location, Location: Data-Intensive Distributed Computing in the Cloud [article]

Michael Luckeneder, Adam Barker
2014 arXiv   pre-print
When orchestrating highly distributed and data-intensive Web service workflows the geographical placement of the orchestration engine can greatly affect the overall performance of a workflow.  ...  Our experimental results show that our proposed optimisation strategy, depending on the particular workflow, can speed up execution time on average by 82.25% compared to local execution.  ...  Data-Aware and Location-Aware Scheduling Amazon have recently added Latency-Based Routing (LBR) [1] to the Route 53 service.  ... 
arXiv:1309.6452v2 fatcat:zl7fqjznf5fznoxm6xpfh3wmii

Data-Centric IO: Potential for Climate/Weather

Julian Kunkel, Luciana Pedro, Bryan Lawrence, Glenn Greed, David Matthews, Hua Huang
2020 Zenodo  
orchestration containers with all software data management plan with data lifecycle time constraints and budget Vision Design Summary Vision: Exploit Workflow Knowledge Enhance workflow description  ...  Utilizing hints for IME to pin data to cache Storing data locally between depending tasks (using modified Slurm) Optimizing initial data allocation (e.g., alternating storage between cycles) These changes  ... 
doi:10.5281/zenodo.3985143 fatcat:tn3diflapzfszceqryulvwk2ta

Process Automation in an IoT–Fog–Cloud Ecosystem: A Survey and Taxonomy

Hossein Chegini, Ranesh Kumar Naha, Aniket Mahanti, Parimala Thulasiraman
2021 IoT  
The big data and heterogeneity challenges also motivated us to design other automatic components for Fog resiliency, which we address as the third challenge in the ecosystem.  ...  Being challenged of processing all IoT big data on Cloud facilities, there is not enough study on automating components to deal with the big data and real-time tasks in the IoT–Fog–Cloud ecosystem.  ...  [41] shows a context-aware CEP model(COLLECT) in a SOA for handling big data in the healthcare domain.  ... 
doi:10.3390/iot2010006 fatcat:nlb2cvotzna4dfctb42eozketa

Collaborative interactive visualization: exploratory concept

Marielle Mokhtari, Valérie Lavigne, Frédéric Drolet, Barbara D. Broome, Timothy P. Hanratty, David L. Hall, James Llinas
2015 Next-Generation Analyst III  
Dealing with an ever increasing amount of data is a challenge that military intelligence analysts or team of analysts face day to day.  ...  Interactive visualization strengthens collaboration because this approach is conducive to incrementally building a mental assessment of the data meaning.  ...  Above this database resides the Big Data Manager responsible for transparent data transmission between the UDS and the rest of the S3 system.  ... 
doi:10.1117/12.2184179 fatcat:nfywufrbc5gwbgpx7dezoahjoe

Eliminating the middleman

Adam Barker, Jon B. Weissman, Jano van Hemert
2008 Proceedings of the 17th international symposium on High performance distributed computing - HPDC '08  
When orchestrating data-centric workflows, centralised servers common to standard workflow systems can become a bottleneck to performance.  ...  Efficiently executing large-scale, data-intensive workflows such as Montage must take into account the volume and pattern of communication.  ...  The intermediate data can be 3 times the size of the input data. And a big problem, e.g. an all-sky mosaic can result in 2-8 TB of data. Such a problem might be run daily.  ... 
doi:10.1145/1383422.1383430 dblp:conf/hpdc/BarkerWH08 fatcat:flwpqkkdyng6xa2xchimsp4aby

Software-Defined Workflows for Distributed Interoperable Closed-Loop Neuromodulation Control Systems

Pradeeban Kathiravelu, Parisa Sarikhani, Ping Gu, Babak Mahmoudi
2021 IEEE Access  
In this paper, we present NEXUS, a workflow orchestration framework for distributed analytics systems.  ...  The centralized NEXUS orchestrator facilitates dynamically composing and managing scientific workflows from the services and existing workflows, with minimal restrictions.  ...  Acknowledgment: We are thankful to Voxility for providing a deployment infrastructure with assistance.  ... 
doi:10.1109/access.2021.3113892 pmid:34631327 pmcid:PMC8500400 fatcat:zhluwxgvm5fnje6vudzr2hz3hm

FLOOD-PREPARED: A NOWCASTING SYSTEM FOR REAL-TIME IMPACT ADAPTION TO SURFACE WATER FLOODING IN CITIES

S. L. Barr, S. L. Barr, S. L. Barr, S. Johnson, X. Ming, M. Peppa, N. Dong, Z. Wen, C. Robson, L. Smith, P. James, D. Wilkinson (+5 others)
2020 ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences  
These models are linked and orchestrated within into a Big Data workflow that allows events to be simulated using emerging rainfall data recorded by a short range weather radar.  ...  Modern sensor networks offer enormous potential for the real-time monitoring of urban systems and potentially allow improved situational awareness of impeding hazards and their impacts such as flooding  ...  The Flood-PREPARED workflow developed in this paper shows the potential of coupling sensor network data feeds with advanced analytics and models using Big Data workflow orchestration tools such as Docker  ... 
doi:10.5194/isprs-annals-vi-4-w2-2020-9-2020 doaj:a85ed23469e24d5fbf3363f09049602c fatcat:p53nv3dmzjgb5jsyrki7zvhpr4

Computational Viability of Fog Methodologies in IoT Enabled Smart City Architectures-A Smart Grid Case Study

Md. Muzakkir Hussain, Mohammad Saad Alam, M.M. Sufyan Beg
2018 EAI Endorsed Transactions on Smart Cities  
Finally, an overview of the core orchestration issues, challenges, and future research directions are presented for FC enabled SCs.  ...  The geo-distributed clusters of IoT -objects‖ produce galactic volume of data that exacerbates the need to make a paradigm shift from centralized data center based processing to a hybrid model that supports  ...  In the evaluation phase, Big-Data-Driven analytics (BD 2 A) and Optimization Algorithms are prime avenues that need to be explored to improve orchestration quality and accelerate optimization for problem  ... 
doi:10.4108/eai.12-2-2018.154104 fatcat:bodmgugpy5flfkdoygrq2moqc4

Fog Orchestration for Internet of Things Services

Zhenyu Wen, Renyu Yang, Peter Garraghan, Tao Lin, Jie Xu, Michael Rovatsos
2017 IEEE Internet Computing  
This article provides an overview of the core issues, challenges and future research directions in Fog-enabled orchestration for IoT services.  ...  The orchestration of such applications can be leveraged to alleviate the difficulties of maintenance and enhance data security and system reliability.  ...  For this, we randomly select four types of orchestration graphs with 50, 100, 150, and 200 workflow nodes, respectively.  ... 
doi:10.1109/mic.2017.36 fatcat:5rvkwbrvj5cb5bamhlnztrwqy4

Proactivecloud Resources Management At Theedge Forefficientreal-Time Big Data Processing

Harald Schöning
2017 Zenodo  
ProactiveCloud Resources Management at theEdge forefficientReal-time Big Data Processing.  ...   network virtualization  dynamic monitoring in real-time processing architectures for Big Data  situation-aware and context-driven adaptation recommender systems  real-time mobile stream processing  ...  Streaming Analytics for efficient real-time big data processing. © 2017 Software AG.  ... 
doi:10.5281/zenodo.1138578 fatcat:ll3vehr6crhb5i2yjlkhzaca5i
« Previous Showing results 1 — 15 out of 2,608 results