35 Hits in 6.2 sec

A Survey of Big Data Machine Learning Applications Optimization in Cloud Data Centers and Networks [article]

Sanaa Hamid Mohamed, Taisir E.H. El-Gorashi, Jaafar M.H. Elmirghani
2019 arXiv   pre-print
MapReduce and Hadoop thus introduce innovative, efficient, and accelerated intensive computations and analytics.  ...  Wide ranging efforts were devoted to optimize systems that handle big data in terms of various applications performance metrics and/or infrastructure energy efficiency.  ...  HaSTE improved the resources utilization and the makespan even for mixed workloads. The study in [63] suggested an SLA-aware energy-efficient scheduling scheme based on DVFS for Hadoop with YARN.  ... 
arXiv:1910.00731v1 fatcat:kvi3br4iwzg3bi7fifpgyly7m4

Resource Scheduling in Data-Centric Systems [chapter]

Zujie Ren, Xiaohong Zhang, Weisong Shi
2015 Handbook on Data Centers  
The proposed energy-aware allocation heuristics provision data center resources to client applications in a way that improves energy efficiency of the data center, while delivering the negotiated Quality  ...  To save the energy consumption caused by communication fabric, DENS [26] combines energy efficiency and network awareness to achieve the balance between job performance, QoS requirement, traffic demands  ... 
doi:10.1007/978-1-4939-2092-1_46 fatcat:dxglcjotsrahtbzehtx7c54ulm

Factors affecting cloud data-center efficiency: a scheduling algorithm-based analysis

Shehloo Arif Ahmad, Butt Muheet Ahmed, Zaman Majid
2021 International Journal of Advanced Technology and Engineering Exploration  
Google developed the MapReduce programming paradigm to counter this problem, which served as the foundation for Apache's open-source Hadoop project.  ...  The author proposed Energy-efficient MR Scheduling YARN (EMRSY), a heuristic methodology for creating suboptimal schedules in polynomial time because the formulated problem is NP-hard.  ...  Article Core Idea Strengths Weaknesses [71] Energy Eefficient Scheduling Scheme  Simplified evaluation due to performance estimation module inadequacy [74] Energy-Aware Scheduling Scheme  It reduces  ... 
doi:10.19101/ijatee.2021.874313 fatcat:2bwdcxpac5ccddxdcgpshrrbye

Guest Editors' Introduction: Special Issue on Storage for the Big Data Era

Vlado Stankovski, Radu Prodan
2018 Journal of Grid Computing  
The study entitled "A New Data Layout Scheme for Energy-Efficient MapReduce Processing Tasks" of the authors Xuan T.  ...  Tran, Tien Van Do, Csaba Rotter and Dosam Hwang addresses the problem of energy efficiency when using the Yet Another Resource Negotiator (YARN) in relation to Big Data processing in the Hadoop Distributed  ... 
doi:10.1007/s10723-018-9439-1 fatcat:gqvgzigve5g2hn7pqpfxvhlqsu

HybridMR: A Hierarchical MapReduce Scheduler for Hybrid Data Centers

Bikash Sharma, Timothy Wood, Chita R. Das
2013 2013 IEEE 33rd International Conference on Distributed Computing Systems  
These results indicate that a hybrid data center with an efficient scheduling mechanism can provide a cost-effective solution for hosting both batch and interactive workloads.  ...  In this paper, we make a case for a hybrid data center consisting of native and virtual environments, and propose a 2-phase hierarchical scheduler, called HybridMR, for the effective resource management  ...  ACKNOWLEDGMENT We thank the anonymous reviewers, Adwait Jog, Amin Jadidi, Mahshid Sedghi, Nachiappan Chidambaram Nachiappan, and Onur Kayiran for their valuable comments towards improving this paper.  ... 
doi:10.1109/icdcs.2013.31 dblp:conf/icdcs/SharmaWD13 fatcat:aifrfiqwsbd73ldyzcx54zrreq

Embedded multi-core computing and applications

Che-Lun Hung, Frédéric Magoulès, Meikang Qiu, Robert C. Hsu, Chun-Yuan Lin
2017 Journal of Supercomputing  
In addition to both of hardware and software architectures, another important issue is a suitable parallel algorithm which can efficiently perform on these architectures.  ...  For the multi-core architectures, the trend is to integrate many powerful computing cores into a single computing component. Meanwhile, many processors can be integrated into a single device as GPU.  ...  Cai et al. propose an SLA-aware (Service Level Agreements) energy-efficient scheduling scheme which allocates appropriate amount of resources to MapReduce applications with YARN architecture since YARN  ... 
doi:10.1007/s11227-017-2107-6 fatcat:h6g5rlshxbatblraj6glpzkzky

2021 Index IEEE Transactions on Cloud Computing Vol. 9

2022 IEEE Transactions on Cloud Computing  
The Author Index contains the primary entry for each item, listed under the first author's name.  ...  ., +, TCC July-Sept. 2021 1075-1088 New Scheduling Algorithms for Improving Performance and Resource Utilization in Hadoop YARN Clusters.  ...  ., +, TCC April -June 2021 696-709 Parallel programming New Scheduling Algorithms for Improving Performance and Resource Utilization in Hadoop YARN Clusters.  ... 
doi:10.1109/tcc.2022.3163598 fatcat:mnahwnctzva4lmjw64vjkhwbka

Task Scheduling in Big Data Platforms: A Systematic Literature Review

Mbarka Soualhia, Foutse Khomh, Sofiène Tahar
2017 Journal of Systems and Software  
, and energy efficiency).  ...  An analysis of the scheduling models proposed for Hadoop, Spark, Storm, and Mesos • A research taxonomy for succinct classi cation of the proposed scheduling models • A discussion of some future challenges  ...  research that can be included in a roadmap for research on task and jobs scheduling in Hadoop, Spark, Storm and Mesos frameworks.  ... 
doi:10.1016/j.jss.2017.09.001 fatcat:hr3w4v3uhzekhe56id4j3ic3fq

Infrastructure and Energy Conservation in Big Data Computing: A Survey

Ewa Niewiadomska-Szynkiewicz, Michał P. Karpowicz
2019 Journal of Telecommunications and Information Technology  
This paper addresses the vital problem of energy-efficient high performance distributed and parallel computing. An overview of recent technologies for Big Data processing is presented.  ...  Reduction in energy consumption is another challenge arising in connection with the development of efficient HPC infrastructures.  ...  Energy-Aware Infrastructure for HPC Computing Energy Conservation and HPC Computing Nowadays, energy efficiency in all sectors, including HPC infrastructure, is a key part of European energy policies  ... 
doi:10.26636/jtit.2019.132419 fatcat:oeokamnhxbdmbkoytyrpqgilfy

SHIYF: A Secured and High-Integrity YARN Framework

Deng, Liu, Wang, Li
2019 Electronics  
In this paper, we propose a secure and high-integrity YARN framework (SHIYF), which establishes a close relationship between speculative execution and the security of Yet Another Resource Negotiator (YARN  ...  The prototype of SHIYF is implemented based on Hadoop 2.8.0.  ...  Acknowledgments: The authors would like to thank the editors and the reviewers for their valuable comments that helped to improve the quality of this paper.  ... 
doi:10.3390/electronics8050548 fatcat:avzpoo6eejdmxm6ottutvs4b6m

A Survey and Taxonomy on Energy-Aware Data Management Strategies in Cloud Environment

Xindong You, Xueqiang Lv, Zhikai Zhao, Junmei Han, Xueping Ren
2020 IEEE Access  
This survey and the taxonomy of the energy-aware data management strategies demonstrate the potential for reducing the energy consumption at the data management level of a cloud storage system, which will  ...  compress more space for energy reduction and finally achieve energy proportionality.  ...  Due to the new resource management and allocation framework (YARN) in the HDFS system, the default data layout schemes are not energy efficient, literature [121] proposed a new data layout scheme, which  ... 
doi:10.1109/access.2020.2992748 fatcat:ad5nhkifjnbonnrikq2ziqyc64

An Energy Efficiency Optimization and Control Model for Hadoop Clusters

Haifeng Wang, Yunpeng Cao
2019 IEEE Access  
We propose a control model based on model prediction control (MPC) for improving energy efficiency of Hadoop cluster while satisfying performance goal.  ...  Therefore improving energy efficiency and minimizing energy consumption when executing each MapReduce job is a critical concern for data centers.  ...  This scheme considers both the communication and computing [12] . Shao et al. proposed a novel energy-aware MapReduce resource scheduling model.  ... 
doi:10.1109/access.2019.2907018 fatcat:bv6jai2yojcbvijztadeotpmcy

Energy-efficient mapping of large-scale workflows under deadline constraints in big data computing systems

Tong Shu, Chase Q. Wu
2017 Future generations computer systems  
The performance superiority of the proposed solution in terms of dynamic energy saving and deadline missing rate is illustrated by extensive simulation results in Hadoop/YARN in comparison with existing  ...  Since this problem is strongly NP-hard, we design a fully polynomialtime approximation scheme (FPTAS) for a special case with a pipeline-structured workflow on a homogeneous cluster and a heuristic for  ...  /malleable job scheduling for energy efficiency.  ... 
doi:10.1016/j.future.2017.07.050 fatcat:rle7hz5m3jbqtftap5so3nnq4m

Host managed contention avoidance storage solutions for Big Data

Pratik Mishra, Arun K. Somani
2017 Journal of Big Data  
BID schemes as a whole is aimed to avoid contentions for disk based storage I/Os following system constraints without compromising SLAs.  ...  BID schemes as a whole is aimed to avoid contentions for disk based storage I/Os following system constraints without compromising SLAs.  ...  this article has been published in the Proceedings of 24th IEEE Modeling, Analysis and Simulation of Computer and Telecommunication Systems "MASCOTS" 2016, London, UK as "Bulk I/O Storage Management for  ... 
doi:10.1186/s40537-017-0080-9 fatcat:t2ni3rvbkvdapk2to7skgp445i

TMaR: a two-stage MapReduce scheduler for heterogeneous environments

Neda Maleki, Hamid Reza Faragardi, Amir Masoud Rahmani, Mauro Conti, Jay Lofstead
2020 Human-Centric Computing and Information Sciences  
In this paper, we propose a two-stage Map and Reduce task scheduler for heterogeneous environments, called TMaR.  ...  The simulation results demonstrate that TMaR outperforms Hadoop-stock and Hadoop-A in terms of makespan and network traffic and achieves by an average of 29%, 36%, and 14% performance using Wordcount,  ...  Hadoop Scheduling User-level Capacity Scheduler Fair Scheduler Job-level FIFO Scheduler Priority Scheduler Task-level Map Tasks Scheduler Local-aware Map Tasks Scheduler Reduce  ... 
doi:10.1186/s13673-020-00247-5 fatcat:d7dg5mk76bfn3agor377dv2oui
« Previous Showing results 1 — 15 out of 35 results