5,368 Hits in 5.0 sec

Evaluating task scheduling in hadoop-based cloud systems

Shengyuan Liu, Jungang Xu, Zongzhen Liu, Xu Liu
2013 2013 IEEE International Conference on Big Data  
In this paper, we use a recently-released Cloud benchmarks suite---CloudRank-D to quantitatively evaluate five different Hadoop task schedulers, including FIFO, capacity, naïve fair sharing, fair sharing  ...  Nowadays, private clouds are widely used for resource sharing. Hadoop-based clusters are the most popular implementations for private clouds.  ...  In the future, we will use more complex workloads to study and evaluate more efficient task schedulers for Hadoop based cloud systems.  ... 
doi:10.1109/bigdata.2013.6691697 dblp:conf/bigdataconf/LiuXLL13 fatcat:tpvvmfinmnfp3mimfldi6bsdw4

Dynamic Load Aware Scheduler of Map Reduce Tasks for Cloud Environments

2019 International journal of recent technology and engineering  
The execution times of Hadoop on cloud can be improved if the virtual resources are effectively used to schedule the tasks by studying the resource usage characteristics of the tasks and resource availability  ...  The proposed work is to build a dynamic scheduler for Hadoop framework which can make scheduling decision dynamically based on job resource usage and node load.  ...  EVALUATION AND RESULTS The proposed dynamic load aware scheduler for Hadoop is evaluated on private and public cloud environments.  ... 
doi:10.35940/ijrte.b1079.0982s1119 fatcat:rrjdnrgikfdujigtnitzpeona4

Locality-aware dynamic VM reconfiguration on MapReduce clouds

Jongse Park, Daewoo Lee, Bokyeong Kim, Jaehyuk Huh, Seungryoul Maeng
2012 Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing - HPDC '12  
Cloud computing based on system virtualization, has been expanding its services to distributed data-intensive platforms such as MapReduce and Hadoop.  ...  DRR dynamically increases or decreases the computing capability of each node to enhance locality-aware task scheduling.  ...  There have been several recent studies to manage virtual resources efficiently in cloud systems [8] . Distributed Resource Scheduler (DRS) proposed a cloud-scale resource management system [6] .  ... 
doi:10.1145/2287076.2287082 dblp:conf/hpdc/ParkLKHM12 fatcat:hvb5mh7hprbehj2n5dqdg4n2u4

SAGS: A SLA-Aware Green Scheduling in Heterogeneous Cloud Using Hadoop YARN

Yadaiah Balagoni, Mahatma Gandhi Institute of Technology, Ramisetty Rao, University College of Engineering, Vizianagaram
2018 International Journal of Intelligent Engineering and Systems  
Hence, this paper proposes the problem of energy-aware heterogeneous Hadoop Yarn cloud with deadline based SLA.  ...  We proposed a SLA-Aware Green Scheduling (SAGS), a Dynamic Voltage/Frequency Scaling (DVFS) based approach along with SLA-Aware scheduling algorithm in the heterogeneous environment.  ...  Our contributions are summarized below, Energy-Aware Eco-friendly SLA-Based scheduler for the heterogeneous environment in the cloud. 2. SAGS is implemented in the Hadoop 2. x open source. 3.  ... 
doi:10.22266/ijies2018.1231.11 fatcat:2nqd6fdrjjeadfdtzqxejfayui

An Accurate and Efficient Scheduler for Hadoop MapReduce Framework

D C Vinutha, G.T Raju
2018 Indonesian Journal of Electrical Engineering and Computer Science  
This work present a novel makespan model for Hadoop MapReduce framework namely OHMR (Optimized Hadoop MapReduce) to process data in real-time and utilize system resource efficiently.  ...  However, the critical challenges of cloud service provider is to meet user task Service level agreement (SLA) requirement (task deadline).  ...  In [13] , an AffordHadoop application is adopted to reduce cost in finishing various tasks and to allocate data and schedule tasks and hence efficiency of system get enhanced.  ... 
doi:10.11591/ijeecs.v12.i3.pp1132-1142 fatcat:oi4s5wxtr5hwbbw4heh6zhuqda

An empirical analysis of scheduling techniques for real-time cloud-based data processing

Linh T. X. Phan, Zhuoyao Zhang, Qi Zheng, Boon Thau Loo, Insup Lee
2011 2011 IEEE International Conference on Service-Oriented Computing and Applications (SOCA)  
Our evaluations on Amazon EC2 show that the existing Hadoop scheduler is ill-equipped to handle jobs with deadlines.  ...  To highlight the challenges in this space, we provide a case study of the online scheduling of MapReduce jobs executed by Hadoop.  ...  Existing Scheduling Policies in Hadoop In the current Hadoop implementation, scheduling of MapReduce jobs proceeds as follows.  ... 
doi:10.1109/soca.2011.6166240 dblp:conf/soca/PhanZZLL11 fatcat:33qzngu6dbfyvg2qhzit7g5hiy

Optimized memory model for hadoop map reduce framework

Archana Bhaskar, Rajeev Ranjan
2019 International Journal of Electrical and Computer Engineering (IJECE)  
This work present a novel memory optimization model for Hadoop Map Reduce framework namely MOHMR (Optimized Hadoop Map Reduce) to process data in real-time and utilize system resource efficiently.  ...  However, the critical challenges of cloud service provider is to meet user task Service level agreement (SLA) requirement (task deadline).  ...  In [13] , an Afford-Hadoop application is adopted to reduce cost in finishing various tasks and to allocate data and schedule tasks and hence efficiency of system get enhanced.  ... 
doi:10.11591/ijece.v9i5.pp4396-4407 fatcat:hrlfba645feivf7uss7jjpy7xu

A Hybrid Scheduling Approach for Scalable Heterogeneous Hadoop Systems

Aysan Rasooli, Douglas G. Down
2012 2012 SC Companion: High Performance Computing, Networking Storage and Analysis  
There is a considerable challenge to schedule the growing number of tasks and resources in a scalable manner.  ...  This solution is being used widely by most Cloud providers. Hadoop schedulers are critical elements for providing desired performance levels. A scheduler assigns MapReduce tasks to Hadoop resources.  ...  for permission to use their workload traces in this research.  ... 
doi:10.1109/sc.companion.2012.155 dblp:conf/sc/OskooeiD12 fatcat:lsg2mdbmkjgxpojr24xbpj74z4

Failure Analysis of Hadoop Schedulers using an Integration of Model Checking and Simulation

Mbarka Soualhia, Foutse Khomh, Sofiene Tahar
2021 Electronic Proceedings in Theoretical Computer Science  
The Hadoop scheduler is a centerpiece of Hadoop, the leading processing framework for data-intensive applications in the cloud.  ...  Next, we use the proposed formal model to analyze the scheduler of OpenCloud, a Hadoop-based cluster that simulates the Hadoop load, in order to illustrate the usability and benefits of our work.  ...  In the cloud, failures are the norm rather than the exception and they often impact the performance of tasks running in Hadoop.  ... 
doi:10.4204/eptcs.342.10 fatcat:nqsbz2rpxjdndo56ppxpmjlamy

Accelerating MapReduce Analytics Using CometCloud

Moustafa AbdelBaky, Hyunjoo Kim, Ivan Rodero, Manish Parashar
2012 2012 IEEE Fifth International Conference on Cloud Computing  
Furthermore, we develop an autonomic manager that can schedule MapReduce tasks based on user objective, provision resources accordingly, and support on-demand scale up and cloudbursts.  ...  Heterogeneity is also unavoidable in scientific applications that process a varying number of datasets of different sizes. In these cases, the performance of MapReduce-Hadoop can be a concern.  ...  The research presented in this paper is supported in part by National Science Foundation via grants numbers IIP 0758566, CCF-0833039, DMS-0835436, CNS 0426354, IIS 0430826, and CNS 0723594, by Department  ... 
doi:10.1109/cloud.2012.150 dblp:conf/IEEEcloud/AbdelBakyKRP12 fatcat:oxqjtykasbffxlhgjgniojb6me

CloudRank-D: benchmarking and ranking cloud computing systems for data processing applications

Chunjie Luo, Jianfeng Zhan, Zhen Jia, Lei Wang, Gang Lu, Lixin Zhang, Cheng-Zhong Xu, Ninghui Sun
2012 Frontiers of Computer Science  
In several case studies, we evaluate two small-scale deployments of cloud computing systems using CloudRank-D.  ...  However, there is no existing benchmark suite for evaluating cloud performance on the whole system level.  ...  Hadoop schedulers The scheduler in Hadoop is a pluggable component, and system managers can customize the scheduler according their requirements. There are three schedulers in Hadoop 0.20.2.  ... 
doi:10.1007/s11704-012-2118-7 fatcat:jatll7tuuvf33dfxssntrowmau

Evaluating Virtualization for Hadoop MapReduce on an OpenNebula Cloud

Pedro Roger Magalhães Vasconcelos, Gisele Azevedo de Araújo Freitas
2014 International Journal of Multimedia and Image Processing  
In this work, we conducted benchmarks to measure the performance of a Hadoop cluster deployed on OpenNebula clouds with KVM and OpenVZ.  ...  In this context, virtualization has been used as a platform for resource-intensive applications, like Hadoop, as it has brought features like server consolidation, scalability and better resources usage  ...  Experimental Evaluation To evaluate the performance of Hadoop cluster on each virtualization platform, we propose the establishment of two private OpenNebula clouds.  ... 
doi:10.20533/ijmip.2042.4647.2014.0029 fatcat:upuyub3fabcq7gtvpsqqyu57sq

Performance evaluation and resource optimization of cloud based parallel Hadoop clusters with an intelligent scheduler

Manishankar S., S. Sathayanarayana
2018 International Journal of Engineering & Technology  
Processing of such a huge incremental data in large scale requires a parallel processing system like Hadoop based cluster.  ...  Major challenge that arises in all cluster-based system is how efficiently the resources of the system can be used.  ...  As cloud took over the storage profile for most of the information systems, private cloud came as solution for the workload balancing for the Hadoop based systems.  ... 
doi:10.14419/ijet.v7i4.13372 fatcat:sjoercq2mzaz7i3kiicourmtqy

LSTPD: Least Slack Time-Based Preemptive Deadline Constraint Scheduler for Hadoop Clusters

Ihsan Ullah, Muhammad Sajjad Khan, Muhammad Amir, Junsu Kim, Su Min Kim
2020 IEEE Access  
In order for better allocation of tasks and load balancing, we first analyze the task scheduling behaviors of the Hadoop platform.  ...  In this paper, we propose an efficient preemptive deadline constraint scheduler based on least slack time and data locality.  ...  PERFORMANCES EVALUATION In this section, we describe our experimental setup for evaluating the proposed LSTPD scheduling scheme by using Hadoop 2.7.2 (stable), Linux operating system, and Cen-tOS.  ... 
doi:10.1109/access.2020.3002565 fatcat:yvvxyliqavakfkmti5sim7tv3a

Empirical Study of Job Scheduling Algorithms in Hadoop MapReduce

Jyoti V. Gautam, Harshadkumar B. Prajapati, Vipul K. Dabhi, Sanjay Chaudhary
2017 Cybernetics and Information Technologies  
This article focuses on empirically evaluating the performance of three schedulers: First In First Out (FIFO), Fair scheduler, and Capacity scheduler.  ...  To carry out the experimental evaluation, we implement our own Hadoop cluster testbed, consisting of four machines, in which one of the machines works as the master node and all four machines work as slave  ...  In [8] , Cloud benchmark suite CloudRank-D is used to evaluate different Hadoop job schedulers, which include FIFO scheduler, Fair scheduler, Fair scheduler with delay, Capacity scheduler, and Hadoop  ... 
doi:10.1515/cait-2017-0012 fatcat:kocust3jjfcqjl4thmkbcouiii
« Previous Showing results 1 — 15 out of 5,368 results