Filters








14,941 Hits in 2.6 sec

Cluster scheduling for explicitly-speculative tasks

David Petrou, Gregory R. Ganger, Garth A. Gibson
2004 Proceedings of the 18th annual international conference on Supercomputing - ICS '04  
Large-scale computing often consists of many speculative tasks to test hypotheses, search for insights, and review potentially finished products.  ...  which tasks are speculative.  ...  explicitly supports this model.  ... 
doi:10.1145/1006209.1006256 dblp:conf/ics/PetrouGG04 fatcat:xodvih624rhdnnx2dxxllvsgtq

Improving MapReduce Performance in Heterogeneous Network Environments and Resource Utilization

Zhenhua Guo, Geoffrey Fox
2012 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012)  
We investigate network heterogeneity aware scheduling of both map and reduce tasks.  ...  To alleviate the issue, we propose Benefit Aware Speculative Execution which predicts the benefit of launching new speculative tasks and greatly eliminates unnecessary runs of speculative tasks.  ...  With default policy, Hadoop will launch a speculative task T 1 ' for T 1 . We assume T 1 ' and T 2 progress equally fast.  ... 
doi:10.1109/ccgrid.2012.12 dblp:conf/ccgrid/GuoF12 fatcat:7cmx4wtff5g2dcsjpzdjqiqfqy

Optimization for Speculative Execution of Multiple Jobs in a MapReduce-like Cluster [article]

Huanle Xu, Wing Cheong Lau
2015 arXiv   pre-print
In this paper, we focus on the design of speculative execution schemes for a parallel processing cluster under different loading conditions.  ...  We also derive the workload threshold under which SCA or SDA should be used for speculative execution.  ...  Input: The jobs in the cluster associated with their running status at time slot l; Output: Scheduling decisions for time slot l. 1 schedule the unassigned tasks of the running jobs in the cluster with  ... 
arXiv:1406.0609v3 fatcat:ept2yi2xabhs3ldn5r6heag2c4

Asynchronous Non-Blocking Algorithm to Handle Straggler Reduce Tasks in Hadoop System

Arwan A. Khoiruddin, Nordin Zakaria, Hitham Seddig Alhussian
2020 International Journal on Advanced Science, Engineering and Information Technology  
When the task is ready to run, it will be sent back to the main task queue for running.  ...  However, the algorithms manage Map and Reduce task similarly, while the straggler root cause might be different for both tasks.  ...  The scheduler is a good default for small to medium-sized clusters.  ... 
doi:10.18517/ijaseit.10.5.9073 fatcat:ope67evg3rcdjnvevsqxuwxeiq

Performance-aware Speculative Resource Oversubscription for Large-scale Clusters

Renyu Yang, Chunming Hu, Xiaoyang Sun, Peter Garraghan, Tianyu Wo, Zhenyu Wen, Hao Peng, Jie Xu, Chao Li
2020 IEEE Transactions on Parallel and Distributed Systems  
Instead of waiting for resource allocation to be confirmed by the centralized scheduler, job managers in ROSE can independently request to launch speculative tasks within specific machines according to  ...  It is a long-standing challenge to achieve a high degree of resource utilization in cluster scheduling.  ...  ACKNOWLEDGMENT We would like to thank Alibaba Group, particularly colleagues from Fuxi scheduling team for their collaboration.  ... 
doi:10.1109/tpds.2020.2970013 fatcat:2tftu6ehofcqflyfhumaytry7u

Task-Cloning Algorithms in a MapReduce Cluster with Competitive Performance Bounds [article]

Huanle Xu, Wing Cheong Lau
2015 arXiv   pre-print
Job scheduling for a MapReduce cluster has been an active research topic in recent years.  ...  cluster based on the Shortest Remaining Processing Time scheduler (SRPT).  ...  A stochastic program formulation for job scheduling For any job, all the map tasks and reduce tasks can only be scheduled after the job arrival at the cluster and hence Finally, a job completes when all  ... 
arXiv:1501.02330v1 fatcat:n5hoiptz7benvp3jjqg32txnvq

Optimization strategies for A/B testing on HADOOP

Andrii Cherniak, Huma Zaidi, Vladimir Zadorozhny
2013 Proceedings of the VLDB Endowment  
Our proposed solution relies on proper resource allocation for concurrent Hive jobs based on data dependency, inter-query optimization and modeling of Hadoop cluster load.  ...  This work was performed on eBay production Hadoop cluster.  ...  We thank Tony Thrall, manager of the EDA team at eBay Inc. for permission to publish the results of this study.  ... 
doi:10.14778/2536222.2536224 fatcat:hu7hwxhcfbf73ka2wbyrnkt4wq

Effective Straggler Mitigation: Attack of the Clones

Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica
2013 Symposium on Networked Systems Design and Implementation  
Small jobs, that are typically run for interactive data analyses in datacenters, continue to be plagued by disproportionately long-running tasks called stragglers.  ...  slower than the median task in that job.  ...  When a job is submitted, its tasks are queued at the scheduler. For every queued task, the scheduler spawns many clones.  ... 
dblp:conf/nsdi/AnanthanarayananGSS13 fatcat:5hmptbtvvbanhdnr6ibjvv42ga

Wrangler

Neeraja J. Yadwadkar, Ganesh Ananthanarayanan, Randy Katz
2014 Proceedings of the ACM Symposium on Cloud Computing - SOCC '14  
In particular, by using these predictions to balance delay in task scheduling against the potential for idling of resources, Wrangler achieves a speed up in the overall job completion time.  ...  For production-level workloads from Facebook and Cloudera's customers, Wrangler improves the 99 th percentile job completion time by up to 61% as compared to speculative execution, a widely used straggler  ...  We also thank our shepherd, Fred Douglis, for help in shaping the final version of the paper.  ... 
doi:10.1145/2670979.2671005 dblp:conf/cloud/YadwadkarAK14 fatcat:rfwiymfinrdj3iq4446ogppvee

The Case for Tiny Tasks in Compute Clusters

Kay Ousterhout, Aurojit Panda, Josh Rosen, Shivaram Venkataraman, Reynold Xin, Sylvia Ratnasamy, Scott Shenker, Ion Stoica
2013 USENIX Workshop on Hot Topics in Operating Systems  
We argue for breaking data-parallel jobs in compute clusters into tiny tasks that each complete in hundreds of milliseconds.  ...  Tiny tasks avoid the need for complex skew mitigation techniques: by breaking a large job into millions of tiny tasks, work will be evenly spread over available resources by the scheduler.  ...  Acknowledgments We thank Matei Zaharia, Colin Scott, John Ousterhout, and Patrick Wendell for useful feedback on earlier drafts of this paper.  ... 
dblp:conf/hotos/OusterhoutPRVXRSS13 fatcat:7jn7fbnrgrclvhinapthwhs2oa

Privacy Preservation in Analyzing EHealth Records in Big Data Environment

E Srimathi
2015 International Journal on Recent and Innovation Trends in Computing and Communication  
Third, data locality can be improved without any impact on fairness using Slot Pre Scheduling.  ...  First, the under-utilization of map and reduce tasks is improved based on Dynamic Hadoop Slot Allocation (DHSA).  ...  Zaharia.M et al. achieved locality and fairness [18] in cluster scheduling using delay scheduling.  ... 
doi:10.17762/ijritcc2321-8169.1504139 fatcat:357gk2cdnneljcdihzyoa2cbkm

Intelligent Resource Scheduling at Scale: A Machine Learning Perspective

Renyu Yang, Xue Ouyang, Yaofeng Chen, Paul Townend, Jie Xu
2018 2018 IEEE Symposium on Service-Oriented System Engineering (SOSE)  
The exhibited heterogeneity of workload and server characteristics in Cloud-scale or Internetscale environments has raised unprecedented new challenges for cluster scheduling.  ...  Compared with ad-hoc heuristics for a multi-resource cluster scheduling problem, machine learning (ML) approaches can in turn facilitate improved efficiency in resource management.  ...  However, its performance can be undermined due to the fact that speculative copies could occupy available resources for new tasks.  ... 
doi:10.1109/sose.2018.00025 dblp:conf/sose/YangOCTX18 fatcat:yxyi5lwxmjdtdpqbrvblbqan5u

Cloud Resource Allocation as Preemptive Scheduling Approach

Suhas YuvrajBadgujar, Anand Bone
2014 International Journal of Computer Applications  
To achieve this many scheme are proposed, Nephele is one of the data processing framework which exploits the dynamic resource allocation offered by IaaS clouds for both task scheduling and execution.  ...  This paper introduces a new Approach for increasing the efficiency of the scheduling algorithm for the real time Cloud Computing services.  ...  Nephele is the data processing framework to explicitly exploit the dynamic resource allocation for both task scheduling and execution.  ... 
doi:10.5120/15452-3989 fatcat:dwzbdkziunbnhixsrzuzld247q

Fault Tolerance in MapReduce: A Survey [chapter]

Bunjamin Memishi, Shadi Ibrahim, María S. Pérez, Gabriel Antoniu
2016 Computer Communications and Networks  
of these tasks on other machines.  ...  Furthermore, in order to mask temporary failures caused by network or machine overload (timing failure) where some tasks are performing relatively slower than other tasks, Hadoop relaunches other copies  ...  Greedy Speculative Scheduling (GS). This algorithm is intended to greedily pick a task that will be scheduled next. 2. Resource Aware Speculative Scheduling (RAS).  ... 
doi:10.1007/978-3-319-44881-7_11 dblp:series/ccn/MemishiIPA16 fatcat:m5x33gpzunhzzgrdslagndiwzy

Observations on Factors Affecting Performance of MapReduce based Apriori on Hadoop Cluster [article]

Sudhakar Singh, Rakhi Garg, P. K. Mishra
2017 arXiv   pre-print
Designing fast and scalable algorithm for mining frequent itemsets is always being a most eminent and promising problem of data mining.  ...  The non-algorithmic factors include speculative execution, nodes with poor performance, data locality & distribution of data blocks, and parallelism control with input split size.  ...  Fig. 3 it can be seen that speculative execution comes in action only for a job taking longer time to complete. Speculative task is not launched for short jobs.  ... 
arXiv:1701.05982v1 fatcat:mt3nog3purhwbkh2pxwnhoc75a
« Previous Showing results 1 — 15 out of 14,941 results