WaxElephant: A Realistic Hadoop Simulator for Parameters Tuning and Scalability Analysis

Zujie Ren, Zhijun Liu, Xianghua Xu, Jian Wan, Weisong Shi, Min Zhou
2012 2012 Seventh ChinaGrid Annual Conference  
MapReduce is becoming the state-of-the-art computation paradigm for processing large-scale datasets on a large cluster with tens or thousands of nodes. Hadoop, an open-source implementation of MapReduce framework, has gained much popularity due to its high scalability and performance. Two challenging issues for a large-scale Hadoop cluster are how to analyze the scalability and identify the optimal parameters configurations. To address these issues, we designed and implemented a Hadoop
more » ... called WaxElephant, which provides the following capabilities: (1) loading real MapReduce workloads derived from the historical log of Hadoop clusters, and replaying the job execution history; (2) synthesizing workloads and executing them based on statistical characteristics of workloads; (3) identifying the optimal parameters configurations; and (4) analyzing the scalability of the cluster. Extensive experiments have been conducted to validate the accuracy of the WaxElephant simulator.
doi:10.1109/chinagrid.2012.25 dblp:conf/chinagrid/RenLXWSZ12 fatcat:6ed3oni6abcmjhqxyzmmdjoaba