NativeTask: A Hadoop compatible framework for high performance

Dong Yang, Xiang Zhong, Dong Yan, Fangqin Dai, Xusen Yin, Cheng Lian, Zhongliang Zhu, Weihua Jiang, Gansha Wu
2013 2013 IEEE International Conference on Big Data  
Although Hadoop MapReduce provides good programming abstractions and horizontal scalability, it's often blamed for its poor single node performance. In the meantime, MapReduce has already achieved a large install base, thus any performance improvement should keep the compatibility. In this paper, we address the challenges via several approaches guided by low-level performance analysis. And we materialize the approaches via NativeTask, a high-performance, fully compatible MapReduce execution
more » ... ne. We evaluate its performance with representative HiBench workloads. The results show that the speedup NativeTask achieves ranges from 10% to 160%, and it paves the way for a better MapReduce that excels on both single node performance and scalability. In the future, hardware acceleration can also be applied to further improve the system's efficiency.
doi:10.1109/bigdata.2013.6691703 dblp:conf/bigdataconf/YangZYDYLZJW13 fatcat:i6oblu6g6ffozaqc6supzl7zri