Parallel computation method for incremental cloud services based on Hadoop

Yonghong Li, Na Zhou, Guofeng Zhao
2016 Proceedings of the 2016 4th International Conference on Electrical & Electronics Engineering and Computer Science (ICEEECS 2016)   unpublished
Performance of Hadoop platform is closely related to the task scheduler. Based on the analysis of the existing Hadoop platform's job scheduling algorithm, I propose a multi-queue job scheduling optimization algorithm in the paper. Based on the actual test and analysis of Hadoop platform, it can be seen that the optimization algorithm proposed in this paper can effectively allocate the node resources according to the degree of demand for the node resource. It can achieve the resource sharing
more » ... esource sharing among multiple queues. And at the same time, it can also effectively avoid the ping-pong effect brought by resource competition at the same time. Optimization algorithm has greater improvements in execution efficiency than Hadoop default algorithm.
doi:10.2991/iceeecs-16.2016.220 fatcat:64rytobwlfdavfyl65kcunwkbe