A Game-Theoretic Approach for Runtime Capacity Allocation in MapReduce

Eugenio Gianniti, Danilo Ardagna, Michele Ciavotta, Mauro Passacantando
2017 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)  
Nowadays many companies have available large amounts of raw, unstructured data. Among Big Data enabling technologies, a central place is held by the MapReduce framework and, in particular, by its open source implementation, Apache Hadoop. For cost effectiveness considerations, a common approach entails sharing server clusters among multiple users. The underlying infrastructure should provide every user with a fair share of computational resources, ensuring that service level agreements (SLAs)
more » ... agreements (SLAs) are met and avoiding wastes. In this paper we consider mathematical models for the optimal allocation of computational resources in a Hadoop 2.x cluster with the aim to develop new capacity allocation techniques that guarantee better performance in shared data centers. Our goal is to get a substantial reduction of power consumption while respecting the deadlines stated in the SLAs and avoiding penalties associated with job rejections. The core of this approach is a distributed algorithm for runtime capacity allocation, based on Game Theory models and techniques, that mimics the Map-Reduce dynamics by means of interacting players, namely the central Resource Manager and Class Managers.
doi:10.1109/ccgrid.2017.135 dblp:conf/ccgrid/GiannitiACP17 fatcat:zmfw6ypwebel5mfyflh3ucfkki