BIGhybrid -- A Toolkit for Simulating MapReduce in Hybrid Infrastructures

Julio C.S. dos Anjos, Gilles Fedak, Claudio F.R. Geyer
2014 2014 International Symposium on Computer Architecture and High Performance Computing Workshop  
Cloud computing has increasingly been used as a platform for running large business and data processing applications. Although clouds have become extremely popular, when it comes to data processing, their use incurs high costs. Conversely, Desktop Grids, have been used in a wide range of projects, and are able to take advantage of the large number of resources provided by volunteers, free of charge. Merging cloud computing and desktop grids into a hybrid infrastructure can provide a feasible
more » ... -cost solution for big data analysis. Although frameworks like MapReduce have been devised to exploit commodity hardware, their use in a hybrid infrastructure raise some challenges due to their large resource heterogeneity and high churn rate. This study introduces BIGhybrid, a toolkit that is used to simulate MapReduce in hybrid environments. Its main goal is to provide a framework for developers and system designers that can enable them to address the issues of Hybrid MapReduce. In this paper, we describe the framework which simulates the assembly of two existing middleware: BitDew-MapReduce for Desktop Grids and Hadoop-BlobSeer for Cloud Computing. The experimental results that are included in this work demonstrate the feasibility of our approach.
doi:10.1109/sbac-padw.2014.8 dblp:conf/sbac-pad/AnjosFG14 fatcat:2iz4zeicovatzfnz6fc3y64rpa