Reshaping text data for efficient processing on Amazon EC2

Gabriela Turcu, Ian Foster, Svetlozar Nestorov
2010 Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing - HPDC '10  
Text analysis tools are nowadays required to process increasingly large corpora which are often organized as small files (abstracts, news articles, etc). Cloud computing offers a convenient, on-demand, pay-as-you-go computing environment for solving such problems. We investigate provisioning on the Amazon EC2 cloud from the user perspective, attempting to provide a scheduling strategy that is both timely and cost effective. We rely on the empirical performance of the application of interest on
more » ... maller subsets of data, to construct an execution plan. A first goal of our performance measurements is to determine an optimal file size for our application to consume. Using the subset-sum first fit heuristic we reshape the input data by merging files in order to match as closely as possible the desired file size. This also speeds up the task of retrieving the results of our application, by having the output be less segmented. Using predictions of the performance of our application based on measurements on small data sets, we devise an execution plan that meets a user specified deadline while minimizing cost.
doi:10.1145/1851476.1851540 dblp:conf/hpdc/TurcuFN10 fatcat:nmmgwvgflrc3dnqr2ptuuhj2xa