Dynamic Load Balancing Method for Apache Flume Log Processing

UnGyu Han, Jinho Ahn
2014 unpublished
Recently, as web log data their users leave daily rapidly emerge as valuable assets for web service companies, designing log aggregator for collecting these log data in an efficient manner is also getting high attention in big data analytics research. However, we observe a representative data aggregator, Apache Flume, used for this purpose has some drawbacks on evenly distributing incoming log workload on collector agents. In this paper, we propose a new load balancing method to overcome this
more » ... to overcome this limitation in terms of the collected performance. The proposed method considerably alleviates the additional overhead incurred by the task migration and makes the load of the entire system as fair as possible by selecting the optimal task migration destination depending on the current load-state values of collector agents unlike the previous round-robin and random ones.
doi:10.14257/astl.2014.79.16 fatcat:brinmutrzfgy5fyhku3432vm44