Streaming big data with self-adjusting computation

Umut A. Acar, Yan Chen
2013 Proceedings of the 2013 workshop on Data driven functional programming - DDFP '13  
Many big data computations involve processing data that changes incrementally or dynamically over time. Using existing techniques, such computations quickly become impractical. For example, computing the frequency of words in the first ten thousand paragraphs of a publicly available Wikipedia data set in a streaming fashion using MapReduce can take as much as a full day. In this paper, we propose an approach based on self-adjusting computation that can dramatically improve the efficiency of
more » ... computations. As an example, we can perform the aforementioned streaming computation in just a couple of minutes.
doi:10.1145/2429376.2429382 dblp:conf/popl/AcarC12 fatcat:bfufhf354zchjaalfnfoym6wk4