Functional programming for dynamic and large data with self-adjusting computation

Yan Chen, Umut A. Acar, Kanat Tangwongsan
2014 SIGPLAN notices  
Combining type theory, language design, and empirical work, we present techniques for computing with large and dynamically changing datasets. Based on lambda calculus, our techniques are suitable for expressing a diverse set of algorithms on large datasets and, via self-adjusting computation, enable computations to respond automatically to changes in their data. Compared to prior work, this work overcomes the main challenge of reducing the space usage of self-adjusting computation without
more » ... portionately decreasing performance. To this end, we present a type system for precise dependency tracking that minimizes the time and space for storing dependency metadata. The type system eliminates an important assumption of prior work that can lead to recording of spurious dependencies. We give a new type-directed translation algorithm that generates correct self-adjusting programs without relying on this assumption. We then show a probabilistic chunking technique to further decrease space usage by controlling the fundamental space-time tradeoff in self-adjusting computation. We implement and evaluate these techniques, showing very promising results on challenging benchmarks and large graphs.
doi:10.1145/2692915.2628150 fatcat:q7xsdfgiyzfb5ek753s3r5zjlm