A Community Cache with Complete Information

Mania Abdi, Amin Mosayyebzadeh, Mohammad Hossein Hajkazemi, Emine Ugur Kaynar, Ata Turk, Larry Rudolph, Orran Krieger, Peter Desnoyers
2021 USENIX Conference on File and Storage Technologies  
Kariz is a new architecture for caching data from datalakes accessed, potentially concurrently, by multiple analytic platforms. It integrates rich information from analytics platforms with global knowledge about demand and resource availability to enable sophisticated cache management and prefetching strategies that, for example, combine historical run time information with job dependency graphs (DAGs), information about the cache state and sharing across compute clusters. Our prototype
more » ... multiple analytic frameworks (Pig/Hadoop and Spark), and we show that the required changes are modest. We have implemented three algorithms in Kariz for optimizing the caching of individual queries (one from the literature, and two novel to our platform) and three policies for optimizing across queries from, potentially, multiple di erent clusters. With an algorithm that fully exploits the rich information available from Kariz, we demonstrate major speedups (as much as ×) for TPC-H and TPC-DS.
dblp:conf/fast/AbdiMHKTRKD21 fatcat:zv3megbxrrfd3ayxkbelbjxpfm