I/O streaming evaluation of batch queries for data-intensive computational turbulence

Kalin Kanov, Eric Perlman, Randal Burns, Yanif Ahmad, Alexander Szalay
2011 Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11  
We describe a method for evaluating computational turbulence queries, including Lagrange Polynomial interpolation, based on partial sums that allows the underlying data to be accessed in any order and in parts. We exploit these properties to stream data from disk in a single pass and concurrently evaluate batch queries. The combination of sequential I/O and data sharing improves performance by an order of magnitude when compared with direct evaluation of each query. The technique also supports
more » ... istributed evaluation of queries in a database cluster, assembling the partial sums from each node at the query mediator. Interpolation is fundamental to computational turbulence, over 95% of queries use these routines, and the partial sums method allows the JHU Turbulence Database Cluster to realize scale and throughput for our scientists' data-intensive workloads.
doi:10.1145/2063384.2063423 dblp:conf/sc/KanovPBAS11 fatcat:lo7pqmg6vzbjfg4tqvlr5pa7vq