Final Report: Efficient Databases for MPC Microdata
Project Overview The purpose of this grant was to develop the theory and practice of high-performance databases for massive streamed datasets. Over the last three years, we have developed fast indexing technology, that is, technology for rapidly ingesting data and storing that data so that it can be efficiently queried and analyzed. Streamed datasets include: Simulation querying: Many high-end simulations produce vast quantities of data, and that data needs to be stored and queried.
... ueried. Astronomical imaging: The entire sky may be imaged on a regular basis and later the images from a particular region in the picture are compared, for example, to monitor supernovae. Reconnaissance monitoring: A set of regions may be imaged daily, and later some regions are compared over time, for example, to monitor enemy movement. Network monitoring: A communications network is instrumented, with each router providing information about network flows and traffic, and then information about traffic from a particular source is examined. For data-analysis applications involving high-bandwidth data streams, HPC practitioners typically must make a difficult choice between employing a database or a file system for storing their persistent data. Databases often perform operations slowly, partially because of the extra weight from the SLQ from end and from the transactional support, leading programmers to choose file systems instead. On the other hand, file systems provide poor data-consistency guarantees, which are increasingly important for large data sets. However, both file systems and databases typically provide poor performance when they are indexing microdata. Microdata means small objects, much smaller than the natural block size of storage devices. Indexing means ingesting objects and ordering them on disk so that they can be queried efficiently. For example, one can ingest mouse-click data roughly in temporal order, but then index the clicks first ordered by user and website location and by then time; then a query to find all mouse clicks on a particularly website from users, say, in Long Island is efficient. As another example, one can ingest astronomical data in rough temporal order but index the data to optimize queries about differences in particular starts or regions of sky over time. During this project we developed the technology so that high-bandwidth data streams can be indexed and queried efficiently. Our technology has been proven to work data sets composed of tens of billions of rows when the data streams arrives at over 40,000 rows per second. We achieved these numbers even on a single disk driven by two cores. Our work comprised (1) new write-optimized data structures with better asymptotic complexity than traditional structures, (2) implementation, and (3) benchmarking. We furthermore developed a prototype of TokuFS, a middleware layer that can handle microdata I/O packaged up in an MPI-IO abstraction. Massively Parallel Streaming B-trees During this grant we built a parallel and concurrent streaming B-tree or Fractal Tree. We developed the technology to transform the streaming B-tree into a highly parallel, concurrent, scalable structure of interest to users of highend clusters. We showed how such a data structure can be used to build a high-performance database capable of