Sizing sketches

Zhe Wang, Wei Dong, William Josephson, Qin Lv, Moses Charikar, Kai Li
2007 Proceedings of the 2007 ACM SIGMETRICS international conference on Measurement and modeling of computer systems - SIGMETRICS '07  
Sketches are compact data structures that can be used to estimate properties of the original data in building largescale search engines and data analysis systems. Recent theoretical and experimental studies have shown that sketches constructed from feature vectors using randomized projections can effectively approximate 1 distance on the feature vectors with the Hamming distance on their sketches. Furthermore, such sketches can achieve good filtering accuracy while reducing the metadata space
more » ... quirement and speeding up similarity searches by an order of magnitude. However, it is not clear how to choose the size of the sketches since it depends on data type, dataset size, and desired filtering quality. In real systems designs, it is necessary to understand how to choose sketch size without the dataset, or at least without the whole dataset. This paper presents an analytical model and experimental results to help system designers make such design decisions. We present a rank-based filtering model that describes the relationship between sketch size and dataset size based on the dataset distance distribution. Our experimental results with several datasets including images, audio, and 3D shapes show that the model yields good, conservative predictions. We show that the parameters of the model can be set with a small sample dataset and the resulting model can make good predictions for a large dataset. We illustrate how to apply the approach with a concrete example.
doi:10.1145/1254882.1254900 dblp:conf/sigmetrics/WangDJLCL07 fatcat:2sno2wjvpzb3pfjourym45u7se