Heavy-tailed distributions and multi-keyword queries

Surajit Chaudhuri, Kenneth Church, Arnd Christian König, Liying Sui
2007 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '07  
Intersecting inverted indexes is a fundamental operation for many applications in information retrieval and databases. Efficient indexing for this operation is known to be a hard problem for arbitrary data distributions. However, text corpora used in Information Retrieval applications often have convenient power-law constraints (also known as Zipf's Law and long tails) that allow us to materialize carefully chosen combinations of multi-keyword indexes, which significantly improve worst-case
more » ... ormance without requiring excessive storage. These multi-keyword indexes limit the number of postings accessed when computing arbitrary index intersections. Our evaluation on an e-commerce collection of 20 million products shows that the indexes of up to four arbitrary keywords can be intersected while accessing less than 20% of the postings in the largest single-keyword index.
doi:10.1145/1277741.1277855 dblp:conf/sigir/ChaudhuriCKS07 fatcat:2e7xp3ocqfcafpyjii6pewb5ae