A document-centric approach to static index pruning in text retrieval systems

Stefan Büttcher, Charles L. A. Clarke
2006 Proceedings of the 15th ACM international conference on Information and knowledge management - CIKM '06  
We present a static index pruning method, to be used in ad-hoc document retrieval tasks, that follows a documentcentric approach to decide whether a posting for a given term should remain in the index or not. The decision is made based on the term's contribution to the document's Kullback-Leibler divergence from the text collection's global language model. Our technique can be used to decrease the size of the index by over 90%, at only a minor decrease in retrieval effectiveness. It thus allows
more » ... us to make the index small enough to fit entirely into the main memory of a single PC, even for large text collections containing millions of documents. This results in great efficiency gains, superior to those of earlier pruning methods, and an average response time around 20 ms on the GOV2 document collection.
doi:10.1145/1183614.1183644 dblp:conf/cikm/ButtcherC06 fatcat:3hq2diemnbennie4kj2t4vb6mq