Comparing inverted files and signature files for searching a large lexicon

Ben Carterette, Fazli Can
2005 Information Processing & Management  
Signature files and inverted files are well-known index structures. In this paper we undertake a direct comparison of the two for searching for partially-specified queries in a large lexicon stored in main memory. Using n-grams to index lexicon terms, a bit-sliced signature file can be compressed to a smaller size than an inverted file if each n-gram sets only one bit in the term signature. With a signature width less than half the number of unique n-grams in the lexicon, the signature file
more » ... od is about as fast as the inverted file method, and significantly smaller. Greater flexibility in memory usage and faster index generation time make signature files appropriate for searching large lexicons or other collections in an environment where memory is at a premium.
doi:10.1016/j.ipm.2003.12.003 fatcat:ndkoxvtxuff7dptnhybzxrz5ja