Comparing representations in Chinese information retrieval

K. L. Kwok
1997 Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '97  
Three representation methods are empirically investigated for Chinese information retrieval: 1-gram (single character), bigram (two contiguous overlapping characters), and short-word indexing based on a simple segmentation of the text. The retrieval collection is the approximately 170 MB TREC-5 Chinese corpus of news articles, and 28 queries that are long and rich in wordings. Evaluation shows that 1-gram indexing is good but not sufficiently competitive, while bigram indexing works
more » ... well. Bigram indexing leads to a large index term space, three times that of short-word indexing, but is as good as short-word indexing in precision, and about 5% better in relevants retrieved. The best average non-interpolated precision is about 0.45, 17% better than 1-gram indexing and quite high for a mainly statistical approach.
doi:10.1145/258525.258531 dblp:conf/sigir/Kwok97 fatcat:g44yl7qpz5hphjpwalrnthquxm