Contextual information retrieval based on algorithmic information theory and statistical outlier detection

Rafael Martinez, Manuel Cebrian, Francisco de Borja Rodriguez, David Camacho
2008 2008 IEEE Information Theory Workshop  
The main contribution of this paper is to design an Information Retrieval (IR) technique based on Algorithmic Information Theory (using the Normalized Compression Distance- NCD), statistical techniques (outliers), and novel organization of data base structure. The paper shows how they can be integrated to retrieve information from generic databases using long (text-based) queries. Two important problems are analyzed in the paper. On the one hand, how to detect "false positives" when the
more » ... among the documents is very low and there is actual similarity. On the other hand, we propose a way to structure a document database which similarities distance estimation depends on the length of the selected text. Finally, the experimental evaluations that have been carried out to study previous problems are shown.
doi:10.1109/itw.2008.4578672 dblp:conf/itw/MartinezCRC08 fatcat:zxqy3u4rivaihgdvzjyr5wc6lm