Creating and exploiting a comparable corpus in cross-language information retrieval

Tuomas Talvensaari, Jorma Laurikkala, Kalervo Järvelin, Martti Juhola, Heikki Keskustalo
2007 ACM Transactions on Information Systems  
We present a method for creating a comparable text corpus from two document collections in different languages. The collections can be very different in origin: in this study we build a comparable corpus from articles by a Swedish news agency and a U.S. newspaper. The keys with best resolution power were extracted from the documents of one collection, the source collection, by using the relative average term frequency (RATF) value. The keys were translated into the language of the other
more » ... on, the The authors are grateful to the Academy of Finland for financial support of the present research (grants 202185, 206568, 200844, 80771, and 204978). target collection, with a dictionary-based query translation program. The translated queries were run against the target collection and an alignment pair was made if the retrieved documents matched given date and similarity score criteria. The resulting comparable collection was used as a similarity thesaurus to translate queries along with a dictionary-based translator. The combined approaches outperformed translation schemes where dictionary-based translation or corpus translation was used alone.
doi:10.1145/1198296.1198300 fatcat:ajomowrcl5agphti32eltqcmfy