Keyword Extraction from a Document using Word Co-occurrence Statistical Information

Yutaka Matsuo, Mitsuru Ishizuka
2002 Transactions of the Japanese society for artificial intelligence  
We present a new keyword extraction algorithm that applies to a single document without using a large corpus. Frequent terms are extracted first, then a set of co-occurrence between each term and the frequent terms, i.e., occurrences in the same sentences, is generated. The distribution of co-occurrence shows the importance of a term in the document as follows. If the probability distribution of co-occurrence between term a and the frequent terms is biased to a particular subset of the frequent
more » ... set of the frequent terms, then term a is likely to be a keyword. The degree of the biases of the distribution is measured by χ 2 -measure. We show our algorithm performs well for indexing technical papers.
doi:10.1527/tjsai.17.217 fatcat:gecn3g6qjrbdpmgdll6sq2qbxq