Impact of Word Segmentation Errors on Automatic Chinese Text Classification

Xi Luo, Wataru Ohyama, Tetsushi Wakabayashi, Fumitaka Kimura
2012 2012 10th IAPR International Workshop on Document Analysis Systems  
In this paper, several sets of experiments were carried out to study the impact of word segmentation errors on automatic Chinese text classification. Comparison experiment of four word-based approaches was first carried out and the results show that the performance was significantly reduced when using automatic word segmentation instead of manual word segmentation which means errors caused by automatic word segmentation have an obvious impact on classification performance. We further conducted
more » ... further conducted the experiment using character-based approach (N-gram). Although N-gram approach produces a large number of ambiguous words, the results show that it performed better than automatic word segmentation.
doi:10.1109/das.2012.43 dblp:conf/das/LuoOWK12 fatcat:a5z2a7cvqncwvma26c63jprn5a