Shared-private Information Bottleneck Method for Cross-modal Clustering

Xiaoqiang Yan, Yangdong Ye, Yiqiao Mao, Hui Yu
2019 IEEE Access  
Cross-modal analysis has recently drawn much attention due to the rapid growth and widespread emergence of multimodal data. It integrates multiple modalities to improve the learning and generalization performance. However, most previous methods just focus on learning a common shared feature space for all modalities and ignore the private information hidden in each individual modality. To address this problem, we propose a novel shared-private information bottleneck (SPIB) method for crossmodal
more » ... lustering. Firstly, we devise a hybrid words model and a consensus clustering model to construct the shared information of multiple modalities, which capture the statistical correlation of low-level features and the semantic relations of the high-level clustering partitions, respectively. Secondly, the shared information of multiple modalities and the private information of individual modalities are maximally preserved through a unified information maximization function. Finally, the optimization of SPIB function is performed by a sequential "draw-and-merge" procedure, which guarantees the function converge to a local maximum. Besides, to solve the lack of tags in cross-modal social images, we also investigate the use of structured prior knowledge in the form of knowledge graph to enrich the information in semantic modality, and design a novel semantic similarity measurement for social images. Experimental results on four types of crossmodal datasets demonstrate that our method outperforms the state-of-the-art approaches. INDEX TERMS Cross-modal clustering, information bottleneck, mutual information, knowledge graph, social images.
doi:10.1109/access.2019.2904554 fatcat:nzjins6k4rfd3atralf23vhwlq