Visual Attention in Multi-Label Image Classification

Yan Luo, Ming Jiang, Qi Zhao
2019 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)  
One of the most significant challenges in multi-label image classification is the learning of representative features that capture the rich semantic information in a cluttered scene. As an information bottleneck, the visual attention mechanism allows humans to selectively process the most important visual input, enabling rapid and accurate scene understanding. In this work, we study the correlation between visual attention and multi-label image classification, and exploit an extra attention
more » ... way for improving multilabel image classification performance. Specifically, we propose a dual-stream neural network that consists of two sub-networks: one is a conventional classification model, and the other is a saliency prediction model trained with human fixations. Features computed with the two subnetworks are trained separately and then fine-tuned jointly using a multiple cross entropy loss. Experimental results show that the new saliency sub-network improves multilabel image classification performance on the MS COCO dataset. The improvement is consistent across various levels of scene clutteredness.
doi:10.1109/cvprw.2019.00110 dblp:conf/cvpr/LuoJZ19 fatcat:75xq6un35vad3dw5o4odosfxaa