Learning-based image super-resolution is considered as a promising solution to reconstruct a high-resolution image from a low-resolution image. To improve the super-resolution performance dramatically, this Letter focuses on the effect of training dataset on the performance and proposes an image super-resolution scheme based on patch orientation-specified network. In particular, a deep neural network is trained using patches with a specific orientation and angular transformation is combined

more »

... the neural network to cope with various orientations in input patches. Experimental results show the suggested network model is superior to existing state-of-the-art super-resolution alternatives. Introduction: Since patch orientation can serve as an important clue for recovering the missing high-frequency (HF) components due to image downscaling, it is useful in electronic displays for accurate image super-resolution (SR). In general, existing learning-based image SR techniques [1-7] tend to utilise public training datasets consisting of image patches with various orientations to keep the robustness of the SR performance to the arbitrary direction of input low-resolution (LR) patches. In contrast to this trend, in this Letter, we notice that building a trained model based on patch datasets with a specific orientation can provide a significant improvement on input LR patches with the specific orientation. As shown in Fig. 1 , to confirm the effect of patch orientation in the training dataset on the SR performance, we perform an investigation by applying an existing state-of-the-art SR algorithm [2] to a LR image having only unidirectional edges. In the investigation, the network 1 model is obtained by training the neural network parameters using general patch datasets with various orientations. Unlike this, the network 2 model is obtained using training patches with a specific orientation only, such as vertical edges. Given a LR image with vertical edges only, we perform the upscaling with a scale factor 3 by using the two trained models, the network 1 and network 2 , respectively. The upscaled images in Fig. 1 show that the orientation-specified network (or network 1 ) outperforms the general non-specified network (or network 2 ) for the LR image by significantly improving a peak signal-to-noise ratio (PSNR) value by 3.77 dB. Based on the investigation, we consider that the network specialised to the input orientation is needed for the SR performance increase. Meanwhile, by re-training model parameters in recent image SR algorithms [1-6] based on convolutional neural network for each orientation separately, it may be possible to achieve the SR performance comparable to that expected from our investigation. However, as expected, it requires a high computational complexity for the training procedure as well as a huge amount of memories for many models for all orientations. To address these problems, in this Letter, we suggest an orientation matching based on angular transformation in the same direction as the dataset and combine it with a neural network particularly designed for accurate image SR.