Increasing Video Perceptual Quality with GANs and Semantic Coding

Leonardo Galteri, Marco Bertini, Lorenzo Seidenari, Tiberio Uricchio, Alberto Del Bimbo
2020 Proceedings of the 28th ACM International Conference on Multimedia  
Receiver Semantic coding Face parsing Artifact Removal original semantic mask adaptive compression Transmission final restored Transmitter adaptive compression Figure 1: The system comprised of the transmitter and receiver. The transmitter detects the parts of the image which are important to the perception of a viewer and assign more bandwidth to them. The receiver runs the artifact removal algorithm which recover the missing details accordingly. ABSTRACT We have seen a rise in video based
more » ... communication in the last year, unfortunately fueled by the spread of COVID-19 disease. Efficient low-latency delay of transmission of video is a challenging problem which must also deal with the segmented nature of network infrastructure not always allowing a high throughput. Lossy video compression is a basic requirement to enable such technology widely. While this may compromise the quality of the streamed video there are recent deep learning based solutions to restore quality of a lossy compressed video. Considering the very nature of video conferencing, bitrate allocation in video streaming could be driven semantically, differentiating quality between the talking subjects and the background. Currently there have not been any work studying the restoration of semantically coded video using deep learning. In this work we show how such videos can be efficiently generated by shifting bitrate with masks derived via computer vision and how a deep generative adversarial network can be trained to restore video quality. Our study shows that the combination of semantic coding and learning based video restoration can provide superior results. KEYWORDS semantic video compression, GANs, video quality enhancement ACM Reference Format:
doi:10.1145/3394171.3413508 dblp:conf/mm/GalteriBSUB20 fatcat:vmtyaf7aojbhfbaic7777zmcce