Computational Photography for Efficient Image and Video Acquisition

Bo Zhang
Image sensors have become ubiquitous because of the increasing need for entertainment demand, mobile applications, Internet of things (IoTs) and auto-driving vehicles. In addition to the requirement for improved image quality, the power consumption is also becoming a key design factor especially for the wireless sensor networks (WSNs) where sensors need to be deployed at large scale. Substantial research work has already been proposed for low power CMOS image sensors. Unlike previous work that
more » ... ptimizes the power consumption through circuitry techniques, this dissertation rethinks the imaging system and introduces methods to achieve extremely low power image acquisition through computational photography techniques. First, we propose a lossy image compression algorithm called Microshift which achieves state-of-the-art on-chip compression performance while preserving hardware friendliness. To implement this algorithm, we propose a hardware architecture and validate it on FPGA. The results on the ASIC design further validates the power efficiency. The sensor achieves power as low as 59.7 pJ/(pixel frame) while running on 1530 frames per second. To enable high-performance decompression, we propose Markov random field method which provides PSNR > 34dB for a 1.25bit/pixel image. Second, we propose DenResUnet to enhance the bit-depth information so that ADCs for the image sensor quantize fewer bits. The DenResUnet adopts extensive residual learning structure, which greatly improves the perceptual visual quality. Furthermore, we develop an extension which decompresses the Microshift images in real-time. Extensive experiments demonstrate that high-quality results can be obtained even from 1 bit/pixel images. Third, we propose to adaptively change the sensor sampling rate for aggressive power saving and interpolate the intermediate frames computationally. We propose to establish the dense correspondence between two frames through halfway domain optimization. To account for large displacement, sparse correspondence is jo [...]
doi:10.6084/m9.figshare.14452980.v1 fatcat:swxh7wm63bbhlcfoa4e35htmny