Object-Based Coding for Plenoptic Videos

King-To Ng, Qing Wu, Shing-Chow Chan, Heung-Yeung Shum
2010 IEEE transactions on circuits and systems for video technology (Print)  
A new object-based coding system for a class of dynamic image-based representations called plenoptic videos (PVs) is proposed. PVs are simplified dynamic light fields, where the videos are taken at regularly spaced locations along line segments instead of a 2-D plane. In the proposed object-based approach, objects at different depth values are segmented to improve the rendering quality. By encoding PVs at the object level, desirable functionalities such as scalability of contents, error
more » ... ce, and interactivity with an individual image-based rendering (IBR) object can be achieved. Besides supporting the coding of texture and binary shape maps for IBR objects with arbitrary shapes, the proposed system also supports the coding of grayscale alpha maps as well as depth maps (geometry information) to respectively facilitate the matting and rendering of the IBR objects. Both temporal and spatial redundancies among the streams in the PV are exploited to improve the coding performance, while avoiding excessive complexity in selective decoding of PVs to support fast rendering speed. Advanced spatial/temporal prediction methods such as global disparity-compensated prediction, as well as direct prediction and its extensions are developed. The bit allocation and rate control scheme employing a new convex optimization-based approach are also introduced. Experimental results show that considerable improvements in coding performance are obtained for both synthetic and real scenes, while supporting the stated object-based functionalities. Index Terms-IBR object coding, image-based rending, MPEG-4, object-based coding, plenoptic videos. in 1999 and 2001, respectively, and the Ph.D. degree in electrical and electronic engineering from the University of Hong Kong, Pokfulam, Hong Kong, in 2007. Currently, he is a Research Associate with the Department of Electrical and Electronic Engineering, University of Hong Kong. His research interests include image-based rendering, video coding/transmission, and rate control.
doi:10.1109/tcsvt.2010.2041820 fatcat:smm6bc2jyrcevdwt6udqeb6cqq