As-consistent-As-possible compositing of virtual objects and video sequences

Guofeng Zhang, Xueying Qin, Xiaobo An, Wei Chen, Hujun Bao
2006 Computer Animation and Virtual Worlds  
We present an efficient approach that merges the virtual objects into video sequences taken by a freely moving camera in a realistic manner. The composition is visually and geometrically consistent through three main steps. First, a robust camera tracking algorithm based on key frames is proposed, which precisely recovers the focal length with a novel multi-frame strategy. Next, the concerned 3D models of the real scenes are reconstructed by means of an extended multi-baseline algorithm.
more » ... , the virtual objects in the form of 3D models are integrated into the real scenes, with special cares on the interaction consistency including shadow casting, occlusions and object animation. A variety of experiments have been implemented, which demonstrate the robustness and efficiency of our approach. 1 Introduction Over the past decade, Augmented Reality (AR), which aims to merge virtual objects into the real scenes, has become an invaluable technique for a wide variety of applications [2, 15] . Augmented Video is an off-line AR technique for highly demanding applications such as film-making, television and environmental assessments, in which seamless composition is of essential importance. Most previous solutions of AR system concentrate on the geometry consistency of virtual and real scenes and thus require precise motion estimation of video camera and 3D models [4, 8] . The structure and motion recovery is a traditional problem in computer vision [17, 9, 7] . Some commercial software packages have been available, such as 2d3 Boujou [1] and REALVIZ MatchMover [19] . To the best of our knowledge, the detailed techniques used by these packages have not been published yet. Pollefeys et al. [17] proposed to begin with an initialization of projective structure and motion, followed by an upgrade to metric framework with self-calibration. They also employed a flexible multiview stereo matching scheme [10] to obtain a dense estimation of the surface geometry. However, the self-calibration technique is not always stable, especially when the initially recovered projective matrices are not adequately accurate.
doi:10.1002/cav.134 fatcat:zuiarihhnjajta56dmhpyt5ls4