Evolution of temporal multimedia synchronization principles

Zixia Huang, Klara Nahrstedt, Ralf Steinmetz
2013 ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP)  
The evolution of multimedia applications has drastically changed human life and behaviors. New communication technologies lead to new requirements for multimedia synchronization. This article presents a historical view of temporal synchronization studies focusing on continuous multimedia. We demonstrate how the development of multimedia systems has created new challenges for synchronization technologies. We conclude with a new application-dependent, multilocation, multirequirement
more » ... n framework to address these new challenges. 34:2 • Z. Huang et al. media events (e.g., image pop-up or text animation). Synchronization of discrete multimedia may come with a coarse granularity where only the temporal order needs to be preserved. Hence, it is also called event synchronization. There have been numerous synchronization papers for both continuous and discrete multimedia [Boronat et al. 2009; Cronin et al. 2004; Buchanan and Zellweger 2005] . Due to space limitations, we only investigate the continuous case. The configuration of a continuous multimedia application can be represented in multiple forms of media components (Section 2), where each component requires different temporal synchronization and triggers diverse user interests. However, the time dependencies of these media components when they are captured by the media sensors may lose track in multiple locations during media computation and distribution, due to variations in computation demands and transmission overhead (over the Internet and transport protocols). A synchronization error in one location can be propagated to future locations. In addition, a single multimedia platform may serve multiple application functionalities, so users can exhibit interests in different synchronization requirements. A two-lens stereo camera system with internal mono microphone is a good example. If it is used for 2D video conferencing, people are only interested in synchronization between the audio and one of its lens. But for 3D depth computation, synchronization between the two lens is more important. As next-generation multimedia applications are growing more complex in terms of hardware configurations, more diverse in terms of application functionalities, and more expensive in terms of consumptions of computation and network resources, preserving time correlations of media data in each application location is difficult. A systematic framework is needed to integrate application-dependent, multilocation, multirequirement synchronization problems in order to achieve their final in-sync presentation at the media outputs. We will show that such a framework is unfortunately missing in existing systems. This article presents a historical view of synchronization studies for continuous multimedia over the past 30 years. Based on synchronization formulations (Section 2), we demonstrate how the development of multimedia systems has created new challenges for synchronization technologies (Section 3). We conclude with a new multidimensional synchronization framework to address these challenges (Section 4). Surveys in Boronat et al. [2009] and Ishibashi and Tasaka [2000] have considered existing continuous multimedia synchronization results extensively, but mainly focused on comparing and evaluating the functionalities and methodologies of control algorithms. While we will use both surveys as a starting point, we will evaluate multimedia synchronization advancements from a completely different point of view. We clearly convey multimedia technological backgrounds and their historical roles in synchronization modeling, protocols, and human perceptual evaluation. Furthermore, we make an argument, that there is an urgent need for the research community to further evolve and advance existing synchronization practices and standards in the setting of next-generation multimedia applications. These synchronization complexities arise from the inclusion of more advanced computing and communication technologies. SYNCHRONIZATION FORMULATION Before the discussion of existing literature, we formulate the term synchronization. We present the mathematical model to facilitate our description in this article. Continuous Multimedia Data Model The architecture of a continuous multimedia data model can be described in a hierarchical fashion. -Session. A session describes the multimedia communications between two or more sites for a shared collaboration. In this article, we use {n 1 , . . . , n N } to denote N sites within the same session.
doi:10.1145/2490821 fatcat:sy2hcd5oybeqrpzrjacehsnudq