Deep Learning for Digitizing, Analysing and Modelling Choreographic 3D Signal Sequences of Intangible Cultural Heritage [article]

(:Unkn) Unknown, National Technological University Of Athens
2022
In performing arts, such as choreography, dance and theatrical kinesiology, movements of human body signals and gestures are essential elements used to describe a storyline in an aesthetic and symbolic way. Although, we, as humans, can inherently perceive and decipher such human body signals in a natural way, this process is challenging for a computer system. One important aspect in the analysis of a performing dance is the automatic extraction of the choreographic patterns/elements since these
more » ... elements provide an abstract and compact representation of the semantic information encoded in the overall dance storyline. One salient issue in the analysis of a performing dance is to automatic extract its choreographic patterns since these elements provide an abstract representation of the semantics of the dance and encodes the overall dance storytelling. However, application of conventional video summarization algorithms on dance sequences cannot appropriately retrieve their choreographic patterns since a dance is composed of an ordered set of sequential elements which are repeated in time. Additionally, the 3D geometry of a dance is too complicated to be described using only the RGB color information. This thesis is distinguished into three parts. Part I describes the theoretical background regarding ICH and the principles with respect to the mathematical modelling of folklore choreographic sequences. In Chapters 1, 2, 3 the recent trends on choreographic representation in terms of machine learning, video summarization, pose identification and dance annotation are described. Part II presents the adopted techniques for content-based sampling of the selected folklore choreographic sequences. This part is oriented on the semantic compression and the video summarization taking into consideration the complexity of the spatio-temporal sequences. In particular, Chapter 4 exploited a hierarchical scheme that implements spatio-temporal variations of the dance features. Chapter 5 describes an abstract representation of the semantic details of choreographic sequences taking into consideration a key-frame selection algorithm. Chapter 6 compares the summarization performances taking into account four sampling algorithms all implemented under a SAE scheme's projected data. Specifically, a SAE framework followed by a hierarchical SMRS algorithm implemented to summarize choreographic sequences. Part III (Chapters 7, 8, 9) focused on modelling and analysis of folklore choreographic sequences. Chapter 7 explored the feasibility of pattern matching between heterogeneous motion capturing systems. In this chapter, a trajectory interpretation in folklore sequences is described. The conducted experiments indicate that if significant levels of precision are ensured during initial data collection, design, development and fine-tuning of the system, then low-cost and widely popular motion capturing sensors suffice to provide a smooth and integrated experience on the user end, which would allow for relevant educational or entertainment applications to be adopted at scale. Chapter 8 focuses on the enhancement of the learning experience of folklore dances by introducing machine learning tools with the capability of providing a scalable quantifiable assessment of a choreography at different level of vi hierarchies; yielding a from coarse to fine evaluation. Chapter 9 describes an adaptable autoregressive and moving average layer (R-ARMA) into a conventional CNN filter to model the dynamic behavior of a choreography. In addition, to face the choreography dynamics, we introduced an adaptation mechanisms in a way that the network weights of the fully connected hidden layer is dynamically updated to fit current environmental characteristics. Experimental results on real-life sequences indicated the efficiency of the proposed model against conventional deep machine learning filters. Chapter 10 summarizes the thesis by representing the overall contribution and the future works. I would like to express my gratitude to the people who have supported me during the elaboration of this PhD thesis. First and foremost to three advisors ; Nikolaos Doulamis, for his outstanding ethos and for believing in me and encouraging me since my early steps on research on 2017 with his relentless optimism, inspiration, friendship and collaboration. Andreas Georgopoulos except of being an important pillar of this PhD and my research career so far, shared with me his principles and ethics in research, shaping my research and personal identity; Nikolaos Grammalidis for his outstanding contribution to the digitization of the Intangible Cultural Heritage, for the valuable discussions on the problems and the solutions found during this dissertation, for introducing me to choreographic datasets and motion capturing systems; Anastasios Doulamis for introducing me to the fundamental principles of Machine Learning and Intangible Cultural Heritage domains during our collaboration in the context of the H2020 projects. Anastasios gave me endless research freedom and believe within the setting of those projects, offer assistance, comments and proposals and was effectively included in most of the parts of this dissertation. At the same time, I would like to thank Athanasios Voulodimos for the moral, psychological, professional support, he generously showed me throughout this path. I will never forget that Athanasios Voulodimos was the man who urged me to pursue doctoral studies. Big Thanks to Eftychios Protopapadakis, Nikolaos Bakalos, Maria Kaselimi, George Kopsiaftis for the fruitful collaboration and support. The excellent participation between all the individuals of the counseling committee served as a steady inspiration for advancement, setting a exceptional establishment for my professional career. I would like to thank the rest of the individuals of the Photogrammetry Lab. Moreover, I would like to thank Ioannis Papadonikolakis for his valuable support, his optimism as well as his valuable advice on multiple levels. Last but not least, I would like to thank my family for supporting me all these years. Contents
doi:10.26240/heal.ntua.22018 fatcat:h336uc75t5ebbmabqmktylk5sm