A Survey on Mobile Social Signal Processing
ACM Computing Surveys
Understanding human behaviour in an automatic but non-intrusive manner is an important area for various applications. This requires the collaboration of information technology with human sciences to transfer existing knowledge of human behaviour into self-acting tools. These tools will reduce human error that is introduced by current obtrusive methods such as questionnaires. To achieve unobtrusiveness, we focus on exploiting the pervasive and ubiquitous character of mobile devices. In this
... le, a survey of existing techniques for extracting social behaviour through mobile devices is provided. Initially we expose the terminology used in the area and introduce a concrete architecture for social signal processing applications on mobile phones, constituted by sensing, social interaction detection, behavioural cues extraction, social signal inference and social behaviour understanding. Furthermore, we present state-of-the-art techniques applied to each stage of the process. Finally, potential applications are shown while arguing about the main challenges of the area. Page 3 of 47 Computing Surveys https://mc.manuscriptcentral.com/csur A:4 N. Palaghias et al. Preprocessing Storage Microphone Camera Bluetooth Cell Tower Signals Accelerometer Gyroscope Mobile'Device' Sensors' Preprocessing Inference WiFi GPS Magnetometer Call Info SMS Info Phone Usage Fig. 2: Application architecture of existing Sensing Frameworks. interactions may also be utilised for filtering data and allowing the development of adaptive sensing and inference techniques. In applications focusing on extracting behavioural information not related to the social aspect of a person, it is strongly encouraged to include this step as it provides important contextual information. Following the identification of on-going social interactions is the extraction of behavioural cues. Different modalities may be leveraged for the extraction of a behavioural cue, depending on the grammar defined in psychology. Each selected sensed modality is forwarded to behavioural cues extraction. Existing literature has been classified into seven categories based on the types of cues each work extracts (See Fig. 1) . The behavioural cues extraction is achieved through techniques such as decision models, statistical analysis etc. The final stage of Mobile SSP is the transition from the understanding of social signals to social behaviour inference. Close collaboration with social sciences may provide the theoretical mapping among behavioural cues, social signals and social behaviours. Literature has been grouped based on the inferred social behaviour through mobile phones. The extracted behavioural cues are fed in decision making techniques to mine social signals and infer in long-term social behaviour. To facilitate the reader's understanding of the field, we provide an outline of the main steps and requirements for an integrated and real-world-enabled Mobile SSP: -Define the context of the Mobile SSP application. -Select the modalities required to infer a particular social behaviour. -Define the grammar of behavioural cues and social signals that will lead to social behaviour inference. -Evaluate and verify the reliability of the approach in a real-world environment based on ground truth. In addition to the above requirements, researchers need to consider the intrusiveness, security and privacy of the system. Researchers need to take into account the computational burden and energy consumption which may endanger user experience. These parameters do not constitute a prerequisite for the realisation of Mobile SSP but will facilitate user experience and privacy. In the following sections, each of the pre-defined stages will be analysed and state-of-the-art research are outlined. The works described in the next sections are summarised in the Electronic Appendix, introducing the techniques developed in each stage of social behaviour inference. SENSING FRAMEWORKS Sensing is the first stage in extracting human behaviour on mobile devices. In this stage, selection of appropriate modalities is performed. These will later on be processed and analysed to reveal information about user's social behaviour. It constitutes the lowest level of the process, which collects raw data from sensors and other interfaces that can provide information relevant to the user (See Fig. 2) . After retrieving information from sensors either the raw data are forwarded to the next stages or lightweight and simplistic processing may be performed to minimise the complexity and computational burden at the upcoming stages. As shown in Fig. 1 , the next stages in social behaviour inference may be performed either on the device or at a backend server. This section introduces and then compares existing sensing frameworks. Through this introduction, the reader should be able to understand the criteria based on which sensing framework should be selected for a desirable social behaviour application. An extensive analysis of existing sensing frameworks is outside the scope of this article and the reader is referred to and ].