WiWeHAR: Multimodal Human Activity Recognition Using Wi-Fi and Wearable Sensing Modalities

Muhammad Muaaz, Ali Chelli, Ahmed Abdelgawwad, Andreu Catala Mallofre, Matthias Patzold
2020 IEEE Access  
Robust and accurate human activity recognition (HAR) systems are essential to many humancentric services within active assisted living and healthcare facilities. Traditional HAR systems mostly leverage a single sensing modality (e.g., either wearable, vision, or radio frequency sensing) combined with machine learning techniques to recognize human activities. Such unimodal HAR systems do not cope well with real-time changes in the environment. To overcome this limitation, new HAR systems that
more » ... HAR systems that incorporate multiple sensing modalities are needed. Multiple diverse sensors can provide more accurate and complete information resulting in better recognition of the performed activities. This article presents WiWeHAR-a multimodal HAR system that uses combined Wi-Fi and wearable sensing modalities to simultaneously sense the performed activities. WiWeHAR makes use of standard Wi-Fi network interface cards to collect the channel state information (CSI) and a wearable inertial measurement unit (IMU) consisting of accelerometer, gyroscope, magnetometer sensors to collect the user's local body movements. We compute the time-variant mean Doppler shift (MDS) from the processed CSI data and magnitude from the inertial data for each sensor of the IMU. Thereafter, we separately extract various time-and frequencydomain features from the magnitude data and the MDS. We apply feature-level fusion to combine the extracted features, and finally supervised learning techniques are used to recognize the performed activities. We evaluate the performance of WiWeHAR by using a multimodal human activity data set, which was obtained from 9 participants. Each participant carried out four activities, such as walking, falling, sitting, and picking up an object from the floor. Our results indicate that the proposed multimodal WiWeHAR system outperforms the unimodal CSI, accelerometer, gyroscope, and magnetometer HAR systems and achieves an overall recognition accuracy of 99.6%-100%. INDEX TERMS Activity recognition, Doppler effect, feature extraction, feature fusion, machine learning, micro-Doppler signature, principal component analysis, radio frequency sensing, wearable sensing.
doi:10.1109/access.2020.3022287 fatcat:isqx42mb4redxcxer4cm4dotbq