SemiMultiPose: A Semi-supervised Multi-animal Pose Estimation Framework [article]

Ari Blau, Christoph Gebhardt, Andres Bendesky, Liam Paninski, Anqi Wu
2022 arXiv   pre-print
Multi-animal pose estimation is essential for studying animals' social behaviors in neuroscience and neuroethology. Advanced approaches have been proposed to support multi-animal estimation and achieve state-of-the-art performance. However, these models rarely exploit unlabeled data during training even though real world applications have exponentially more unlabeled frames than labeled frames. Manually adding dense annotations for a large number of images or videos is costly and
more » ... , especially for multiple instances. Given these deficiencies, we propose a novel semi-supervised architecture for multi-animal pose estimation, leveraging the abundant structures pervasive in unlabeled frames in behavior videos to enhance training, which is critical for sparsely-labeled problems. The resulting algorithm will provide superior multi-animal pose estimation results on three animal experiments compared to the state-of-the-art baseline and exhibits more predictive power in sparsely-labeled data regimes.
arXiv:2204.07072v1 fatcat:c5npnbrqb5ggzfgqs3fmnqjswu