Implicitly using Human Skeleton in Self-supervised Learning: Influence on Spatio-temporal Puzzle Solving and on Video Action Recognition

Mathieu Riand (1, 2) , Laurent Dollé (2) , Patrick Le Callet (1)

1 LS2N - équipe IPI - Image Perception Interaction
2 DP2L - CEA Tech Pays-de-la-Loire

In this paper we studied the influence of adding skeleton data on top of human actions videos when performing self-supervised learning and action recognition. We show that adding this information without additional constraints actually hurts the accuracy of the network; we argue that the added skeleton is not considered by the network and seen as a noise masking part of the natural image. We bring first results on puzzle solving and video action recognition to support this hypothesis.

Mots clés: Self-supervised Learning; Siamese Network; Skeleton Keypoints; Action Recognition; Few-shot Learning

Domaines: Intelligence artificielle [cs.AI]

Mathieu Riand, Laurent Dollé, Patrick Le Callet. Implicitly using Human Skeleton in Self-supervised Learning: Influence on Spatio-temporal Puzzle Solving and on Video Action Recognition. ROBOVIS 2021 : 2nd International Conference on Robotics, Computer Vision and Intelligent Systems, Oct 2021, Online streaming, France. ⟨10.5220/0010689500003061⟩. ⟨hal-03946524⟩