Object Knowledge Integration for Skeleton-Based Action Recognition

Abstract

3D-Skeleton-based action recognition has been widely adopted due to its efficiency and robustness to complex backgrounds. While it is capable of conveying a significant amount of information regarding the dynamics of human poses, we argue that its performance is curtailed when confronted with actions involving interactions between humans and objects due to the absence of the study of the surrounding objects. It is of great importance to delve deeper into the study of human-object interactions for skeleton-based action recognition. This paper proposes a novel approach to represent the spatial-temporal skeleton features, along with the present nearby objects and their dynamics. To accomplish this, a new formulation named object knowledge is presented, which entails the categorization of object characteristics, based on whether or not the object necessitates a motion analysis. With a piece of prior knowledge, in cases where it is required, the motion is calculated, while for those where it is not necessary, only the category of object is considered. This object knowledge is then early-fusion along with the skeleton representation, in such a way that it fits into the self-attention model. The experimental results on different popular action recognition datasets (NTU RGB+D 60, NTU RGB+ D 120 illustrate that the proposed approach outperforms the current state-of-the-art methods.