RPFNET: COMPLEMENTARY FEATURE FUSION FOR HAND GESTURE RECOGNITION

초록

Hand gesture recognition (HGR) is one of the most challenging tasks because it is very sensitive to occlusion or background. Various modalities such as RGB, depth, and point cloud as well as their combinations have been proposed to improve the performance of HGR, but the fusion of RGB and point cloud with complementary characteristics has never been attempted. This paper analyzes the synergistic effect of the two complementary modalities, and then proposes a new multi-modal fusion network that quantifies and converges the mutual influence of two modalities. Also, to overcome the inherent limitation that the predicted mutual influence does not match the actual one, we propose the self-labeling-based adaptive guidance. Experimental results show that the proposed method achieved 2.46% higher performance than the SOTA method in the case of the NVGesture dataset.

키워드