Efficient 3D Human Body Reconstruction from Monocular Video with Depth-Guided Learning

초록

Numerous opportunities exist for monocular 3D human body reconstructions in VR and AR applications. However, existing methods often struggle to convey detailed information such as hair or clothing, or they require significant computational resources to generate a comprehensive 3D shape. To address the challenge of achieving efficient 3D human body reconstruction from a monocular video with near real-time speed, we propose a method that overcomes the above limitations. Our approach involves an efficient and stable pipeline leveraging network distillation and GPGPU for real-time 3D human body reconstruction from a monocular video. The proposed method consists of multiple stages, including depth-based model learning, RGB-based model learning, and real-time 3D body reconstruction. In the depth-based model learning step, the model receives a 2D depth frame and reconstructs a 3D human shape. Afterward, we employ knowledge distillation on the feature map from the pre-trained depth-based model onto a new network trained to reconstruct a 3D human body using RGB frames. Finally, GPGPU-based post-processing improves the runtime and achieves near real-time speed for 3D human body reconstruction. Experimental results show that the proposed method achieves better qualitative and quantitative results compared to the baseline, without increasing inference time. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.

키워드