REPLAY: Robot Embodiment via Intent-aware Policy Imitation by Replicating Human Demonstrations From Video

초록

Learning manipulation skills from human demonstrations traditionally relies on structured datasets, teleoperations, or environment-specific supervision, limiting scalability and generalization. In this paper, we introduce REPLAY, a novel imitation learning framework that enables robots to acquire intent-aware behaviors directly from raw monocular videos, including unstructured sources such as YouTube. REPLAY decomposes demonstration videos into a sequence of semantically meaningful sub-tasks through action segmentation, scene and task reasoning via vision-language models, and fine-grained action understanding using 3D human pose estimation. These extracted trajectories are then retargeted to robot embodiments through intent-aware motion adaptation that accounts for embodiment differences and environmental constraints. To support scalable evaluation, we also present Video2Sim, a method that reconstructs realistic 3D simulation environments directly from demonstration videos, enabling repeatable testing and training. We demonstrated that REPLAY outperformed strong baselines in both motion fidelity and task success on complex object manipulation tasks, illustrating its potential as a scalable, generalizable approach for real-world imitation learning from in-the-wild video data.

키워드