Joint Flow Trajectory Optimization For Feasible Robot Motion Generation from Video Demonstrations

Xiaoxiang Dong · Matthew Johnson-Roberson · Weiming Zhi

Fig. 3: We present an overview of the Joint Flow Trajectory Optimization (JFTO) framework. Human demonstration videos are processed to extract the grasp pose and object trajectory. These are then used to jointly optimize a feasible executio — FIGURE FROM THE PAPER

TL;DR

A framework turns human videos into feasible robot grasps and object trajectories by jointly optimizing grasp, motion likelihood, and collision avoidance.

BRIEF

Human videos are a cheap way to teach robots, but a robot arm cannot copy a human hand directly. Its joints have limits, its gripper differs from a hand, and it must avoid collisions. This work treats the demonstration as a guide for how the object moves, not how the hand moves. It models object motion with flow matching on SE(3), which keeps multiple valid ways of doing a task. A single differentiable objective then combines grasp similarity, trajectory likelihood, and collision penalties. Tests were run in simulation and on real manipulation tasks.

Abstract

Learning from human video demonstrations offers a scalable alternative to teleoperation or kinesthetic teaching, but poses challenges for robot manipulators due to embodiment differences and joint feasibility constraints. We address this problem by proposing the Joint Flow Trajectory Optimization (JFTO) framework for grasp pose generation and object trajectory imitation under the video-based Learning-from-Demonstration (LfD) paradigm. Rather than directly imitating human hand motions, our method treats demonstrations as object-centric guides, balancing three objectives: (i) selecting a feasible grasp pose, (ii) generating object trajectories consistent with demonstrated motions, and (iii) ensuring collision-free execution within robot kinematics. To capture the multimodal nature of demonstrations, we extend flow matching to $\SE(3)$ for probabilistic modeling of object trajectories, enabling density-aware imitation that avoids mode collapse. The resulting optimization integrates grasp similarity, trajectory likelihood, and collision penalties into a unified differentiable objective. We validate our approach in both simulation and real-world experiments across diverse real-world manipulation tasks.