Building 3D Representations and Generating Motions From a Single Image via Video-Generation
Wei-Ming Zhi · Ziyong Ma · Tian-Yi Zhang · Matthew Johnson-Roberson
TL;DR
VGER turns one RGB image into a generated camera-motion video, then a 3D point cloud, to produce smooth collision-free robot motions.
BRIEF
A robot that must move safely around obstacles usually needs a 3D model of its surroundings. Getting that from one photo is hard: depth-estimation models produce frustum-shaped errors that make the output unreliable for planning. VGER instead asks a video-generation model to imagine a camera moving through the scene, then feeds those frames to a 3D foundation model to get a dense point cloud. An implicit representation trained with multi-scale noise then guides a motion generator, producing smooth, collision-free motions in indoor and outdoor scenes.
Abstract
Autonomous robots typically need to construct representations of their surroundings and adapt their motions to the geometry of their environment. Here, we tackle the problem of constructing a policy model for collision-free motion generation, consistent with the environment, from a single input RGB image. Extracting 3D structures from a single image often involves monocular depth estimation. De-velopments in depth estimation have given rise to large pre-trained models such as DepthAnything . However, using outputs of these models for downstream motion generation is challenging due to frustum-shaped errors that arise. Instead, we propose a framework known as Video-Generation Environment Representation (VGER), which leverages the advances of large-scale video generation models to generate a moving camera video conditioned on the input image. Frames of this video, which form a multiview dataset, are then input into a pre-trained 3D foundation model to produce a dense point cloud. We then introduce a multi-scale noise approach to train an implicit representation of the environment structure and build a motion generation model that complies with the geometry of the representation. We extensively evaluate VGER over a diverse set of indoor and outdoor environments. We demonstrate its ability to produce smooth motions that account for the captured geometry of a scene, all from a single RGB input image.