Simultaneous Geometry and Pose Estimation of Held Objects Via 3D Foundation Models
Weiming Zhi · Haozhan Tang · Tianyi Zhang · Matthew Johnson-Roberson
TL;DR
A robot recovers the shape, scale, and pose of a grasped object from uncalibrated RGB images by aligning 3D foundation model estimates to its own frame.
BRIEF
Robots that use a held object as a tool need to know where that object is and how big it is relative to the gripper. This work estimates the shape and pose of a grasped object from RGB images taken by an outside camera whose position has not been calibrated. A 3D foundation model gives a first shape estimate, but its scale is wrong and it sits in the camera's frame. A coordinate-alignment problem then recovers true scale and moves the shape into the robot's frame. Forward kinematics can then track points on the object as the arm moves. Tests used a manipulator holding a range of real objects.
Abstract
Humans have the remarkable ability to use held objects as tools to interact with their environment. Humans internally estimate how hand movements affect the object's movement. We wish to endow robots with this capability. We contribute methodology to jointly estimate the geometry and pose of objects grasped by a robot, from RGB images captured by an external camera. Notably, our method transforms the estimated geometry into the robot's coordinate frame, while not requiring the extrinsic parameters of the external camera to be calibrated. Our approach leverages 3D foundation models, large models pre-trained on huge datasets for 3D vision tasks, to produce initial estimates of the in-hand object. These initial estimations do not have physically correct scales and are in the camera's frame. Then, we formulate, and efficiently solve, a coordinate-alignment problem to recover accurate scales, along with a transformation of the objects to the coordinate frame of the robot. Forward kinematics mappings can subsequently be defined from the manipulator's joint angles to specified points on the object. These mappings enable the estimation of points on the held object at arbitrary configurations, enabling robot motion to be designed with respect to coordinates on the grasped objects. We empirically evaluate our approach on a manipulator holding a diverse set of real-world objects.