Bi-Manual Joint Camera Calibration and Scene Representation

Haozhan Tang · Tianyi Zhang · Matthew Johnson-Roberson · Weiming Zhi

Fig. 1: We tackle a bi-manual setup, where the extrinsics of both cameras and the relative poses of the robot bases to one another are unknown. Bi-JCR solves to recover all three transformations. — FIGURE FROM THE PAPER

TL;DR

Bi-JCR calibrates cameras on two robot arms and builds a shared 3D scene model from ordinary images, with no calibration markers.

BRIEF

Robots with cameras on their arms need to know where each camera sits relative to the gripper, and, with two arms, where the arms sit relative to each other. The usual fix is to photograph a printed calibration marker many times. Bi-JCR skips the marker. It uses 3D foundation models to find correspondences across images from both arms, then estimates both camera-to-gripper transforms, the pose between the two arms, and a shared 3D model of the workspace. That model supports collision checking and segmentation.

Abstract

Robot manipulation, especially bimanual manipulation, often requires setting up multiple cameras on multiple robot manipulators. Before robot manipulators can generate motion or even build representations of their environments, the cameras rigidly mounted to the robot need to be calibrated. Camera calibration is a cumbersome process involving collecting a set of images, with each capturing a pre-determined marker. In this work, we introduce the Bi-Manual Joint Calibration and Representation Framework (Bi-JCR). Bi-JCR enables multiple robot manipulators, each with cameras mounted, to circumvent taking images of calibration markers. By leveraging 3D foundation models for dense, marker-free multi-view correspondence, Bi-JCR jointly estimates: (i) the extrinsic transformation from each camera to its end-effector, (ii) the inter-arm relative poses between manipulators, and (iii) a unified, scale-consistent 3D representation of the shared workspace, all from the same captured RGB image sets. The representation, jointly constructed from images captured by cameras on both manipulators, lives in a common coordinate frame and supports collision checking and semantic segmentation to facilitate downstream bimanual coordination tasks. We empirically evaluate the robustness of Bi-JCR on a variety of tabletop environments, and demonstrate its applicability on a variety of downstream tasks.