GraphSeg: Segmented 3D Representations via Graph Edge Addition and Contraction

Haozhan Tang · Tianyi Zhang · Oliver Kroemer · Matthew Johnson-Roberson · Weiming Zhi

Fig. 1: Robots often need to work with object-level representations. In this work, we tackle the problem of building segmented 3D scenes from a set of images, and introduce the GraphSeg framework. GraphSeg solves multi-view 3D segmentation — FIGURE FROM THE PAPER

TL;DR

GraphSeg merges over-segmented 2D masks from a few RGB images into consistent 3D object segments using graph edge addition and contraction.

BRIEF

Robots that work in cluttered spaces need to know which points in a scene belong to which object. 2D segmentation models such as SAM often split one object into many pieces and label the same object inconsistently across camera views. GraphSeg builds two correspondence graphs from a few RGB images, one from pixel-level similarity and one from inferred 3D structure. It adds edges between matching masks, then contracts the graph to merge them into object-level segments. 3D foundation models then produce a segmented 3D representation, which improved downstream manipulation.

Abstract

Robots operating in unstructured environments often require accurate and consistent object-level representations. This typically requires segmenting individual objects from the robot's surroundings. While recent large models such as Segment Anything (SAM) offer strong performance in 2D image segmentation. These advances do not translate directly to performance in the physical 3D world, where they often over-segment objects and fail to produce consistent mask correspondences across views. In this paper, we present GraphSeg, a framework for generating consistent 3D object segmentations from a sparse set of 2D images of the environment without any depth information. GraphSeg adds edges to graphs and constructs dual correspondence graphs: one from 2D pixel-level similarities and one from inferred 3D structure. We formulate segmentation as a problem of edge addition, then subsequent graph contraction, which merges multiple 2D masks into unified object-level segmentations. We can then leverage 3D foundation models to produce segmented 3D representations. GraphSeg achieves robust segmentation with significantly fewer images and greater accuracy than prior methods. We demonstrate state-of-the-art performance on tabletop scenes and show that GraphSeg enables improved performance on downstream robotic manipulation tasks. Code available at https://github.com/tomtang502/graphseg.git.