Scalable Benchmarking and Robust Learning for Noise-Free Ego-Motion and 3D Reconstruction from Noisy Video

Xiaohao Xu · Tianyi Zhang · Shibo Zhao · Xiang Li · Sibo Wang · Yongqi Chen · Ye Li · Bhiksha Raj · Matthew Johnson-Roberson · Sebastian Scherer · Xiaonan Huang

Figure 1: Towards robust ego-motion and photorealistic 3D reconstruction. (a) Previous approaches rely on synthetic datasets with perfect conditions (noise-free and smooth motion). (b) Real-world data introduce inherent noise and complexiti — FIGURE FROM THE PAPER

TL;DR

A noise-synthesis pipeline, the Robust-Ego3D benchmark, and CorrGS test-time adaptation make ego-motion and 3D reconstruction hold up on noisy video.

BRIEF

Ego-motion and 3D reconstruction models are usually tested on clean, smooth video, so they often degrade sharply on real footage with fast motion, sensor flaws, and timing errors. This work builds a scalable pipeline that synthesizes such noise, and uses it to create Robust-Ego3D, a benchmark that exposes where current learning-based methods break down. It also proposes CorrGS, a test-time adaptation method that keeps an internal clean 3D Gaussian representation and aligns noisy frames to its renderings using visual correspondence. It outperforms prior methods, especially under rapid motion and changing illumination.

Abstract

We aim to redefine robust ego-motion estimation and photorealistic 3D reconstruction by addressing a critical limitation: the reliance on noise-free data in existing models. While such sanitized conditions simplify evaluation, they fail to capture the unpredictable, noisy complexities of real-world environments. Dynamic motion, sensor imperfections, and synchronization perturbations lead to sharp performance declines when these models are deployed in practice, revealing an urgent need for frameworks that embrace and excel under real-world noise. To bridge this gap, we tackle three core challenges: scalable data generation, comprehensive benchmarking, and model robustness enhancement. First, we introduce a scalable noisy data synthesis pipeline that generates diverse datasets simulating complex motion, sensor imperfections, and synchronization errors. Second, we leverage this pipeline to create Robust-Ego3D, a benchmark rigorously designed to expose noise-induced performance degradation, highlighting the limitations of current learning-based methods in ego-motion accuracy and 3D reconstruction quality. Third, we propose Correspondence-guided Gaussian Splatting (CorrGS), a novel test-time adaptation method that progressively refines an internal clean 3D representation by aligning noisy observations with rendered RGB-D frames from clean 3D map, enhancing geometric alignment and appearance restoration through visual correspondence. Extensive experiments on synthetic and real-world data demonstrate that CorrGS consistently outperforms prior state-of-the-art methods, particularly in scenarios involving rapid motion and dynamic illumination.