Unified Self-supervised Representation Learning for Multi-modal and Single-modal 3D Perception
Xiao-Hao Xu · Ye Li · Tian-Yi Zhang · Jin-Rong Yang · Matthew Johnson-Roberson · Xiao-Nan Huang
TL;DR
NS-MAE is a single self-supervised pretraining method for LiDAR and camera data that uses NeRF-style rendering to improve driving perception with few labels.
BRIEF
Self-driving systems rely on LiDAR and cameras, but labeling enough data to train perception models for both is expensive. Existing self-supervised pretraining usually treats each sensor separately. NS-MAE takes a unified approach inspired by NeRF: it hides parts of LiDAR and image inputs, encodes appearance and geometry, and renders features in perspective and bird's-eye views to reconstruct the original data. It transfers well across 3D perception tasks and does especially well on BEV map segmentation with limited labels.
Abstract
Constructing large-scale labeled datasets for multi-modal perception in autonomous driving is challenging, driving the need for self-supervised pretraining. However, existing methods typically adopt separate pretraining strategies for each modality. Inspired by Neural Radiance Field (NeRF), which unifies multi-modal optimization, we propose NeRF-Supervised Masked Autoencoder (NS-MAE), a unified pretraining strategy for LiDAR-Camera 3D perception. NS-MAE encodes both appearance and geometry, enabling efficient masked reconstruction across modalities. Specifically, it extracts embeddings from corrupted LiDAR point clouds and images, conditioned on view direction and location. These embeddings are then rendered into multi-modal feature maps from two key perspectives for autonomous driving: perspective view and bird’s-eye view (BEV). The original uncorrupted data serve as reconstruction targets, facilitating self-supervised learning. Extensive experiments demonstrate NS-MAE’s superior transferability across various 3D perception tasks. Notably, NS-MAE outperforms prior SOTA pretraining methods which rely on separate modality-specific strategies, particularly in BEV map segmentation under label-efficient fine-tuning. Our code is publicly available at https://github.com/Xiaohao-Xu/Unified-Pretrain-AD/.