Sensor Transfer: Learning Optimal Sensor Effect Image Augmentation for Sim-to-Real Domain Adaptation

Alexandra Carlson · Katherine A. Skinner · Ram Vasudevan · Matthew Johnson-Roberson

Fig. 2: The schematic of the proposed sensor transfer network structure. The style loss trains the sensor effect parameter generators (represented as the yellow boxes) to select parameters that transform the input synthetic images based upo — FIGURE FROM THE PAPER

TL;DR

Learning camera-specific effects like blur, noise, and color temperature from real data and applying them to synthetic images reduces the sim-to-real gap in driving detection.

BRIEF

Object detectors trained on synthetic images often perform poorly on real photos, partly because every camera leaves its own signature in the pixels. Most sim-to-real work models scene layout, weather, or time of day, and few works model the sensor itself. This work learns a set of physically based image effects (chromatic aberration, blur, exposure, noise, and color temperature) that make synthetic images resemble a chosen real dataset. Training on the augmented synthetic data narrows the synthetic-to-real gap for object detection in urban driving scenes.

Abstract

Performance on benchmark datasets has drastically improved with advances in deep learning. Still, cross-dataset generalization performance remains relatively low due to the domain shift that can occur between two different datasets. This domain shift is especially exaggerated between synthetic and real datasets. Significant research has been done to reduce this gap, specifically via modeling variation in the spatial layout of a scene, such as occlusions, and scene environmental factors, such as time of day and weather effects. However, few works have addressed modeling the variation in the sensor domain as a means of reducing the synthetic to real domain gap. The camera or sensor used to capture a dataset introduces artifacts into the image data that are unique to the sensor model, suggesting that sensor effects may also contribute to domain shift. To address this, we propose a learned augmentation network composed of physically-based augmentation functions. Our proposed augmentation pipeline transfers specific effects of the sensor model – chromatic aberration, blur, exposure, noise, and color temperature – from a real dataset to a synthetic dataset. We provide experiments that demonstrate that augmenting synthetic training datasets with the proposed learned augmentation framework reduces the domain gap between synthetic and real domains for object detection in urban driving scenes.