DispSegNet: Leveraging Semantics for End-to-End Learning of Disparity Estimation from Stereo Imagery
Junming Zhang · Katherine A. Skinner · Ram Vasudevan · Matthew Johnson-Roberson
TL;DR
A single stereo network refines disparity using features learned for semantic segmentation, training without ground-truth depth and outputting both depth and labels.
BRIEF
Stereo depth estimation with neural networks still makes errors on textureless surfaces, occluded areas, and reflections. This work couples depth estimation with semantic segmentation in one network. An initial disparity estimate is refined using an embedding learned by the segmentation branch, so knowing what an object is helps correct where it sits in depth. Training is unsupervised: one stereo image is warped and compared against the other. A single network outputs both disparity and semantic labels, which suits driving applications with real-time limits. Tests on KITTI and Cityscapes show state-of-the-art results.
Abstract
Recent work has shown that convolutional neural networks (CNNs) can be applied successfully in disparity estimation, but these methods still suffer from errors in regions of low texture, occlusions, and reflections. Concurrently, deep learning for semantic segmentation has shown great progress in recent years. In this letter, we design a CNN architecture that combines these two tasks to improve the quality and accuracy of disparity estimation with the help of semantic segmentation. Specifically, we propose a network structure in which these two tasks are highly coupled. One key novelty of this approach is the two-stage refinement process. Initial disparity estimates are refined with an embedding learned from the semantic segmentation branch of the network. The proposed model is trained using an unsupervised approach, in which images from one half of the stereo pair are warped and compared against images from the other camera. Another key advantage of the proposed approach is that a single network is capable of outputting disparity estimates and semantic labels. These outputs are of great use in autonomous vehicle operation; with real-time constraints being key, such performance improvements increase the viability of driving applications. Experiments on KITTI and Cityscapes datasets show that our model can achieve state-of-the-art results and that leveraging embedding learned from semantic segmentation improves the performance of disparity estimation.