Towards real-time underwater 3D reconstruction with plenoptic cameras
K. Skinner · M. Johnson-Roberson
TL;DR
A plenoptic camera and GPU pipeline produce accurate, real-time dense 3D reconstructions of objects underwater, with correction for light attenuation.
BRIEF
Robots working underwater need to build 3D maps as they move, but methods that work on land struggle there. Light fades and scatters in water, which breaks common assumptions, and the depth sensors used on land often fail. This work uses a plenoptic (light field) camera, which records both color and ray direction, to get color and depth from one passive sensor. A GPU-based pipeline builds dense 3D models while correcting for attenuation. Water tank tests against an in-air ground truth model show accurate, real-time reconstructions of underwater objects.
Abstract
Achieving real-time perception is critical to developing a fully autonomous system that can sense, navigate, and interact with its environment. Perception tasks such as online 3D reconstruction and mapping have been intensely studied for terrestrial robotics applications. However, characteristics of the underwater domain such as light attenuation and light scattering violate the brightness constancy constraint, which is an underlying assumption in methods developed for land-based applications. Furthermore, the complex nature of light propagation underwater limits or even prevents subsea use of real-time depth sensors used in state-of-the-art terrestrial mapping techniques. There have been recent advances in the development of plenoptic (also known as light field) cameras, which use an array of micro lenses capturing both intensity and ray direction to enable color and depth measurement from a single passive sensor. This paper presents an end-to-end system to harness these cameras to produce real-time 3D reconstructions underwater. Our system builds upon the state-of-the-art in online terrestrial 3D reconstruction, transferring these approaches to the underwater domain by gathering real-time color and depth (RGB-D) data underwater using a plenoptic camera, and performing dense 3D reconstruction while compensating for attenuation effects of the underwater environment simultaneously, using a graphics processing unit (GPU) to achieve real-time performance. Results are presented for data gathered in a water tank and the proposed technique is validated quantitatively through comparison with a ground truth 3D model gathered in air to demonstrate that the proposed approach can generate accurate 3D models of objects underwater in real-time.