Infinite Leagues Under the Sea: Photorealistic 3D Underwater Terrain Generation by Latent Fractal Diffusion Models
Tianyi Zhang · Weiming Zhi · Joshua Mangelson · Matthew Johnson-Roberson
TL;DR
DreamSea trains a diffusion model on robot-survey images to generate large, consistent, photorealistic 3D underwater terrain rendered with Gaussian splatting.
BRIEF
Generic image generators are trained mostly on everyday photos, so their underwater scenes often look unconvincing; the seafloor is rare in internet data. DreamSea instead learns from images collected by underwater robot surveys. Visual foundation models extract geometry and semantics from those noisy images, and a diffusion model generates RGBD seafloor views conditioned on fractal-based latent embeddings. The generated views are fused into a 3D Gaussian splatting map, guided by 2D diffusion priors, for novel-view rendering. Applications include filming, gaming, and robot simulation.
Abstract
This paper tackles the problem of generating representations of underwater 3D terrain. Off-the-shelf generative models, trained on Internet-scale data but not on specialized underwater images, exhibit downgraded realism, as images of the seafloor are relatively uncommon. To this end, we introduce DreamSea, a generative model to generate hyper-realistic underwater scenes. DreamSea is trained on real-world image databases collected from underwater robot surveys. Images from these surveys contain massive real seafloor observations and covering large areas, but are prone to noise and artifacts from the real world. We extract 3D geometry and semantics from the data with visual foundation models, and train a diffusion model that generates realistic seafloor images in RGBD channels, conditioned on novel fractal distribution-based latent embeddings. We then fuse the generated images into a 3D map, building a 3DGS model supervised by 2D diffusion priors which allows photorealistic novel view rendering. DreamSea is rigorously evaluated, demonstrating the ability to robustly generate large-scale underwater scenes that are consistent, diverse, and photorealistic. Our work drives impact in multiple domains, spanning filming, gaming, and robot simulation.