Crowdsourced saliency for mining robotically gathered 3D maps using multitouch interaction on smartphones and tablets
M. Johnson-Roberson · M. Bryson · B. Douillard · O. Pizarro · Stefan B. Williams
TL;DR
Touchscreen interaction logs from people exploring underwater 3D maps can be mined to find salient points, validated against eye-tracking data.
BRIEF
Underwater robots collect huge amounts of 3D seafloor data, but it is hard to know which parts people actually find interesting. This work built a smartphone and tablet app for exploring 3D reconstructions from an Autonomous Underwater Vehicle, then logged hundreds of thousands of user interactions. Two models infer salient points from that behavior: one measures how long points stay on screen, and the other uses a Hidden Markov model on the camera path. Results are compared with traditional visual saliency methods and checked against gaze-tracking data.
Abstract
This paper presents a system for crowdsourcing saliency interest points for robotically gathered 3D maps rendered on smartphones and tablets. An app was created that is capable of interactively rendering 3D reconstructions gathered with an Autonomous Underwater Vehicle. Through hundreds of thousands of logged user interactions with the models we attempt to data-mine salient interest points. To this end we propose two models for calculating saliency from human interaction with the data. The first uses the view frustum of the camera to track the amount of time points are on screen. The second treats the camera's path as a time series and uses a Hidden Markov model to learn the classification of salient and non-salient points. To provide a comparison to existing techniques, several traditional visual saliency approaches are applied to orthographic views of the models' photo-texturing. The results of all approaches are validated with human attention ground truth gathered using a remote gaze-tracking system that recorded the locations of the person's attention while exploring the models.