Instructing Robots by Sketching: Learning from Demonstration via Probabilistic Diagrammatic Teaching

Weiming Zhi · Tianyi Zhang · Matthew Johnson-Roberson

Fig. 1: Diagrammatically teaching a quadruped with a mounted arm to shut a drawer, by sketching robot demonstrations over 2D images. The user sketches trajectories of the desired movement over images of the scene (top left and right). The r — FIGURE FROM THE PAPER

TL;DR

Users teach robots new skills by sketching trajectories on 2D camera images, which are converted into a probabilistic model of 3D motion.

BRIEF

Teaching a robot a new skill usually means physically moving its arm or operating a teleoperation rig. This work lets a user instead sketch the desired motion directly on camera images of the scene. The method turns probability densities from the 2D sketches into regions in 3D space using ray-tracing, then fits a probabilistic model of trajectories to them. New trajectories that mimic the sketches can be sampled from this model. It was tested in simulation and on a fixed-base arm and a quadruped-mounted arm.

Abstract

Learning for Demonstration (LfD) enables robots to acquire new skills by imitating expert demonstrations, allowing users to communicate their instructions in an intuitive manner. Recent progress in LfD often relies on kinesthetic teaching or teleoperation as the medium for users to specify the demonstrations. Kinesthetic teaching requires physical handling of the robot, while teleoperation demands proficiency with additional hardware. This paper introduces an alternative paradigm for LfD called Diagrammatic Teaching. Diagrammatic Teaching aims to teach robots novel skills by prompting the user to sketch out demonstration trajectories on 2D images of the scene, these are then synthesised as a generative model of motion trajectories in 3D task space. Additionally, we present the Ray-tracing Probabilistic Trajectory Learning (RPTL) framework for Diagrammatic Teaching. RPTL extracts time-varying probability densities from the 2D sketches, applies ray-tracing to find corresponding regions in 3D Cartesian space, and fits a probabilistic model of motion trajectories to these regions. New motion trajectories, which mimic those sketched by the user, can then be generated from the probabilistic model. We empirically validate our framework both in simulation and on real robots, which include a fixed-base manipulator and a quadruped-mounted manipulator.