In this work, the problem of programming contact-based manipulation tasks is addressed by having an operator specify the task within a simulated virtual environment. Generation of models within this virtual environment is facilitated using a fast, occlusion-tolerant, 3D grey-scale vision system which can recognize and accurately locate objects within the work site. A major goal of this work is to make robotic programming easy and intuitive for untrained users working with standard desktop hardware. Simulation can help accomplish this, offering the ease-of-use benefits of ``programming by demonstration'', coupled with the ability to create a programmer-friendly virtual environment. Within a simulated environment, it is also straightforward to track and interpret an operator's actions.
The simulation emulates rigid-body contact, combined with first-order dynamics, so that manipulated objects will slide along and around each other. In this way, the constraints presented by other objects in the workspace can actually be used to make part placement easier, rather than harder. In particular, the dynamic interactions among the objects, when coupled with artificial graphical fixtures, make it easy to manipulate and place objects using a simple 2D mouse.
When the operator has completed a desired manipulation, local planning techniques are used to automatically turn the virtual environment's motion sequence history into a set of robot motion commands capable of realizing the prescribed task.
Some examples of the system in action are available here as short MPEG videos, with demonstration tasks involving a puzzle of wooden blocks. An overview video (1.2 Mbytes) starts by showing the vision system identifying and locating blocks within the scene, and the Identified objects being loaded into the virtual environment. An operator then clicks on one of the blocks, producing a ``dragger fixture'' with which the block is moved and eventually placed in a corner. Simulated contact interactions, combined with first-order dynamics, allow the block to slide into the corner easily, with the operator directing the motion in only a very coarse way. After the block has been placed, the system examines the motion history in the simulated environment and produces a program to execute the prescribed task.
A more detailed video of a cornering task (2.5 Mbytes) shows a block being slid along the outside of the puzzle frame, and finally being placed into a far inside corner. A time-sampled display then shows the complete path followed by the object during the simulation. The system then optimizes this path, by treating it as an ``elastic band'' and shrinking it into a more compact form for robotic execution. Local path planning is used to keep the final path away from undesired contacts, and extraneous actions, such as the initial contact with the outside of the frame, are removed.
When manipulating an object in simulation, it is also possible to specify the preservation of contacts with selected objects. In this way, more complicated tasks, such as dragging a block around a corner (2.4 Mbytes) can be specified. When the time comes to shrink the initial path into an optimal one for robotic execution, the desired contact states are maintained, so that the essential contact aspects of the task are preserved.