Manipulator and object tracking for in-hand 3D object modeling

Michael Krainin, Peter Henry, Xiaofeng Ren and Dieter Fox

The International Journal of Robotics Research 2011 30: 1311 originally published online 7 July 2011

In this paper, the authors write about using a PrimeSense depth camera (functionally the same as a Kinect, with RGB and Depth info), to create an algorithm which allows robots to create a 3D model an unknown object.

The process relies on finding the correct alignment between the object and the robot in each sensor frame.  Prior work has used only the manipulator or the object being modeled.  However, by using RGB-D and encoder data from both the robot's manipulator and the object being modeled, the authors can achieve a much more accurate alignment.

Several steps are taken in the algorithm to ensure accuracy for the model.  Most are somewhat complicated but I'll try to sum them up here in a few words.

• Kalman filtering helps 'maintain temporal consistency' between input frames and also provide estimates of uncertainty by keeping track of the manipulator's joint angles and the rotation of the object, among other things.  
• Articulated Iterative Closest Point tracking is used to estimate 'joint angles by attempting to minimize an error function over two point clouds'. 

For the actual modeling:

• 'Surfels' used as the representation for easy addition of points and removing superfluous points, as well as dealing with occlusion and updating the model.
• Loop closure to connect pieces of the model.  This involves 'maintaining a graph whose nodes are a subset of the surfels in the object model'.  Edges show that both nodes were visible in the frame, and allows for computation of connected components.
• Object re-grasping is exactly what it sounds like.  Since some parts of the object will be occluded by itself or by the manipulator, you have to look at it from multiple angles.  It's really complicated.  Put the object down, then pick it back up in a different orientation.

While this doesn't really influence our work that much, it was pretty interesting to read.

Source: http://ijr.sagepub.com/content/30/11/1311.full.pdf+html