Teleoperation of a 5-DOF Robotic Arm Using a Microsoft Kinect Sensor

Project Team

1. Ryan Lloyd
2. Sandeep Dhull
3. Ruben D'Sa

Experimental setup

Human arm & robotic arm with the reference coordinate system and joint angles

“Mr. Dummy” by Albert Alvarez from: 3D ContentCentral

http://www.3dcontentcentral.com/download-model.aspx?catalogid=171&id=191554

Computer vision module (using a Kinect, along with OpenNI drivers and the PrimeSense (Nite) SDK)

Screenshots of the tracker software in operation

Joint angle testing

In these tests, we show the mapping of various joint angles

a) Gripper testing

In this test, the gripper is expected to follow the state of the human hand (open vs. closed).

b) Wrist

 Here, we are tracking the wrist of the human arm. As you can notice, we are tracking the left wrist. For some reason (which we are trying to figure out), the right wrist wasn't working as well as the left wrist, so we decided to track the left wrist for the purposes of this video.

c) Elbow

Here, we are tracking the elbow joint of the human arm. This was the first test we conducted after we combined our vision module and POSIX threads-based robotic arm controller module into an initial version of the arm tracker.

d) Shoulder ("phi")

This video shows the shoulder joint testing. The human shoulder has a ball-socket joint (Refer to http://medicalimages.allrefer.com/large/shoulder-joint.jpg), which has 3 degrees of freedom. To map this joint to the base of the robotic arm, a spherical coordinate system was used.  We treated the human upper arm as a vector in 3D.  The tail of the vector is at the human shoulder.  "Theta" was used to represent the rotation of the upper arm about the z-axis, while "Phi" was used to represent the angle between the arm and the z-axis.

This test shows the test of the "Phi" angle

NOTE: "phi" is varying from 0 to 90 degrees in our implementation. We haven't used the full range of "phi" as of now.  Phi could extend to 180 degrees if the subject's arm crosses his torso.  However, the manipulator is unable to replicate angles of phi above 90.  Nonetheless, the manipulator could bend backwards to achieve negative angles down to -90 degrees.

e) Shoulder ("theta")

Here we are testing the angle "theta" for the shoulder (from the spherical coordinate system).

We have learned from our experiments that using a spherical coordinate system for a robotic arm tracker system isn't the best way to go. This is due to the issues faced by angle "theta" bouncing around between 0 and 90 degrees when the human upper arm's projection onto the xy-plane is near the origin.  Tiny movements in the location of the elbow in space can cause rapid, large changes in theta.

f) Full system

The full system is shown here. It isn't perfect ... but we'll get there! :)


Lessons learned from the project


Spherical coordinate system isn't the best way to try and map human arm motion to a robotic arm. This is due to the fact that the slightest movement of the shoulder beyond the XZ plane makes "theta" jump to a high value. This causes the remaining joints of the robotic arm to lose their pose relative to the shoulder links.


This can be taken care of by the following two steps (both are necessary),

• Use inverse kinematics in combination with the spherical coordinate system
• Use a robotic arm with greater degrees of freedom, especially at the elbow joint to allow independent rotation of those joints, and thus helping the system keep the relative pose of all joints constant

Future work


1) Use a different robotic arm, with greater degrees of freedom to test the system
2) Get the arm tracker working for both arms, both independently and simultaneously
3) Try to perform tasks to evaluate the effectiveness and quality of the arm tracker system

About the Team

All team members are from the Center for Distributed Robotics, University of Minnesota. Ryan and Sandeep are graduate students with the Department of  Computer Science, while Ruben is an undergraduate student with the Department of Mechanical Engineering.