ROS 2 Object Tracking with PointNet and xArm 6

Overview

Built a robotics perception and control pipeline for an xArm 6 as part of a General Motors automotive-part handling challenge. The project combined an Azure Kinect RGB-D camera, point-cloud processing, a custom PointNet classifier, ROS 2 inference, and physical visual tracking.

3D Deep Learning

A custom point-cloud dataset was generated from the target automotive part using RGB-D data and 3D geometry. Point clouds were normalized to 1,024 points and augmented before training.

A simplified PointNet model was implemented in TensorFlow/Keras with a T-Net, shared MLP layers, global max pooling, dropout, and softmax classification.

Results:

• 99.1% training accuracy.
• 96.4% test accuracy.
• Sub-30 ms CPU inference inside a ROS 2 node.

Physical Tracking

For the physical arm, a lightweight RGB pipeline segmented the target object, computed its centroid, and translated image error into incremental joint commands.

The physical tracking loop:

• Ran at 10 Hz.
• Used 0.02 rad joint increments.
• Maintained sub-50 ms end-to-end visual-servoing latency.
• Tracked object motion up to 0.3 m/s without losing visual lock.

Important Technical Note

The PointNet classifier and physical visual-servoing loop were validated independently. Due to limited lab access, the PointNet inference node was not fully connected into the final physical arm-control loop.