Visual Tracking for On-Orbit Robotic Manipulation

Abstract

On-Orbit Servicing (OOS) missions are central to reducing space debris, and their success depends on accurate pose estimation of non-cooperative satellites. Existing state-of-the-art tracking frameworks, such as the Multi-Modal Model-based Tracker (M3T), rely on color histograms and therefore degrade significantly when constrained to grayscale input, which is the standard in space. This thesis investigates how performance of pose estimation is affected when integrating M3T into the space environment, and how segmentation masks can restore robustness under these conditions. We first benchmark M3T’s performance when transitioning from RGB to grayscale input, confirming a substantial decline in accuracy and stability. Ground-truth segmentation masks gives a reference for achievable performance under ideal conditions, demonstrating that improved foreground–background separation can dramatically recover performance. Building on this, we integrate the Segment Anything Model 2 (SAM2) into the tracking pipeline and introduce multiple fusion strategies to combine segmentation with histogram-based estimates. To enable systematic evaluation, a custom visualization platform was developed, and the extended pipeline was prepared for deployment on the On-Orbit Servicing Simulator (OOS-SIM). Results show that segmentation masks significantly enhance pose estimation robustness in grayscale conditions. While challenges remain in highly symmetric and rapidly-moving scenarios, this work provides a practical pathway for adapting M3T to realistic orbital conditions. The contributions include the first systematic evaluation of M3T under space-like constraints, the integration of segmentation into the tracking pipeline, and the development of tools and frameworks that lay the foundation for OOS tracking systems.