Hand Motion Estimation Using Image-based Reconstruction of the Forearm

Abstract

With the evolution of prosthetics through years, sophisticated devices, fitted to each individual’s needs, are capable to return amputees to the everyday-life they desire to have or were habituated to. Moreover, the astonishing progress in the field of computer vision has led to amazing results. Taking these as a starting point, this work focuses on exploiting a new approach to estimate the motion of the hand based on the deformations of the surface of the arm due to muscle movements. In a phychophysical experiment with ten healthy subjects, a camera captures images of the forearm, which are further processed to extract features. Visual fiducial markers, AprilTags, are created from an ordinary printer and placed on the subjects’ forearm. The AprilTag detection software computes the precise 3D position, orientation, and identity of the tags relative to the camera. As ground truth, a visual stimulus was used, avoiding the need for finger sensors (force/position sensors, datagloves, etc.). With the assistance of two regression models, one linear and one nonlinear, we can then predict the motion of the fingers. Since we seek to detect motion of tendons and muscles in the forearm, this project addresses to below- elbow amputees with some remaining musculoskeletal activity. The results are highly promising, showing an average normalized root mean square error in the range of 0.05 to 0.22.