Integration of Eye-Tracking and Assistant Based Speech Recognition for the command prediction at the controller working position

Abstract

This thesis aims to improve task-specific Assistant Based Speech Recognition (ABSR) for Air Traffic Controllers (ATCOs) by extracting information from multiple modalities (gaze and mouse data). ABSR processes the radar information and generates a set of hypotheses with uniform probability which is used as an additional input for speech recognition along with the audio signal. Each proposal has a command type, command value associated with an aircraft available on the screen. A framework for integrating multimodal information with proposed hypotheses by redistributing the probability for each proposal based on a rescoring algorithm is formulated. The rescoring algorithm to identify aircraft of interest has been developed based on ATCO’s gaze and mouse interaction with the radar screen. Based on trial runs done at DLR, the rescoring algorithm shows improvement in probability for predicting the correct aircraft by a factor of 4.7 in comparison with uniform probability. Further, a rescoring method for identifying the most likely command based on aircraft situation is also proposed. This framework could further be used to integrate other possible modalities in the future.