Integration of Eye-Tracking and Assistant Based Speech Recognition for the interaction at the Controller Working Position

Abstract

Supporting current digitization activities in air traffic management, DLR developed prototypes to apply assistant based speech recognition and eye tracking at the controller working position. Currently, speech recognition output is shown to the ATCO in aircraft radar labels and needs manual acknowledgment even if output is seldomly wrong. This thesis analyses how eye tracking can replace the need for this manual confirmation. If the ATCO visually scanned displayed output as detected by an eye tracker and did not intervene in a certain amount of time, the output shall directly be inserted into the ATC system. If the ATCO did not scan the output, visual saliency of aircraft labels is gradually escalated to drag ATCO’s attention. The implemented system uses color-coded frames according to dwell times and de-highlights output as soon as it was notified by the ATCO. Hence, manual ATCO workload can be reduced without strong disruptions of workflows. Two ATCOs took part in a simulation trial of the proposed system, at DLR Braunschweig in May 2021. The exercise considered of the D¨usseldorf approach scenario. Feedback was gathered from the ATCOs regarding the implemented system, contributing to the enhancement of future implementation on the concept. Also, a preliminary conclusion is drawn on the basis of the data extracted from the trials