A Serious Gaming Schematic Design for Efficient and Application-Specific Chatbot Validation on the Example of Air Traffic Control

Abstract

The prevalence of chatbot systems is on the rise globally. However, it remains challenging to conduct a comprehensive evaluation of their use cases outside of the domain of so-called service chatbots. This is particularly evident in the context of air traffic control, where safety concerns necessitate a high degree of reliability and accuracy. In this domain, the validation of a chatbot solely based on its performance is impractical and difficult to execute, given the limitations of current large language model technologies, which lack deterministic behavior. To address this challenge, this study proposes a novel approach to validate air traffic control chatbots through the use of serious gaming. By simplifying the real-world scenario, serious gaming offers cost-effective and safe means of testing unproven designs without the need for operational experts. To this end, a methodology integrating standard chatbot evaluation strategies with the european operational concept validation methodology is developed. This methodology was applied to assess the quality of a chatbot that assumes the role of a tower controller. It was found that a chatbot system could be validated with reduced effort using serious gaming. A total of 107 students took part in three distinct studies. The air traffic control chatbot was usable, achieving an aboveaverage system usability score of 85 of 100 when tested with the latest version of the game designed for the proposed validation environment.