Operationalizing AI explainability using interpretability cues in the cockpit: Insights from User-Centered Development of the Intelligent Pilot Advisory System

Abstract

This paper presents a concept for operationalizing Artificial Intelligence (AI) explainability for the Intelligent Pilot Advisory System (IPAS) as requested in the European Aviation Safety Agency’s AI Roadmap 2.0 in order to meet the requirement of Trustworthy AI. The IPAS is currently being developed to provide AI-based decision support in commercial aircraft to assist the flight crew, especially in emergency situations. The development of the IPAS is following a user-centred and exploratory design approach, with the active involvement of airline pilots in the early stages of development to iteratively tailor the system to their requirements. The concept presented in this paper aims to provide interpretability cues to achieve “operational explainability of AI”, which should enable commercial aircraft pilots to understand and adequately trust the recommendations generated by AI when making decisions in emergencies. Focus of the research was to identify initial interpretability requirements and to answer the question of what interpretation cues pilots need from the AI-based system. Based on a user study with airline pilots, four requirements for interpretation cues were formulated. These results will form the basis for the next iteration of the IPAS, where the requirements will be implemented.