Iterative Scenario-Based Testing in an Operational Design Domain for Artificial Intelligence Based Systems in Aviation

Abstract

The development of Artificial Intelligence (AI) based systems is becoming increasingly prominent in various industries. The aviation industry is also gradually adopting AI-based systems. An example could be using Machine Learning algorithms for flight assistance. There are several reasons why adopting these technologies poses additional obstacles in aviation compared to other industries. One reason is strong safety requirements, which lead to obligatory assurance activities such as thorough testing to obtain certification. Amongst many other technical challenges, a systematic approach is needed for developing, deploying, and assessing test cases for AI-based systems in aviation. This paper proposes a method for iterative scenario-based testing for AI-based systems. The method contains three major parts: First, a high-level description of test scenarios; second, the generation and execution of these scenarios; and last, monitoring of scenario parameters during scenario execution. The scenario parameters, which can be for instance environmental or system parameters, are refined and the test steps are executed iteratively. The method forms a basis for developing iterative scenario-based testing solutions. As a domain-specific example, a practical implementation of this method is illustrated. For an object detection application used on an airplane, flight scenarios, including multiple airplanes, are generated from a descriptive scenario model and executed in a simulation environment. The parameters are monitored using a custom Operational Design Domain monitoring tool and refined in the process of iterative scenario generation and execution. The proposed iterative scenario-based testing method helps in generating precise test cases for AI-based systems while having a high potential for automation.