Automated Scenario Generation from Operational Design Domain Model for Testing AI-Based Systems in Aviation

Abstract

Applications based on artificial intelligence (AI) promise benefits, ranging from improved performance to increased capabilities in many industries. In the aviation domain, one example is the new Airborne Collision Avoidance System (ACAS X). The current investigation aims at combining ACAS X and AI to maintain its performance while decreasing the memory footprint. However, the anticipation of AI being increasingly used confronts regulators with challenges in terms of safety assurance and certification. Consequently, the European Union Aviation Safety Agency (EASA) published a concept paper for machine learning applications in aviation. Both, the Concept of Operation (ConOps) in combination with an Operational Design Domain (ODD), are listed as objectives to be met for the safety analysis. From a developer’s perspective, this raises questions on how to effectively derive the ODD from ConOps and test the given system based on the ODD description. Based on an exemplary use case of a Near Mid-Air Collision avoidance between two aircraft through the advisories of ACAS X, a highly automated framework for generating and testing synthetic data is proposed. Using this framework, 1800 Near Mid-Air Collision scenario files are created and automatically executed in the simulation environment FlightGear. Scenario-based testing is used for the logging of ACAS X advisory data and evaluating it against predefined requirements. By this approach, an efficient way of verifying system requirements and conducting automated testing based on the ODD definition is demonstrated. Throughout this process, Model-Based Systems Engineering (MBSE) is used to reduce and manage complexity. The framework in this paper enables a systematic and highly automated approach for scenario generation based on the ODD.