Safety-by-Design Methodology for Simulation-Based Dataset Creation and AI Model Development: Applying Selected Objectives of the EASA Guidance for Level 1 & 2 Machine Learning Applications to a Runway Object Detection Use Case

Kurzfassung

Foreign Object Debris (FOD), referring to unwanted objects or materials on airport runways, poses a significant safety risk to flight operations, causing billions of dollars' worth of damage to the aviation industry each year. Artificial intelligence offers a promising solution for the automated and early detection of such objects. However, the use of machine learning in safety-critical aviation applications requires a structured development process that meets regulatory requirements. In its concept paper Guidance for Level 1 & 2 Machine Learning Applications, the European Union Aviation Safety Agency (EASA) has presented a set of objectives. These objectives remain at an abstract level and provide little guidance on how they should be implemented in practice. This results in a gap between regulatory guidance and concrete development practice.

This thesis addresses the gap by developing a Safety-by-Design methodology that integrates six selected EASA objectives. The methodology covers the Data Management, Learning Process Management, and Model Training phases of a structured development process. It involves the ODD-to-Data Traceability Matrix, the operationalization of data quality requirements, controlled data processing, and independent dataset partitioning. The methodology is supplemented by the systematic definition of the model architecture, the learning process, the training environment, and the execution and documentation of training.

The developed methodology was validated using the specific use case of object detection on Runway 23 at Hamburg Airport during the approach. This dataset consists of 13496 annotated frames from 56 simulation runs across eight object classes. It was created in Unreal Engine 5.3 using ProjectAirSim. A YOLO11m model was fine-tuned on this dataset in a two-stage process, achieving a detection performance of mAP@0.5 = 0.263 after full fine-tuning. This represents a 27.8 \% relative improvement over head-only training and more than two orders of magnitude above the zero-shot baseline of 0.0008.

These results demonstrate that the developed Safety-by-Design methodology successfully translates the selected EASA objectives into a practical, applicable development process, ensuring regulatory compliance and traceability while providing a structured foundation that measurably improves model performance through systematic data quality assurance and controlled training. This thesis bridges the gap between abstract regulatory guidance and concrete development practice, providing a replicable and validated foundation for future research on verification, integration, and certification of AI-based aviation systems.