Toward Standardized Benchmarking of Search-Based Scenario 2 Selection Methods in Autonomous System Validation

Abstract

Autonomous systems must demonstrate reliable safety and robustness even under rare, safety-critical conditions. Because the vast range of possible operating situations cannot be exhaustively tested, scenario-based testing has emerged as a structured approach to expose systems to representative and challenging situations. Search-based scenario selection methods (SBSSMs) algorithmically explore the scenario space to identify critical and informative cases. Yet, existing implementations are often tightly coupled to specific domains and simulation environments, hindering reproducibility, comparability, and generalizability across application areas. This paper introduces a benchmarking framework that addresses these limitations by providing a domain-independent environment for the systematic evaluation of SBSSMs. Building on empirical observations from maritime simulation studies and the literature, the framework generates synthetic test instances that mimic the structural patterns found there while abstracting away application-specific semantics. It supplies standardized performance metrics and reference implementations of key method classes, enabling direct comparison of new approaches with established ones under controlled, repeatable conditions. Illustrative demonstrations show how the benchmark reveals characteristic strengths and weaknesses of different methods across diverse structural settings. As a reproducible and extensible baseline, the framework promotes standardized performance assessment and the transfer of methodological advances from maritime navigation to domains such as automotive safety.