Scenario Generation for Testing of Automated Driving Functions based on Real Data

Abstract

Scenario-based testing is state-of-the-art for testing Advanced Driving Assistance System / Autonomous Driving (ADAS/AD). The challenge in scenario-based testing is the generation and selection of the scenarios. To generate reproducible scenarios and to efficiently perform tests of ADAS/AD, simulation environments are used because the environment is under control. However, an open research question on this topic is the realism of the emerging scenarios within the simulation. Realism is a challenge because the ADAS/AD must eventually function in the real world. To solve this challenge, we contribute a concept (1) to use a simulation environment to generate realistic synthetic scenarios and (2) to evaluate their realism. We focus our research on dynamic objects within the scenarios. We parameterize the microscopic traffic simulation environment SUMO and generate synthetic scenarios by simulation. We base the evaluation of realism on real scenarios observed by the testbed Lower Saxony. To measure realism, we define ten different characteristics in different aspects. With these characteristics, we measure realism by comparing the characteristics against the real data. As a prototype, we implement this concept and compare three different methods of parameterization concerning their realism: (a) expert-based, (b) optimization-based, and (c) clustering-based. Based on our evaluation, we find that parameterization has a strong influence on the realism of criticality metrics such as the Time To Collision (TTC). In contrast, we find that the influence of parameterization on other aspects is comparatively low. We observe that realism depends on the parameterization and the capabilities of the simulation model. We discover that expert-based parameterization generates the most realistic scenes compared to the other methods and about 2.5 times as many realistic scenes during the same period as without parameterization. Each parameterization has its own strengths concerning different aspects of realism. We conclude that SUMO generates realistic dynamic objects in scenarios in many aspects.