Adigun, Jubril Gbolahan und Philip Huck, Tom und Camilli, Matteo und Felderer, Michael (2023) Risk-driven Online Testing and Test Case Diversity Analysis for ML-enabled Critical Systems. In: 34th IEEE International Symposium on Software Reliability Engineering, ISSRE 2023, Seiten 344-354. ISSRE 2023, 2023-10-09 - 2023-10-12, Florenz, Italien. doi: 10.1109/ISSRE59848.2023.00017. ISBN 979-835031594-3. ISSN 1071-9458.
![]() |
PDF
1MB |
Offizielle URL: https://dx.doi.org/10.1109/ISSRE59848.2023.00017
Kurzfassung
Machine Learning (ML)-enabled systems that run in safety-critical settings expose humans to risks. Hence, it is important to build such systems with strong assurances for domain-specific safety requirements. Simulation as well as metaheuristic optimizing search have proven to be valuable tools for online testing of ML-enabled systems for early detection of hazards. However, the efficient generation of effective test cases remains a challenging issue. In particular, the testing process shall produce as many failures as possible but also unveil diverse sets of failure scenarios.To study this phenomenon, we introduce a risk-driven test case generation and diversity analysis method tailored to ML-enabled systems. Our approach uses an online testing technique based on metaheuristic optimizing search to falsify domain-specific safety requirements. All test cases leading to hazards are then analyzed to assess their diversity by using clustering and interpretable ML. We evaluated our approach in a collaborative robotics case study showing that generating tests considering risk metrics represents an effective strategy. Furthermore, we compare alternative optimizing search algorithms and rank them based on the overall diversity of the test cases, ultimately showing that selecting the testing strategy based on the number of failures only may be misleading.
elib-URL des Eintrags: | https://elib.dlr.de/201699/ | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Dokumentart: | Konferenzbeitrag (Vortrag) | ||||||||||||||||||||
Titel: | Risk-driven Online Testing and Test Case Diversity Analysis for ML-enabled Critical Systems | ||||||||||||||||||||
Autoren: |
| ||||||||||||||||||||
Datum: | 2023 | ||||||||||||||||||||
Erschienen in: | 34th IEEE International Symposium on Software Reliability Engineering, ISSRE 2023 | ||||||||||||||||||||
Referierte Publikation: | Ja | ||||||||||||||||||||
Open Access: | Ja | ||||||||||||||||||||
Gold Open Access: | Nein | ||||||||||||||||||||
In SCOPUS: | Ja | ||||||||||||||||||||
In ISI Web of Science: | Ja | ||||||||||||||||||||
DOI: | 10.1109/ISSRE59848.2023.00017 | ||||||||||||||||||||
Seitenbereich: | Seiten 344-354 | ||||||||||||||||||||
ISSN: | 1071-9458 | ||||||||||||||||||||
ISBN: | 979-835031594-3 | ||||||||||||||||||||
Status: | veröffentlicht | ||||||||||||||||||||
Stichwörter: | Software Testing Safety ML-enabled Critical Systems Test Case Diversity Quality Assurance Robotics | ||||||||||||||||||||
Veranstaltungstitel: | ISSRE 2023 | ||||||||||||||||||||
Veranstaltungsort: | Florenz, Italien | ||||||||||||||||||||
Veranstaltungsart: | internationale Konferenz | ||||||||||||||||||||
Veranstaltungsbeginn: | 9 Oktober 2023 | ||||||||||||||||||||
Veranstaltungsende: | 12 Oktober 2023 | ||||||||||||||||||||
HGF - Forschungsbereich: | Luftfahrt, Raumfahrt und Verkehr | ||||||||||||||||||||
HGF - Programm: | Raumfahrt | ||||||||||||||||||||
HGF - Programmthema: | Robotik | ||||||||||||||||||||
DLR - Schwerpunkt: | Raumfahrt | ||||||||||||||||||||
DLR - Forschungsgebiet: | R RO - Robotik | ||||||||||||||||||||
DLR - Teilgebiet (Projekt, Vorhaben): | R - Roboterdynamik & Simulation [RO], R - Software Engineering und Qualitätssicherung (SeQu), D - Kurzstudien [KIZ] | ||||||||||||||||||||
Standort: | Köln-Porz | ||||||||||||||||||||
Institute & Einrichtungen: | Institut für Softwaretechnologie | ||||||||||||||||||||
Hinterlegt von: | Felderer, Michael | ||||||||||||||||||||
Hinterlegt am: | 12 Jan 2024 11:14 | ||||||||||||||||||||
Letzte Änderung: | 24 Apr 2024 21:02 |
Nur für Mitarbeiter des Archivs: Kontrollseite des Eintrags