Enhancing the Validation of Human Factors in User Interface Software Testing with AI

Abstract

This paper introduces a novel approach to human factor focused software testing in the loop that combines Optical Character Recognition (OCR), and large language models (LLMs) for scenario-driven, non-intrusive testing of graphical user interfaces (GUIs). By defining a testing scenario and analyzing a captured image of the UI, the system identifies visual elements using OCR and detection algorithms, storing them in a standardized format. This output serves as input for an LLM, which interprets scenario requirements and generates user actions, such as mouse clicks or keyboard inputs, to achieve the defined objectives. The system operates in a closed loop, iterating until the target outcome, e.g., the scenario is achieved, failures are detected, or usability metrics are generated. This non-intrusive method avoids software modification, making it ideal for cases where direct instrumentation is infeasible, like closed-source software. Fine-tuning the LLM for domain-specific metrics, such as cognitive load and task complexity, enhances usability testing by simulating users with varying expertise. The concept also supports environmental simulations, enabling realistic testing of safety-critical systems like automotive or aviation software. This idea offers a scalable and flexible addition to traditional GUI testing, enabling early in the software development loop human factor insights, and a first level of functionality validation for safety-relevant applications.