Comparing Workload in Program Comprehension Between Human-Generated and AI-Generated Program Code

Abstract

Since the introduction of GitHub Copilot in 2021 and ChatGPT in 2022, the number of AI-assistant tools in programming has increased. Consequently, we investigate the impact of AI-generated code on cognitive processes in code comprehension. The thesis comprises two main parts: a systematic literature review, followed by an experimental study involving 23 professional programmers. This study explores the effect of AI-generated code on workload during program comprehension. We used six code snippets in Python to compare the workload while reading human-written code to that of comprehending AI-generated code prompted to be “efficient” or “easy-to-read”. To assess objective workload, we recorded electrical brain activation with an EEG device. Additionally, we evaluated subjective workload using the NASA-TLX survey. Our findings reveal that subjective and objective workloads were highest when reading human-written code, while the workload for comprehending AI-generated code prompted to be “easy-to-read” was low for both. Although we observed significant differences in subjective workload between reading the AI-generated “easy-to-read” code snippets and human-written code, as well as AI-generated code prompted to be “efficient”, no significant results were found for the objective measured workload. The outcomes highlight the impact of code source and prompting on workload, particularly in reducing the workload while reading AI-generated code that is prompted to be “easy-to-read”. To take advantage of this workload reduction, developers should carefully select their prompts and improve their skills accordingly. Consequently, AI-assisted programming should be considered in future studies of cognitive processes in software development.