Usability of LLMs for Assisting Software Engineering: a Literature Review

Abstract

LLMs have been employed in various fields including software engineering. Recently, there has been an increase in studies exploring their application across a broad range of software engineering tasks, including literature reviews focusing primarily on how LLMs can be exploited to optimize processes and outcomes [13][28]. This thesis, however, presents a comprehensive literature review on the usability of Large Language Models (LLMs) for assisting in software engineering tasks. Following Kitchenham´s systematic review guidelines presented in [16], we collect and analyze 14 studies from 2017 to 2024 to answer the following research questions: (1) What methods have been employed to evaluate the usability of LLMs in addressing software engineering tasks? (2) Who are the primary users that can benefit from LLMs in software engineering assistance? (3) What difficulties are encountered when utilizing LLMs for software engineering support? In this review, we identify gaps in current usability studies of LLM-based tools and highlights areas needing further development, thereby pinpointing opportunities for future research.