Prompt Engineering for Steering Emotions in Large Language Model (LLM) Responses

Abstract

Emotion-awareness in conversational agents has been shown to be beneficial and can be exploited in high-stress scenarios, such as soothing flight controllers' emotions in aerospace. Despite advances in Large Language Models, steering responses to specific emotions remains challenging, but crucial for emotion-aware and neutral communication needs. Recently, new methods for steering model responses have emerged. Following these developments, this thesis investigates the potential of steering emotions in model responses through one new method called Prompt Engineering. The aim is to comprehensively evaluate the effectiveness of Prompt Engineering for steering emotions and compare its effectiveness to Fine-Tuning. This work employs the ChatGPT base model, GPT-3.5-turbo, from OpenAI over the OpenAI interface. The experiments are conducted iteratively, with each prompting technique representing one cycle, i.e., Zero-Shot, Zero-Shot Chain-of-Thought, Few-Shot, and Chain-of-Thought Prompting. The most effective prompt from each cycle serves as the basis for the next. The different prompts are designed based on three prompting guides and relevant scientific findings. Additionally, the same model is fine-tuned on a novel dataset of 120 human-written examples, each expressing one of six Ekman basic emotions. Both quantitative and qualitative methods are used to evaluate the results. The quantitative evaluation includes an Emotion Score based on the F1 Score of a selftrained emotion classifier and textual quality metrics such as BERTScore, Flesch Reading Ease Score, Distinct-1, Distinct-2, and Accuracy. Qualitative evaluation involves categorising outstanding features in the responses. The findings indicate that Prompt Engineering is a valuable tool for steering emotions in model responses. The most effective approach was a Few-Shot Prompt with six examples written by the same author, each representing one of the six Ekman emotions. Even the simplest Zero-Shot Prompts yielded good results. However, including examples mitigated unwanted model behaviour. In comparison, ZeroShot Chain-of-Thought and Chain-of-Thought Prompts were less effective. Overall, Prompt Engineering outperformed Fine-Tuning in terms of Emotion Score and textual quality, suggesting the need for a more diverse dataset and higher temperature settings for effective Fine-Tuning. Future research can focus on collecting more diverse data and exploring different languages and cultures. Additionally, it would be valuable to use human raters for evaluation, examine various models, such as open-source models, and compare other methods to Prompt Engineering.