Knowledge Distillation of Large Language Models for Use Cases of the German Aerospace Center

Abstract

Large Language Models (LLMs) have shown remarkable capabilities; however, their high computational demands present notable challenges for deployment in resource-limited environments, especially in real-time, domain-specific applications at the German Aerospace Center (DLR). This thesis tackles this issue by using Knowledge Distillation (KD) to compress a large, powerful teacher model into a smaller, more computationally efficient student model. This work proposes a comprehensive framework for distilling knowledge from an 8-billion-parameter teacher (LLaMA 3.1) to a 1-billion-parameter student (LLaMA 3.2). The methodology is centred around a DLR-specific use case: an autonomous vehicle system which handles natural language voice commands in real-time, integrating contextual sensor data to ensure safe and efficient command execution. For this purpose, a novel synthetic data generation pipeline was created to build a domain-specific dataset. The distilled student model was tested thoroughly against the teacher model and also a baseline student of the same size. For the DLR-specific task, the distilled model showed better safety-awareness, with a clear reduction in critical failures compared to the baseline. While the improvements on a difficult public function-calling benchmark were more modest, the distillation still led to a big increase in the baseline’s accuracy. These performance gains came together with an eightfold reduction in model size and almost a fivefold boost in inference speed, which shows the model’s potential for on-board deployment. All in all, these results suggest that knowledge distillation is a valid and practical strategy for building efficient and more reliable LLMs for specialised, high-stakes use cases.