An Approach to Training and Fine-tuning Large Language Models for Low-Resource Languages

Abstract

The essence of communication is information exchange. However, people living in less connected areas often face socioeconomic and technological barriers, which limit their access to crucial information that can improve their livelihood. Developing a Sinhala Large Language Model (LLM) presents significant challenges. Sinhala is a low-resource language with limited online content and a substantial portion of its literature confined to print. Preparing a diverse and clean dataset required significant effort in text generation and translation, and noise reduction. Additionally, Sinhala's complex orthography necessitated custom tokenization strategies compatible with pre-trained models, further complicating the process. Severe hardware limitations and memory bottlenecks were also major constraints that were experienced during the continued pre-training, finetuning, and quantization phases. Despite these challenges, the Sinhala LLM represents a critical step in bridging the digital divide for Sinhala speakers. Although the constraints restrict the model's performance, it gives important insights into the training of LLMs for underrepresented languages. It contributes to the fast-growing topic of low-resource language modeling and paves the way for further research in this domain.