End-to-End Adaptation of LLMs for Low-Resource Languages

Kurzfassung

While Large Language Models (LLMs) have revolutionized information processing, their benefits are disproportionately skewed toward high-resource languages, leaving languages like Sinhala behind. Building on our earlier work, An Approach to Training and Fine-Tuning Large Language Models for Low-Resource Languages, this extended version presents the complete development and evaluation of a Sinhala Large Language Model (LLM) obtained by training a LLaMA 3.1 base model for this under-represented language. Developing the Sinhala LLM required addressing challenges inherent to low-resource language modelling. The scarcity of online Sinhala text and the prevalence of printed-only literature required extensive efforts in corpus creation, translation, and noise reduction. Sinhala’s complex orthography further necessitated the design of a custom tokenization strategy aligned with existing pre-trained architectures. Additionally, the process of continued pre-training, fine-tuning, and quantization was constrained by limited hardware and memory resources. Despite these limitations, the trained SinhalaLLMdemonstrates significant progress in adapting large-scale architectures to a low-resource context. Experimental results highlight consistent improvements in text generation quality, contextual understanding, and response coherence. This study demonstrates that effective LLMs can be built for low-resource languages even with limited hardware, providing a reproducible framework for researchers facing similar constraints.