Knowledge Graph-based Engineering Decision Making: A Natural Language Interface for Accessing Manufacturability Data

Abstract

Large Language Models (LLMs) have revolutionized natural language processing by enabling intuitive interfaces between humans and complex datasets. This thesis investigates the potential of LLMs, particularly the Llama 3.1 models, in addressing two pivotal tasks: Text-to-SPARQL and Triple-to-Text. These tasks are essential for democratizing access to knowledge graphs, which traditionally require specialized expertise in query languages like SPARQL. By bridging the gap between natural language and structured data, LLMs hold the promise of making semantic knowledge systems more accessible and versatile. The study evaluates the performance of the 8B and 70B Llama 3.1 models using a novel dataset designed to benchmark these tasks. For the Text-to-SPARQL task, the 70B model exhibited robust performance, demonstrating high syntactic validity and semantic alignment. Enhancements were observed when schema-level knowledge (T-Box) was incorporated, underscoring the importance of contextual information in query generation. In contrast, the 8B model consistently failed to generate reliable queries, struggling with prefix adherence and graph structure interpretation. For the Triple-to-Text task, which translates RDF triples into natural language, both models excelled in semantic alignment, as evidenced by strong BERTScores. However, lexical fidelity, as measured by BLEU and ROUGE, remained a challenge. The 70B model distinguished itself by consistently generating contextually coherent and semantically precise responses, making it a more reliable choice for applications prioritizing meaning over exact wording. This research contributes a comprehensive evaluation framework, benchmark dataset, and critical insights into the role of LLMs in structured data interaction. By highlighting the strengths and limitations of current models, it paves the way for future advancements, including multimodal capabilities, robust error handling, and enhanced adaptability to real-world knowledge systems.