Master's Thesis Yaren Dalgic

Investigating the Adoption of Conversational Search by Customer Service Agents

Abstract:

Large Language Models (LLMs) have recently demonstrated substantial language understanding and almost human-like conversation capabilities. However, relying solely on LLMs for question-answering tasks is insufficient as they are limited by their training data and prone to producing inaccurate information. Retrieval-Augmented Generation (RAG) techniques aim to solve those limitations by introducing external knowledge sources, improving the precision of answers generated by LLM. This study evaluates various state-of-the-art RAG techniques in improving LLM performance across different question complexities. Furthermore, it investigates the strengths and weaknesses of RAG systems that retrieve data from vector databases versus graph databases. Our findings indicate that vector-based RAG systems excel in answering general questions and are better suited for LLMs that are less powerful in the processing of complex rules. In contrast, graph-based systems are more effective on complex questions that require data from multiple documents. This research is based on Wikipedia and the Wikidata Knowledge Graph (KG), providing a comprehensive guide for evaluating and selecting the appropriate RAG technique for other domain-specific datasets.

Motivation:

This study aims to systematically compare different embedding-based and graph-based RAG techniques to analyze their abilities in augmenting LLMs. Additionally, our goal is to understand how vector and graph database structures influence the performance of RAG systems across various question complexities. This study paves the way for further research into enhancing retrieval approaches and contributing to refining RAG systems. Furthermore, this study seeks to provide comprehensive guidelines and metrics for evaluating retrieval systems, enabling the selection and subsequent tailoring of an appropriate system for specific datasets and applications.

Various RAG techniques have emerged to enhance the accuracy of LLMs, varying in their re-
trieval process and database structures. While some RAG approaches rely on vector databases and embeddings to collect documents close to user queries in semantic similarity, others leverage the structured nature of graph databases to gather necessary information for the LLM. Vector databases, known for their ease of implementation and ability to handle unstructured data, connect questions to relevant documents in the database through vector similarity. Graph databases, on the other hand, structure data with deeper relational reasoning, storing additional document relationships on top of the textual data. This architecture allows the retrieval process to navigate from one document to another, constructing richer contextual data at each step.

Research Questions:

1. How do vector and graph databases differ in their performance when augmenting LLMs in question-answering tasks?

2. What are existing retrieval approaches for RAG methods using vector and graph databases?

3. How can a vector database be aligned with a graph database to include the same information and be comparable in terms of retrieval performance?

4. How can the quality of question-answering performance be systematically evaluated across different LLM-based RAG systems?