Investigating Complex Answer Attribution Approaches with Large Language Models

This master's thesis explores the attribution of answers in complex question-answering scenarios utilizing large language models (LLMs). The research aims to assess and enhance the traceability of answers back to their source documents, a critical aspect of knowledge retrieval using LLMs. By examining the relationship between the questions posed and the corresponding answers provided by LLMs, the study seeks to determine the extent to which these answers can be attributed to specific source materials, and to identify the challenges and limitations inherent in the current attribution processes.

Through a methodical inquiry structured around four research questions, the study investigates the categorization of complex questions, the creation of a specialized dataset for analysis, and the examination of LLMs’ patterns in answering and attributing responses. The effectiveness of existing methods for answer attribution is scrutinized, and potential improvements are considered. In pursuit of broader applicability, the thesis also proposes a generalized framework for evaluating answer attribution that could potentially be independent of the domain and the complexity of the questions involved.

The thesis extends its analysis to various enterprise-level domains, evaluating the universality of the proposed categorizations and attribution methods. By conducting cross-domain assessments, the research highlights the adaptability of the developed approaches to diverse sets of questions and subject matters. The findings are expected to contribute to the field by advancing our understanding of LLMs' performance in complex Q&A settings and by proposing refinements to the processes used to ensure accurate and reliable answer attribution.