Combining Large Language Models and Structured Knowledge Representations for User-Personalized Conversations of In-Car Assistants

Abstract

 Current in-car voice assistant systems lack the ability to maintain long-term memory of user preferences across conversations, resulting in impersonal interactions. This thesis investigates integrating large language model (LLM) approaches with structured user preference representations to enhance this personalization. Specifically, the research focuses on (1) developing a domain-specific conversational dataset to assess the capability of memorizing user preferences, and (2) establishing baselines for the core components of a preference-memory system using this dataset. This is achieved by extracting and storing user preferences from conversations, retrieving contextually relevant preferences for current user queries, and maintaining the user preference storage as preferences change over time.

A primary contribution of this work is the creation of a synthetic dataset tailored for the automotive domain. This dataset contains realistic user-assistant dialogues with domain-specific user preferences, allowing for targeted evaluation of the memory-system. Human evaluations confirmed the dataset's realism and utility, establishing its value for future research independent of the subsequent solution system. Furthermore, we introduce a method using LLM function-calling to achieve structured extraction of user preferences within a predefined category schema, enhancing transparency and user control. This method reliably extracts and categorizes preferences but requires refinement to reduce incorrect extractions when no preference exists. For preference retrieval, we enhance embedding-based retrieval with semantically enriched embeddings using the structured preference representation. Additionally, we offer an alternative structured retrieval method based on categorizing the user query by an LLM. While this approach achieves high retrieval precision, its increased cost and latency make it less practical. To maintain an up-to-date preference storage, we adapt an existing method to a novel LLM-based approach. Our results show the necessity and benefit of a maintenance strategy to manage an excessively growing preference storage with increasing redundant or contradictory entries. The development and evaluation of the preference-memory components demonstrate the dataset's utility and provide baseline benchmark values for further research.

Research Questions

To achieve our objectives, this research will address the following key questions: