Master's Thesis Yaren Mändle

Investigating the Adoption of Conversational Search by Customer Service Agents

Abstract:

The importance of artificial intelligence (AI)-based conversational agents (CAs) for customer service is increasing rapidly. Companies willing to improve their efficiency are integrating AI-based features into their existing systems. While there are many studies focus on the adoption of customer-facing CAs, there is a lack of research on employee-facing CAs within the specific organizational context. This thesis addresses this research gap by conducting interviews with 12 customer service agents and a survey with 17 agents to identify the key factors in adopting a newly integrated large language model (LLM)-based CA to an existing knowledge management software. We identify scenarios where the CA is preferred over traditional keyword search, factors influencing agents' choice of tool, the strengths and limitations of both search tools and evaluation metrics that play a key role in user adoption. Our study advances research on the adoption of conversational agents integrated into existing knowledge management software within an organization and offers valuable insights for those adopting CAs in their specific contexts.

Motivation:

A research gap exists in understanding the effects of AI integration into existing knowledge management software for customer service agents within a specific organizational context. Thus, in the form of a case study, customer service agents' adoption of an AI-based conversational search tool into an existing knowledge management software will be investigated in the context of a large German insurance firm. While numerous studies in the literature examine the adoption and acceptance of conversational AI and chatbots by customers in the context of customer service, there is a noticeable lack of research on the adoption of these tools by customer service employees within organizations. Thus, the main goal of this study is to examine: How do customer service agents perceive the adoption of an AI-based conversational search as a tool for addressing customer inquiries, particularly in comparison to traditional search methods, and what are their evaluations regarding its effectiveness and usability in improving their workflow?

This research aims to address the gap in understanding the adoption of AI integration within an insurance firm's knowledge management software. Specifically, it will explore how customer service agents respond to this integration and assess its effectiveness. Insights will be offered into how AI adoption affects the process of finding the correct information. By examining when and how AI-based tools are most beneficial, the research will identify use cases within the context of customer service. The outcomes are expected to offer practical strategies for increasing the adoption of new technology in customer service and guide processes after the integration of AI tools in knowledge management systems, serving as a reference for other organizations adopting similar technologies.

Research Questions:

1. What factors influence customer service agents' choice between the conversational search and the traditional keyword search?

2. How can an LLM-based conversational agent be evaluated?

3. What are the benefits and challenges of adopting LLMs within existing knowledge management systems after their integration?