Master's Thesis Doruk Gercel

An LLM-based Method to Generate Instructions for Users How to Navigate Web-based User Interfaces

Abstract:

The popularity of Large Language Models (LLMs) and the development of business applications based on LLMs are increasing rapidly. Different types of businesses and companies integrate LLMs into their businesses to develop innovative solutions. While much research is being conducted, there is a lack of LLM-based methods to generate instructions for users and guide them on navigating web-based user interfaces. This thesis addresses this research gap by proposing a grammar to textually represent an industrial web user interface (UI) application and generating an instruction and prompt format to receive UI instructions from the LLM. Using these methodologies and foundations, a chatbot application is designed and implemented to provide user guidance. This study advances research on developing a methodology that leverages the intelligence of LLMs to provide user guidance in web UI applications and develop a state-of-the-art chatbot application.

Motivation:

A modern technology company wants to develop a state-of-the-art solution to make the web UI setup easier with virtual assistance and guide the user in the UI according to the user’s prompts. The main objective of this study is to find an LLM-based method to generate instructions for users to navigate web-based UI. The study aims to develop a solution that can be easily integrated into the web application to guide users in the application conveniently by considering the user experience and enhancing the user journey. Also, utilizing the merits of the LLM is a critical point of the study, and its intelligence shall be leveraged to develop an innovative solution. To achieve this goal, efficient approaches to interact with the LLM, describe the web application, and generate the required guidance steps are formulated and tested to evaluate whether they meet the requirements. Afterward, these approaches are combined in harmony, and an LLM-based chatbot is designed, partially implemented for specific use cases, and integrated into the web UI application.

Research Questions:

1. How to model UI components to feed LLM to generate the required steps to perform UI actions?

2. How to generate UI instructions from the textual output generated from the LLM?

3. How to integrate this chatbot into the product by following and considering software engineering principles?