Master's Thesis Patrick Kufner

Evaluating the Usefulness and Effectiveness of a Visualization for a Field of Study Hierarchy Graph to Explore NLP Research

Objective:

This study endeavors to assess the effectiveness of a hierarchical graph-based exploratory search interface. The assessment is split into two parts. First, an information gathering evaluation and second a user interface evaluation. The information gathering evaluation consists of two scenarios the participants must complete (using different views) while noting down their findings. The user interface evaluation consists of four questionnaire sections. First, the System Usability Scale (SUS). Second, a personal evaluation which is used to compare the tree view to the lookup view. These two sections are in a 5-point Likert scale format. Third, an open feedback section in which the participants can suggest improvements in open text fields.

The assessments will be split into two rounds. The first round will be conducted with experts and their gathered knowledge will be used as a “gold dataset”. The second round will be conducted with a representation of the target group. Their gathered knowledge will then be compared to the “gold dataset”.

By comparing the gathered knowledge as well as the questionnaire results the usefulness and effectiveness of the tree view compared to the lookup view can be analysed. Furthermore, using the feedback of all participants improvements to the tree view will be implemented and reported.

Hypothesis:
We hypothesize that the utilization of the hierarchical graph view enhances the efficiency and reliability of information research for users lacking prior knowledge in the (sub-)fields, in comparison to the lookup view. We anticipate that the hierarchical graph view will outperform the lookup view in terms of speed and accuracy in accessing relevant information.

Furthermore, we posit that the hierarchical graph view contributes to a more intuitive understanding of how different (sub-)fields interconnect and build upon each other, as opposed to the lookup view. We expect that the hierarchical graph view will ease the cognitive load associated with comprehending the hierarchical relationships within the (sub-)fields, providing users with a clearer and more cohesive insight into the subject matter.

Research Questions:

1. What are the contemporary trends and advancements in graph-based exploratory search approaches?
2. How can the graph view be optimized to suit the distinctive characteristics of NLP research papers?
3. What methods are effective for evaluating exploratory search through a hierarchical graph visualization approach?
4. What insights can be gleaned from a user study comparing the effectiveness and usefulness of graph view and lookup search in NLP paper discovery?