Exploring the Impact of LLM-assisted Re-ranking in Studification for Schizophrenia Research

Abstract

This thesis investigates methods for improving the ranking of medical studies in systematic literature reviews. The goal is to ensure that the most relevant studies appear at the top of retrieved results, thereby supporting more efficient evidence synthesis. The approach builds on an initial retrieval step using vector search, followed by a re-ranking phase that integrates multiple signals, including study aspects (condition, intervention, outcome) predicted with large language models, as well as metadata such as participants, study design, and authorship. The system is evaluated with high-recall information retrieval metrics (e.g., Recall@1, Precision@r%) to assess its effectiveness in identifying clinically relevant studies. By combining similarity-based retrieval with task-specific features, this work aims to advance technology-assisted review methods in the medical domain.

Research Questions

RQ1: How accurately can certain PICO-like aspects (condition, intervention, outcome) be predicted from study abstracts, titles and initial results using LLMs vs. traditional methods?

RQ2: What is the impact of threshold-based aspect extraction on aspect prediction performance (hitrate)?

RQ3: How can extracted aspects (participants, study design, duration) improve re-ranking beyond baseline similarity?

RQ4: What is the optimal weighting strategy (α, β) for combining similarity scores with predicted aspects and study features?

RQ5: Does aspect-aware re-ranking significantly improve Recall@1 compared to baseline vector search?