Adaptive Minds: Enhancing Domain Versatility of LLMs through Parameter-Efficient Fine-Tuning

Introduction & Motivation

The advent of Large Language Models (LLMs) has significantly simplified various natural language processing tasks through simple prompts. However, these models encounter challenges with domain-specific knowledge. For instance, while they can suggest possible diseases based on symptoms, they are unable to provide precise medical prescriptions. Additionally, they often struggle to deliver accurate results for tasks requiring an in-depth understanding of specific domains, sometimes leading to hallucinations or inaccuracies.

To address these challenges, we employ parameter-efficient fine-tuning (PEFT) techniques to refine our LLM, using LLaMA3 for the task of summarization. Our approach involves adapting the model across four distinct domains: medical, legal, scientific, and news. We experiment with various PEFT technologies to enhance the model's performance. Our goal is to extend domain adaptation strategies to better comprehend low-resource domains. By fine-tuning the LLM on multiple high-resource domains, we aim to capture the intrinsic properties of language, thereby improving performance on various NLP tasks in low-resource domains. Specifically, this work focuses on the summarization task, aiming to generate high-quality and coherent summaries for low-resource domains.

Research Questions

• R1: Can a smaller Large Language Model (LLM) adapt well to an unseen domain using a parameter-efficient fine-tuning method and perform effectively on the task of summarization?
  
  
• R2: Are there parameter-efficient fine-tuning techniques that enable an LLM to adapt well to specific domains, or do they perform effectively across multiple domains for an LLM?
  
  
• R3: Can we leverage multiple parameter-efficient fine-tuning techniques to enable an LLM to simultaneously learn and perform well across various domains?
  
  
• R4: After training an LLM on multiple high-resource domains, can it achieve strong performance on an unseen low-resource domain?

References

1. Ling, C., Zhao, X., Lu, J., Deng, C., Zheng, C., Wang, J., Chowdhury, T., Li, Y., Cui, H., Zhang, X., Zhao, T., Panalkar, A., Mehta, D., Pasquali, S., Cheng, W., Wang, H., Liu, Y., Chen, Z., Chen, H., . . . Zhao, L. (2023). Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey. ArXiv. /abs/2305.18703
   
   
2. Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., & Raffel, C. (2022). Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning. ArXiv. /abs/2205.05638
   
   
3. Afzal, A., Chalumattu, R., Matthes, F., & Espuny, L. M. (2024). AdaptEval: Evaluating Large Language Models on Domain Adaptation for Text Summarization. ArXiv. /abs/2407.11591