INFDok - Dokumentenvolltextserver des Fachbereichs Informatik

Kurzfassung auf Englisch:

This work investigates the task of automated patient-trial matching through fine-tuning Llama2 chat (13B) proposing TrialLlama. TrialLlama is trained on Clinical Trials from a snapshot of the clinicaltrials.gov website and synthetic patient descriptions provided by the TREC Clinical Trials track, using a supervised classification approach. Two primary tasks are explored with this fine-tuned model: 1) patient-trial classification, where the model categorises patient-trial pairs into one of three labels (eligible, excluded, irrelevant), and 2) reasoning, where it extracts and discusses the eligibility criteria from a clinical trial to determine a patient’s eligibility to get enrolled in the corresponding trial. In the patient-trial matching task treated as a binary classification, combining the two negative labels into one class, TrialLlama achieved an accuracy of 0.813 and an F1 score of 0.883. For the original three-label classification task, TrialLlama achieved accuracy and F1-scores of 0.634 and 0.530, respectively. Notably, TrialLlama excelled in the reasoning task, exceeding Llama2 by 0.640 in precision and 0.666 in accuracy. Despite being fine-tuned for classification, TrialLlama demonstrated proficiency in extracting eligibility criteria and assessing a patient’s eligibility concisely and logically. However, several limitations are identified, including fine-tuning difficulties due to dataset limitations, a bias towards extracting inclusion criteria, hallucination issues, and comparability to other systems. Nevertheless, TrialLlama and its open-source codebase hold promise for advancing research in automated patient-trial matching and AI-driven medical assistants.