A team examined explanations generated by a large language model (LLM) for its performance of a clinical prediction task. They had found that, after fine-tuning, the LLM, GPT-4o from San Francisco-based OpenAI, performed comparably to four more traditional types of machine learning (ML) for predicting which patients would discontinue their home cancer medications before planned treatment completion.
From the Department of Biomedical Informatics, research fellow Congning Ni, PhD, Associate Professor Zhijun Yin, PhD, and colleagues reported their findings in the e-book series “Studies in Health Technology and Informatics.” The team used electronic health records and pharmacy surveys from 2,364 cancer patients.
The LLM achieved an F1 score of 87%, while the closest ML model scored 83%. For interpreting the latter model, the team used SHAP, or Shapley additive explanations, a widely used method for exploring (among other things) the internal structure of ML models, revealing how features are weighed. For interpreting the much larger GPT-4o, they asked the LLM to explain its reasoning for individual predictions; to derive feature-importance scores from this output, they used a new method they call mimic-SHAP.
The two models were found to agree on top features — body mass index and age. For secondary features, the LLM was found to lean more on patients’ prior conditions, the ML model on drug exposures and health care procedures.
Many cancer patients discontinue medications taken at home early, for reasons ranging from side effects and lack of response to nonmedical issues such as costs. Predicting early discontinuation could aid efforts to improve treatment adherence.
Others on the study from Vanderbilt include Qingyuan Song, Qingxia Chen, PhD, Lijun Song, PhD, S. Trent Rosenbloom, MD, MPH, Autumn Zuckerman, PharmD, Bridget Lynch, PharmD, MS, and Bradley Malin, PhD. They were joined by Jeremy Warner, MD, MS, of Brown University. The study was supported by National Institutes of Health award R37CA237452.
Leave a Reply