Author: Jianping He, MS (2025)
Primary advisor: Degui Zhi, PhD
Committee members: Cui Tao, PhD, Hua Xu, PhD and Laila Rasmy, PhD
PhD thesis: McWilliams School of Biomedical Informatics at UTHealth Houston.
ABSTRACT
Enhancements in foundation models have provided significant potentials for clinical applications. This dissertation leverages and advances foundation models for a range of clinical tasks using electronic health record (EHR) data, demonstrating their applicability in Clinical Temporal Relation Extraction (CTRE), disease prediction, and patients trajectory analysis. Aim 1 is to efficiently adapt large language models (LLMs) for CTRE in both full data and few-shot settings. This study leveraged four LLMs: GatorTron-Base, GatorTron-Large, LLaMA3.1, and MeLLaMA. The proposed fine-tuning strategies include: (a) standard fine-tuning; (b) hard-prompting; (c) soft-prompting; (d) Low-Rank Adaptation (LoRA). We found that nearly all proposed fine-tuning strategies outperformed existing state-of-the-art (SOTA) methods on the CTRE task, with improvements in micro-F1 scores ranging from 1.11% to 3.86%, except for soft prompting, which yielded suboptimal results. Our findings also offer practical insights for the efficient adaptation of foundation models under varying computational and data constraints. For instance, hard-prompting outperformed standard fine-tuning, and selectively updating only the query and value layers of the Transformer architecture was more effective than tuning all linear layers. Moreover, Decoder-based models demonstrated superior performance when ample training data or trainable parameters were available. In contrast, Encoder-based models performed better in low-resource settings, likely due to their domain-specific pretraining and architectural advantages for classification tasks.
Aim 2 is to advance disease prediction performance using the clinical foundation model Med-BERT. In this aim, we reformulated the conventional binary disease prediction task into two alternative formulations: a token prediction task and a next-visit mask token prediction task, both designed to align with Med-BERT’s pretraining objective. This reformulation led to improved accuracy in pancreatic cancer prediction across both few-shot scenarios and large-scale data settings. These findings demonstrate that aligning downstream tasks with the pretraining format of foundation models can facilitate more efficient and effective adaptation to specific clinical prediction tasks.?
Aim 3 seeks to advance disease trajectory analysis by leveraging the clinical foundation model Med-BERT in conjunction with evolving and dynamic patient medical histories. This study focused on a cohort of patients with mild cognitive impairment (MCI) and proposed a comprehensive framework for modeling disease progression using longitudinal electronic health record (EHR) data. Specifically, we integrated BiGRU with Med-BERT to generate trajectory until visit level patient representations that capture the cumulative medical history up to each clinical encounter. These representations were then used to perform clustering and pseudotime analyses, enabling the trajectory analysis of MCI patients. The proposed workflow offers a scalable approach for characterizing temporal disease dynamics and can be applied to other disease cohorts. Additionally, our analysis identified several factors potentially associated with an decreased risk of developing Alzheimer’s disease (AD), consistent with findings from prior studies—thus providing external validation for our framework.
This study introduces key insights that enhance both the effectiveness and efficiency of adapting these models to diverse downstream tasks. Furthermore, the methodological advancements presented—spanning improvements in CTRE performance, disease prediction accuracy, and the uncovering of hidden patterns in disease progression—collectively hold promise for informing personalized interventions and ultimately improving patient outcomes.