Enhancing Multimodal Clinical Pretraining for ICU Modality Prediction

This repository contains a PyTorch implementation of a multimodal clinical pretraining model for ICU modality prediction. Our model achieves state-of-the-art performance on the downstream task of ICU modality prediction by leveraging a pre-trained model and fine-tuning it with a novel neural network structure and loss function.

Pretraining Multimodal Mimic

Fine-tuning Multimodal Mimic for the Downstream Task

Training Customized LLM

For training the customized LLM model. Please use tmux

``` tmux new -s sessionname tmux ls tmux a -t sessionname time python experiments/measurementnotes/measurementnotesllm.py > trainlog.txt 2>&1 Control+B D

tail -f train_log.txt ```

Training Traditional Models

For training the traditional ML model, please use Makefile.

Developer

The entire project structure should be like below:

• Download pre-trained model from aisuko/in-hospital-motality-6-48-contrast-learning and put it into exp_outputs/multimodal-mimic-3-pretraining-epoch-200
• Download in-hospital-motality-6-48.tar.gz dataset from above project and put them into the root path
• Download raw-mimic3.tar.gz raw data put the folder into the root path
• Download valset.tar.gz and put it into multimodal_clinical_pretraining/resources/

``` ubuntu@ip:~/workspace/multimodal-mimic3-pretraining-epoch200$ tree -L 2 . ├── CITATION.cff ├── Makefile ├── README.md ├── READMEMODELARCH.md ├── READMlog.md ├── cost-time.md ├── documents │ └── dataset.md ├── exp │ └── in-hospital-mortality ├── expoutputs │ └── multimodal-mimic-3-pretraining-epoch-200 ├── experiments │ └── measurementnotes ├── imgs │ ├── W&B Chart 332025, 112437 am.png │ ├── W&B Chart 332025, 112750 am.png │ ├── W&B Chart 332025, 112812 am.png │ ├── W&B Chart 732025, 103454 am.png │ ├── W&B Chart 732025, 103512 am.png │ ├── W&B Chart 732025, 103533 am.png │ ├── W&B Chart 732025, 103544 am.png │ ├── W&B Chart 732025, 105050 am.png │ ├── W&B Chart 732025, 105357 am.png │ ├── W&B Chart 732025, 105850 am.png │ ├── W&B Chart 732025, 105857 am.png │ ├── W&B Chart 732025, 105902 am.png │ ├── resultofevaluationds.png │ └── trainingtime.png ├── in-hospital-mortality-12 │ ├── test │ ├── testlistfile.csv │ ├── train │ ├── trainlistfile.csv │ └── vallistfile.csv ├── in-hospital-mortality-18 │ ├── test │ ├── testlistfile.csv │ ├── train │ ├── trainlistfile.csv │ └── vallistfile.csv ├── in-hospital-mortality-24 │ ├── test │ ├── testlistfile.csv │ ├── train │ ├── trainlistfile.csv │ └── vallistfile.csv ├── in-hospital-mortality-30 │ ├── 1percenttestlistfile.csv │ ├── 1percenttrainlistfile.csv │ ├── 1percentvallistfile.csv │ ├── test │ └── train ├── in-hospital-mortality-36 │ ├── 1percenttestlistfile.csv │ ├── 1percenttrainlistfile.csv │ ├── 1percentvallistfile.csv │ ├── test │ └── train ├── in-hospital-mortality-42 │ ├── 1percenttestlistfile.csv │ ├── 1percenttrainlistfile.csv │ ├── 1percentvallistfile.csv │ ├── test │ └── train ├── in-hospital-mortality-48 │ ├── test │ ├── testlistfile.csv │ ├── train │ ├── trainlistfile.csv │ └── vallistfile.csv ├── in-hospital-mortality-6 │ ├── test │ ├── testlistfile.csv │ ├── train │ ├── trainlistfile.csv │ └── vallistfile.csv ├── in-hospital-mortality-6-48.tar.gz ├── logs │ ├── 12hlog5dec.txt │ ├── trainlog36600.txt │ └── trainlogs4824nov.txt ├── mimic3-benchmarks │ ├── createdecompensation.py │ ├── createinhospitalmortality.py │ ├── createlengthofstay.py │ ├── createmultitask.py │ ├── createphenotyping.py │ ├── extractepisodesfromsubjects.py │ ├── in-hospital-mortality │ ├── in-hospital-mortality-downstream │ └── root ├── multimodalclinicalpretraining │ ├── init.py │ ├── pycache │ ├── data │ ├── distributedutils.py │ ├── loss.py │ ├── models │ ├── optim │ ├── pretrain │ ├── resources │ ├── scheduler │ └── utils.py ├── raw-mimic3 │ ├── ICUSTAYS.csv │ └── NOTEEVENTS.csv ├── scripts │ └── calculateexecutiontime.sh ├── testnotesdataset.pkl ├── trainnotesdataset.pkl ├── valnotesdataset.pkl └── wandb ├── debug-internal.log -> run-20250304100151-bqulgoqf/logs/debug-internal.log ├── debug.log -> run-20250304100151-bqulgoqf/logs/debug.log ├── latest-run -> run-20250304100151-bqulgoqf ├── run-20250302051114-nnfq92sr ├── run-20250302231213-6odzmeub ├── run-20250302231826-g8u7nzsm ├── run-20250304025141-5o65hj3j ├── run-20250304045655-v46aka9n ├── run-20250304061911-c5pnhukq ├── run-20250304062932-t2zgvzww ├── run-20250304064307-m5ss0f6h ├── run-20250304064926-em2k41io ├── run-20250304070123-fcbuonjr ├── run-20250304070611-stzzyoax ├── run-20250304071730-t5s3jpn9 ├── run-20250304072430-6jpgoob4 ├── run-20250304073736-32tqbycx ├── run-20250304074443-13w4jjnl ├── run-20250304075835-o3mnqra5 ├── run-20250304084711-z0on6zav └── run-20250304100151-bqulgoqf

69 directories, 117 files ```

Citation

bibtex @software{Li_Clinical_Learning_for_2024, author = {Li, Bowen}, doi = {<>}, month = dec, title = {{Clinical Learning for Early Recognition}}, url = {https://github.com/Aisuko/clear}, version = {1.0.0}, year = {2024} }

Acknowledgements

• Ryan King etc al.
• YerevaNN

Thanks for your contribution.