CS 598: Deep Learning For Healthcare Final Project

By: Armaan R. Butt and Harikrishna Bojja {arbutt2, hbojja2}@illinois.edu

Group ID: 213, Paper ID: 283

Paper: Categorization of free-text drug orders using character-level recurrent neural networks [1]

Dependencies

Computer Specs

You will need a machine with the following specs:

• CPU: 2.9 GHz - 8 Cores
• Memory: 64 GB

Runtime

You will need a machine with Python 3.9.7 installed.

Python Libraries

Please have the following Python libraries installed. We have provided the requirements.txt file in the project root for your convenience.

keras==2.8.0 matplotlib==3.5.1 nltk==3.7 numpy==1.22.0 pandarallel==1.6.1 pandas==1.4.2 scikit_learn==1.0.2 tensorflow==2.8.0

Data Download Instructions

Please download the data from https://physionet.org/content/mimiciii-demo/1.4/ and extract the NOTEEVENTS.csv and PRESCRIPTIONS.csv to /data/real-mimic-iii-database.

Exploratory Data Analysis(EDA)

The jupyter notebooks below shows how we have used Seaborn and Matplotlib to analyze the Precriptions and Noteevents data.

1. /src/data_profiling/profile_prescription_data.ipynb
2. /src/data_profiling/profile_free_text.ipynb

Data Pre-Processing Code

Run the following jupyter notebooks in order. It takes approximately 1 hour to finish the data preprocessing.

1. /src/data_processing/extract_drug_codes.ipynb
2. /src/data_processing/note_events_processing.ipynb

Once complete it will generate two new files in /data/processed/:

• NOTEEVENTS_ML_DATASET.csv
• ndc_codes_extracted.csv

Train and Evaluate Models Code

To train and evaluate the SVM and GRU Models run the following jupyter notebooks:

1. /src/ml/multi_class_svm.ipynb
2. /src/ml/gru_model.ipynb

Results will be persisted in two csvs in the /data/results directory.

• GRU_RESULTS.csv
• SVM_results.csv

Results

Baseline SVM

The baseline model SVM model was trained on the top 22 common drugs in our dataset (NDC). The SVM model used a linear kernel (LinearSVC) with the input text data being vectorized at the character level using TfidVectoriezer using scikit-learn. TfidVectoriezer was configure to generate trigrams from the text data.

| NDC | Accuracy (%) | Precision (%) | Recall (%) | | ----------- | ------------ | ------------- | ---------- | | 00713016550 | 90 | 83 | 76 | | 00487950125 | 92 | 89 | 76 | | 00517391025 | 92 | 92 | 91 | | 51079001920 | 90 | 90 | 91 | | 11098003002 | 95 | 88 | 44 | | 00054829725 | 93 | 88 | 72 | | 00045025501 | 98 | 80 | 5 | | 00338055002 | 86 | 88 | 91 | | 00409131230 | 94 | 90 | 59 | | 00045152510 | 93 | 87 | 65 | | 00074407532 | 87 | 87 | 77 | | 51079080120 | 91 | 90 | 85 | | 51079025520 | 91 | 90 | 85 | | 00074176201 | 87 | 86 | 71 | | 00781305714 | 97 | 87 | 13 | | 00054465025 | 95 | 86 | 65 | | 00008084199 | 94 | 91 | 74 | | 58177000104 | 93 | 95 | 91 | | 00781188313 | 96 | 93 | 55 | | 00517293025 | 93 | 94 | 95 | | 00338355248 | 90 | 84 | 71 | | 00002735501 | 89 | 89 | 83 | | Average | 92 | 88 | 70 |

GRU - RNN

| Model | Hidden State Size | Number of Epochs | Mean Training Accuracy (%) | Mean Test Accuracy (%) | Mean Training Loss (%) | Mean Test Loss (%) | | ----------------- | ----------------- | ---------------- | -------------------------- | ---------------------- | ---------------------- | ------------------ | | Bidirectional GRU | 32 | 3 | 75.44 | 75.69 | 56.44 | 55.49 | | Bidirectional GRU | 64 | 3 | 75.45 | 75.69 | 56.22 | 55.52 | | Bidirectional GRU | 128 | 3 | 75.44 | 75.69 | 56.26 | 55.66 |

References

[1] Raiskin Y, Eickhoff C, Beeler PE. Categorization of free-text drug orders using character-level recurrent neural networks. Int J Med Inform. 2019 Sep;129:20-28. doi: 10.1016/j.ijmedinf.2019.05.020. Epub 2019 May 23. PMID: 31445256.