UMIE_datasets

🤩 About the Project

Warning: This project is currently in alpha stage and may be subject to major changes

This repository presents a suite of unified scripts to standardize, preprocess, and integrate 882,774 images from 20 open-source medical imaging datasets, spanning modalities such as X-ray, CT, and MR. The scripts allow for seamless and fast download of a diverse medical data set. We create a unified set of annotations allowing for merging the datasets together without mislabelling. Each dataset is preprocessed with a custom sklearn pipeline. The pipeline steps are reusable across the datasets. The code was designed so that preorocessing a new dataset is simple and requires only reusing the available pipeline steps with customization performed through setting the appropriate values of the pipeline params.

The labels and segmentation masks were unified to be compliant with RadLex ontology.

Datasets

| uid | Dataset | Modality | TASK | |:---:|:--------------------------------------------------------------------------------------------------------------------------------------------:|:--------:|:----------------------------:| | 0 | KITS-23 | CT | Classification/Segmentation | | 1 | CoronaHack | XRAY | Classification | | 2 | Alzheimers Dataset | MRI | Classification | | 3 | Brain Tumor Classification | MRI | Classification | | 4 | COVID-19 Detection X-Ray | XRAY | Classification | | 5 | Finding and Measuring Lungs in CT Data | CT | Segmentation | | 6 | Brain CT Images with Intracranial Hemorrhage Masks | CT | Classification | | 7 | Liver and Liver Tumor Segmentation | CT | Classification, Segmentation | | 8 | Brain MRI Images for Brain Tumor Detection | MRI | Classification | | 9 | Knee Osteoarthritis Dataset with Severity Grading | XRAY | Classification | | 10 | Brain Tumor Progression | MRI | Segmentation | | 11 | Chest X-ray 14 | XRAY | Classification | | 12 | COCA- Coronary Calcium and chest CTs | CT | Segmentation | | 13 | BrainMetShare | MRI | Segmentation | | 14 | CT-ORG | CT | Segmentation | | 17 | LIDC-IDRI | CT | Segmentation | | 18 | CMMD | MG | Classification |

Using the datasets

Installing requirements

bash poetry install

Creating the dataset

Due to the copyright restrictions of the source datasets, we can't share the files directly. To obtain the full dataset you have to download the source datasets yourself and run the preprocessing scripts.

To preprocess the dataset that is not among the above, search the preprocessing folder. It contains the reusable steps for changing imaging formats, extracting masks, creating file trees, etc. Go to the config file to check which masks and label encodings are available. Append new labels and mask encodings if needed.

Overall the dataset should have 882,774 images in .png format * CT - 500k+ * X-Ray - 250k+ * MRI - 100k+

🎯 Roadmap

• [X] dcm
• [x] jpg
• [x] nii
• [x] tif
• [x] Shared radlex ontology
• [ ] Huggingface datasets
• [ ] Data dashboards

:wave: Contributors

:handshake: Contact

Barbara Klaudel

TheLion.AI

Development

Pre-commits

Install pre-commits https://pre-commit.com/#installation

If you are using VS-code install the extention https://marketplace.visualstudio.com/items?itemName=MarkLarah.pre-commit-vscode

To make a dry-run of the pre-commits to see if your code passes run pre-commit run --all-files

Adding python packages

Dependencies are handeled by poetry framework, to add new dependency run poetry add <package_name>

Debugging

To modify and debug the app, development in containers can be useful .

Testing

bash run_tests.sh