Table of Contents

• Summary
• AUDIT workflow
• AUDIT analysis modes
• Online Web application
• Getting Started
  • 1 Installation
  • 2 Configuration
  • 2.1 Feature extraction config file
  • 2.2 Metric extraction config file
  • 2.3 APP config file
  • 3 Run AUDIT backend
  • 4 Run AUDIT app
  • 5 Additional configurations
  • 5.1 ITK-Snap
    • 5.1.1 On Mac OS
    • 5.1.2 On Linux OS
• Authors
• License

Summary

AUDIT, Analysis & evalUation Dashboard of artIficial inTelligence, is a tool designed to provide researchers and developers with an interactive way to better analyze and explore MRI datasets and segmentation models. Given its functionalities to extract the most relevant features and metrics from your multiple data sources, it allows for uncovering biases both intra and inter-dataset as well as in the model predictions.

Details of our work are provided in our paper AUDIT: An open-source Python library for AI model evaluation with use cases in MRI brain tumor segmentation. We hope that users will leverage AUDIT to gain novel insights into the field of medical image segmentation.

AUDIT workflow

The diagram below illustrates the overall workflow of AUDIT, from input data to data visualization on the APP.

For more details, please refer to the AUDIT paper.

AUDIT analysis modes:

• Home Page: The main landing page of the tool.
• Univariate: Exploration of individual features to understand how they are distributed.
• Multivariate: Analysis of multiple features simultaneously to explore relationships and hidden patterns.
• Segmentation error matrix: Overview of disagreements between ground truth and predicted segmentation through a class-wise error matrix.
• Single model performance: Evaluation of the performance of a single model based on extracted features.
• Pairwise model performance: Perform pairwise comparisons between models to find statistically significant differences.
• Multi-model performance: Comparative analysis of performance metrics across multiple models.
• Longitudinal measurements: Analysis of data collected over time to observe trends and changes in model performance.
• Subjects' exploration: Detailed examination of individual subjects within the dataset.

Online Web application

Last released version of AUDIT is hosted at https://auditapp.streamlitapp.com for an online overview of its functionalities.

Getting Started

AUDIT can be installed either from our GitHub repository or from PyPI using the command pip install auditapp. In this guide, we will show how to install it from the GitHub repository. For a more detailed exploration of AUDIT, please check our official documentation.

1 Installation

Create an isolated Anaconda environment (recommended to avoid dependency conflicts):

bash conda create -n audit_env python=3.10 conda activate audit_env

Clone the repository: bash git clone https://github.com/caumente/AUDIT.git cd AUDIT

Install the required packages: bash pip install -r requirements.txt

2. Configuration

Edit the config files in ./src/audit/configs/ directory to set up the paths for data loading and other configurations:

2.1 Feature extraction config file

This configuration file is used to set paths, labels, features, and longitudinal study parameters for feature extraction in AUDIT.

2.2 Metric extraction config file

This configuration file defines the paths to datasets and model predictions, the labels, metrics to compute, and output paths for metric extraction in AUDIT.

2.3 APP config file

This configuration file sets up paths for datasets, extracted features, metrics, and model predictions. It also defines available sequences and label mappings used by the AUDIT application.

3. Run AUDIT backend

Use the following commands in your terminal to run the Feature extraction and Metric extraction scripts:

bash python src/audit/feature_extraction.py

bash python src/audit/metric_extraction.py

After running either script, a logs folder will be created to keep track of the execution. All output files will be stored in the folder defined in the corresponding config file (by default, in the outputs folder).

4. Run AUDIT app

AUDIT app is built on top of the Streamlit library. Use the following command to run the app and start exploring your data:

bash python src/audit/app/launcher.py

5. Additional configurations

5.1. ITK-Snap

AUDIT can be adjusted for opening cases with ITK-Snap while exploring the data in the different dashboards. The ITK-Snap tool must have been installed and preconfigured before. Below are placeholders for the necessary configuration for each operating system. Information will be added soon.

5.1.1. On Mac OS

5.1.2. On Linux OS

Authors

Please feel free to contact us with any issues, comments, or questions.

• Carlos Aumente (UO297103@uniovi.es)
• Mauricio Reyes
• Michael Muller
• Jorge Dez
• Beatriz Remeseiro

License

Apache License 2.0