A toolkit to evaluate neural network explanations

PyTorch and TensorFlow [](https://colab.research.google.com/github/understandable-machine-intelligence-lab/Quantus/blob/main/tutorials/Tutorial_ImageNet_Example_All_Metrics.ipynb) [](https://mybinder.org/v2/gh/understandable-machine-intelligence-lab/Quantus/HEAD?labpath=tutorials)  [](https://badge.fury.io/py/quantus) [](https://github.com/psf/black) [](https://quantus.readthedocs.io/en/latest/?badge=latest) [](https://codecov.io/github/understandable-machine-intelligence-lab/Quantus?branch=master) [](https://pepy.tech/project/quantus) _Quantus is currently under active development so carefully note the Quantus release version to ensure reproducibility of your work._ [📑 Shortcut to paper!](https://jmlr.org/papers/volume24/22-0142/22-0142.pdf) If you want to contribute/ improve/ extend Quantus, join our [Discord](https://discord.gg/HB77krUE)! ## News and Highlights! :rocket: - 🐼 For **training data attribution** evaluation, check out [quanda](https://github.com/dilyabareeva/quanda)! - New [batch implementation](https://github.com/understandable-machine-intelligence-lab/Quantus/pull/351) for 12X speedup of existing faithfulness metrics (!) - New metrics added: [EfficientMPRT](https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/quantus/metrics/randomisation/efficient_mprt.py) and [SmoothMPRT](https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/quantus/metrics/randomisation/smooth_mprt.py) by [Hedström et al., (2023)](https://openreview.net/pdf?id=vVpefYmnsG) - Accepted to Journal of Machine Learning Research (MLOSS), read the [paper](https://jmlr.org/papers/v24/22-0142.html) - Offers more than **35+ metrics in 6 categories** for XAI evaluation - Supports different data types (image, time-series, tabular, NLP next up!) and models (PyTorch, TensorFlow) - Extended built-in support for explanation methods ([captum](https://captum.ai/), [tf-explain](https://tf-explain.readthedocs.io/en/latest/) and [zennit](https://github.com/chr5tphr/zennit)) ## Citation If you find this toolkit or its companion paper [**Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond**](https://jmlr.org/papers/v24/22-0142.html) interesting or useful in your research, use the following Bibtex annotation to cite us: ```bibtex @article{hedstrom2023quantus, author = {Anna Hedstr{\"{o}}m and Leander Weber and Daniel Krakowczyk and Dilyara Bareeva and Franz Motzkus and Wojciech Samek and Sebastian Lapuschkin and Marina Marina M.{-}C. H{\"{o}}hne}, title = {Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond}, journal = {Journal of Machine Learning Research}, year = {2023}, volume = {24}, number = {34}, pages = {1--11}, url = {http://jmlr.org/papers/v24/22-0142.html} } ``` When applying the individual metrics of Quantus, please make sure to also properly cite the work of the original authors (as linked below). ## Table of contents * [Library overview](#library-overview) * [Installation](#installation) * [Getting started](#getting-started) * [Tutorials](#tutorials) * [Contributing](#contributing) ## Library overview A simple visual comparison of eXplainable Artificial Intelligence (XAI) methods is often not sufficient to decide which explanation method works best as shown exemplarily in Figure a) for four gradient-based methods — Saliency ([Mørch et al., 1995](https://ieeexplore.ieee.org/document/488997); [Baehrens et al., 2010](https://www.jmlr.org/papers/volume11/baehrens10a/baehrens10a.pdf)), Integrated Gradients ([Sundararajan et al., 2017](http://proceedings.mlr.press/v70/sundararajan17a/sundararajan17a.pdf)), GradientShap ([Lundberg and Lee, 2017](https://arxiv.org/abs/1705.07874)) or FusionGrad ([Bykov et al., 2021](https://arxiv.org/abs/2106.10185)), yet it is a common practice for evaluation XAI methods in absence of ground truth data. Therefore, we developed Quantus, an easy-to-use yet comprehensive toolbox for quantitative evaluation of explanations — including 30+ different metrics.

With Quantus, we can obtain richer insights on how the methods compare e.g., b) by holistic quantification on several evaluation criteria and c) by providing sensitivity analysis of how a single parameter e.g. the pixel replacement strategy of a faithfulness test influences the ranking of the XAI methods.

Metrics

This project started with the goal of collecting existing evaluation metrics that have been introduced in the context of XAI research — to help automate the task of XAI quantification. Along the way of implementation, it became clear that XAI metrics most often belong to one out of six categories i.e., 1) faithfulness, 2) robustness, 3) localisation 4) complexity 5) randomisation (sensitivity) or 6) axiomatic metrics. The library contains implementations of the following evaluation metrics:

Additional metrics will be included in future releases. Please open an issue if you have a metric you believe should be apart of Quantus.

Disclaimers. It is worth noting that the implementations of the metrics in this library have not been verified by the original authors. Thus any metric implementation in this library may differ from the original authors. Further, bear in mind that evaluation metrics for XAI methods are often empirical interpretations (or translations) of qualities that some researcher(s) claimed were important for explanations to fulfil, so it may be a discrepancy between what the author claims to measure by the proposed metric and what is actually measured e.g., using entropy as an operationalisation of explanation complexity. Please read the user guidelines for further guidance on how to best use the library.

Installation

If you already have PyTorch or TensorFlow installed on your machine, the most light-weight version of Quantus can be obtained from PyPI as follows (no additional explainability functionality or deep learning framework will be included):

setup pip install quantus Alternatively, you can simply add the desired deep learning framework (in brackets) to have the package installed together with Quantus. To install Quantus with PyTorch, please run: setup pip install "quantus[torch]"

For TensorFlow, please run:

setup pip install "quantus[tensorflow]"

Package requirements

The package requirements are as follows: python>=3.8.0 torch>=1.11.0 tensorflow>=2.5.0

Please note that the exact PyTorch and/ or TensorFlow versions to be installed depends on your Python version (3.8-3.11) and platform (darwin, linux, …). See [project.optional-dependencies] section in the pyproject.toml file.

Getting started

The following will give a short introduction to how to get started with Quantus. Note that this example is based on the PyTorch framework, but we also support TensorFlow, which would differ only in the loading of the model, data and explanations. To get started with Quantus, you need: * A model (model), inputs (x_batch) and labels (y_batch) * Some explanations you want to evaluate (a_batch)

Step 2. Load explanations

We still need some explanations to evaluate. For this, there are two possibilities in Quantus. You can provide either: 1. a set of re-computed attributions (`np.ndarray`) 2. any arbitrary explanation function (`callable`), e.g., the built-in method `quantus.explain` or your own customised function We show the different options below. #### Using pre-computed explanations Quantus allows you to evaluate explanations that you have pre-computed, assuming that they match the data you provide in `x_batch`. Let's say you have explanations for [Saliency](https://arxiv.org/abs/1312.6034) and [Integrated Gradients](https://arxiv.org/abs/1703.01365) already pre-computed. In that case, you can simply load these into corresponding variables `a_batch_saliency` and `a_batch_intgrad`: ```python a_batch_saliency = load("path/to/precomputed/saliency/explanations") a_batch_intgrad = load("path/to/precomputed/intgrad/explanations") ``` Another option is to simply obtain the attributions using one of many XAI frameworks out there, such as [Captum](https://captum.ai/), [Zennit](https://github.com/chr5tphr/zennit), [tf.explain](https://github.com/sicara/tf-explain), or [iNNvestigate](https://github.com/albermax/innvestigate). The following code example shows how to obtain explanations ([Saliency](https://arxiv.org/abs/1312.6034) and [Integrated Gradients](https://arxiv.org/abs/1703.01365), to be specific) using [Captum](https://captum.ai/): ```python import captum from captum.attr import Saliency, IntegratedGradients # Generate Integrated Gradients attributions of the first batch of the test set. a_batch_saliency = Saliency(model).attribute(inputs=torch.tensor(x_batch, dtype=torch.float32), target=torch.tensor(y_batch, dtype=torch.int64), abs=True).sum(axis=1).cpu().cpu().numpy() a_batch_intgrad = IntegratedGradients(model).attribute(inputs=torch.tensor(x_batch, dtype=torch.float32), target=torch.tensor(y_batch, dtype=torch.int64)).sum(axis=1).cpu().numpy() # Quick assert. assert [isinstance(obj, np.ndarray) for obj in [x_batch, y_batch, a_batch_saliency, a_batch_intgrad]] ``` #### Passing an explanation function If you don't have a pre-computed set of explanations but rather want to pass an arbitrary explanation function that you wish to evaluate with Quantus, this option exists. For this, you can for example rely on the built-in `quantus.explain` function to get started, which includes some popular explanation methods (please run `quantus.available_methods()` to see which ones). Examples of how to use `quantus.explain` or your own customised explanation function are included in the next section. As seen in the above image, the qualitative aspects of explanations may look fairly uninterpretable --- since we lack ground truth of what the explanations should be looking like, it is hard to draw conclusions about the explainable evidence. To gather quantitative evidence for the quality of the different explanation methods, we can apply Quantus.

Please see Getting started tutorial to run code similar to this example. For more information on how to customise metrics and extend Quantus' functionality, please see Getting started guide.

Tutorials

Further tutorials are available that showcase the many types of analysis that can be done using Quantus. For this purpose, please see notebooks in the tutorials folder which includes examples such as: * All Metrics ImageNet Example: shows how to instantiate the different metrics for ImageNet dataset * Metric Parameterisation Analysis: explores how sensitive a metric could be to its hyperparameters * Robustness Analysis Model Training: measures robustness of explanations as model accuracy increases * Full Quantification with Quantus: example of benchmarking explanation methods * Tabular Data Example: example of how to use Quantus with tabular data * Quantus and TensorFlow Data Example: showcases how to use Quantus with TensorFlow

... and more.

Contributing

We welcome any sort of contribution to Quantus! For a detailed contribution guide, please refer to Contributing documentation first.

If you have any developer-related questions, please open an issue or write us at hedstroem.anna@gmail.com.