TERP

Thermodynamically Explainable Representations of AI and other black-box Paradigms

Detailed tutorial showing use of TERP and how to interpret results can be found here: https://github.com/shams-mehdi/TERP_tutorial/tree/main

TERP is a post-hoc interpretation scheme for explaining black-box AI predictions. TERP works by constructing a linear, local interpretable model that approximates the black-box in the vicinity of the instance being explained. TERP determines the accuracy-interpretability trade-off by introducing and using the concept of interpretation entropy. Essential packages for this python implementation are listed in requirements.txt file.

A simple Google Colab notebook demonstrating this Python implementation for tabular data is provided (TERPtabularexample.ipynb)

Getting TERP interpretation for numeric data type involves the following steps:

1. Feature selection: Generate neighborhood using TERPneighborhoodgenerator.py ``` !python TERPneighborhoodgenerator.py -seed 0 --progressbar -inputnumeric inputdata.npy -numsamples 5000 -index 102975

-index # row index of the input (e.g, inputnumeric) file whose prediction needs to be explained -seed # random seed -inputnumeric # location of a numpy array with a representative distribution of the black-box model training data e.g, the training data itself -numsamples # size of the generated neighborhood ``` 2. obtain black-box model prediction by passing generated neighborhood saved at DATA/makepredictionnumeric.npy and save the predicted results in a numpy array. Rows of this array should represent datapoints and columns should represent different classes (e.g, neighborhoodstateprobabilities.npy - see next step). Note: A numpy array DATA/TERPnumeric.npy is also created which will be used in the next step 3. Form a feature sub-space by identifying irrelevant features using a single round of linear regression ``` !python TERPoptimizer01.py -TERPinput DATA/TERPnumeric.npy -blackboxprediction DATA/neighborhoodstate_probabilities.npy

-TERPinput # Standardized neighborhood data location -blackboxprediction # Prediction probabilities for different classes as obtained from the black-box model

Note: A numpy array selectedfeatures.npy will be created at TERPresults/selectedfeatures.npy 4. Generate a neighborhood by sampling the reduced feature space for improved interpretation !python TERPneighborhoodgenerator.py -seed 0 --progressbar -inputnumeric inputdata.npy -numsamples 5000 -index 102975 -selectedfeatures TERPresults/selectedfeatures.npy

All the options are the same as in step 1. However, pass the additional -selectedfeatures flag to analyze the sub-space only ``` 5. Obtain black-box model prediction by passing generated neighborhood saved at DATA2/makepredictionnumeric.npy and save the predicted results in a numpy array. Rows of this array should represent datapoints and columns should represent different classes (e.g, neighborhoodstateprobabilities.npy - see next step). Note A numpy array DATA2/TERPnumeric.npy is also created which will be used in the next step 6. Perform forward feature selection to obtain final result ``` The final result is stored at TERPresults2/optimalfeatureweights.npy. This one-dimensional array will have size equal to the number of input features and each entry represents the corresponding feature importance to a particular prediction. !python TERPoptimizer02.py -TERPinput DATA2/TERPnumeric.npy -blackboxprediction DATA2/neighborhoodstateprobabilities.npy -selectedfeatures TERPresults/selectedfeatures.npy

Note: -blackbox_prediction #prediction probabilites file for this step should be different from the one passed in step 3 because the neighborhood has been regenerated If you are using this repository, please cite the following paper: @article{mehdi2024thermodynamics, title={Thermodynamics-inspired explanations of artificial intelligence}, author={Mehdi, Shams and Tiwary, Pratyush}, journal={Nature Communications}, volume={15}, number={1}, pages={7859}, year={2024}, publisher={Nature Publishing Group UK London} } ```