SeqMetrics: a unified library for performance metrics calculation in Python

The purpose of this repository to collect various classification and regression performance metrics or errors which can be calculated for time-series/sequential/tabular data, at one place. Currently only 1-dimensional data is supported.

How to Install

You can install SeqMetrics using pip

pip install SeqMetrics

or using GitHub link for the latest code

python -m pip install git+https://github.com/AtrCheema/SeqMetrics.git

or using setup file, go to folder where repo is downloaded

python setup.py install

You can also install SeqMetrics with all of its dependencies by making use of all option

pip install SeqMetrics[all]

This will install scipy and easy_mpl libraries. The scipy library is used to calculate some additional metrics such as kendall_tau or mapeforpeaks while easy_mpl is used for plotting purpose.

How to Use

SeqMetrics provides a uniform API for calculation of both regression and classification metrics. It has a functional API and a class based API.

Regression Metrics

The use of the functional API is as straightforward as calling the required function and providing it with true and predicted arrays or array-like objects (lists, tuples, DataFrames, Series, tensors).

```python import numpy as np from SeqMetrics import nse

true = np.random.random((20, 1)) pred = np.random.random((20, 1))

nse(true, pred) # calculate Nash Sutcliff efficiency

```

The method for calling functions is consistent across all 100+ metrics.

Alternatively, the same outcome can be achieved using a class-based API.

```python import numpy as np from SeqMetrics import RegressionMetrics

true = np.random.random((20, 1)) pred = np.random.random((20, 1))

er = RegressionMetrics(true, pred)

for m in er.all_methods: print("{:20}".format(m)) # get names of all availabe methods

er.nse() # calculate Nash Sutcliff efficiency

er.calculate_all(verbose=True) # or calculate errors using all available methods ```

We can visualize the calcuated performance metrics if we have easy_mpl package installed. ```python import numpy as np from SeqMetrics import RegressionMetrics, plot_metrics

np.random.seed(313) true = np.random.random((20, 1)) pred = np.random.random((20, 1))

er = RegressionMetrics(true, pred)

plotmetrics(er.calculateall(), color="Blues") ```

RegressionMetrics currently, calculates following performane metrics for regression.

| Name | Name in this repository | | -------------------------- | ------------- | | Absolute Percent Bias | abs_pbias | | Agreement Index | agreement_index | | Aitchison Distance | aitchison | | Alpha decomposition of the NSE | nse_alpha | | Anomaly correction coefficient | acc | | Bias | bias | | Beta decomposition of NSE | nse_beta | | Bounded NSE | nse_bound | | Bounded KGE | kge_bound | | Brier Score | brier_score | | Correlation Coefficient | corr_coeff | | Coefficient of Determination | r2 | | Centered Root Mean Square Deviation | centered_rms_dev | | Covariances | covariance | | Decomposed Mean Square Error | decomposed_mse | | Explained variance score | exp_var_score | | Euclid Distance | euclid_distance | | Geometric Mean Difference | gmaen_diff | | Geometric Mean Absolute Error | gmae | | Geometric Mean Relative Absolute Error | gmrae | | Inertial Root Squared Error | irmse | | Integral Normalized Root Squared Error | inrse | | Inter-percentile Normalized Root Mean Squared Error | nrmse_ipercentile | | Jensen-shannon divergence | JS | | Kling-Gupta Efficiency | kge | | Legate-McCabe Efficiency Index | lm_index | | Logrithmic Nash Sutcliff Efficiency | log_nse | | Logrithmic probability distribution | log_prob | | maximum error | max_error | | Mean Absolute Error | mae | | Mean Absolute Percentage Deviation | mapd | | Mean Absolute Percentage Error | mape | | Mean Absolute Relative Error | mare | | Mean Absolute Scaled Error | mase | | Mean Arctangle Absolute Percentage Error | maape | | Mean Bias Error | mean_bias_error | | Mean Bounded relative Absolute Error | mbrae | | Mean Errors | me | | Mean Gamma Deviances | mean_gamma_deviance | | Mean Log Error | mle | | Mean Normalized Root Mean Square Error | nrmse_mean | | Mean Percentage Error | mpe | | Mean Poisson Deviance | mean_poisson_deviance | | Mean Relative Absolute Error | mrae | | Mean Square Error | mse | | Mean Square Logrithmic Errors | mean_square_log_error | | Mean Variance | mean_var | | Median Absolute Error | median_abs_error | | Median Absolute Percentage Error | mdape | | Median Dictionary Accuracy | | | Median Error | mde | | Median Relative Absolute Error | mdrae | | Median Squared Error | med_seq_error | | Mielke-Berry R | mb_r | | Modified Agreement of Index | mod_agreement_index | | Modified Kling-Gupta Efficiency | kge_mod | | Modified Nash-Sutcliff Efficiency | nse_mod | | Nash-Sutcliff Efficiency | nse | | Non parametric Kling-Gupta Efficiency | kge_np | | Normalized Absolute Error | norm_ae | | Normalized Absolute Percentage Error | norm_ape | | Normalized Euclid Distance | norm_euclid_distance | | Normalized Root Mean Square Error | nrmse | | Peak flow bias of the flow duration curve | fdc_fhv | | Pearson correlation coefficient | person_r | | Percent Bias | pbias | | Range Normalized root mean square | nrmse_range | | Refined Agreement of Index | ref_agreement_index | | Relative Agreement of Index | rel_agreement_index | | Relative Absolute Error | rae | | Relative Root Mean Squared Error | relative_rmse | | Relative Nash-Sutcliff Efficiency | nse_rel | | Root Mean Square Errors | rmse | | Root Mean Square Log Error | rmsle | | Root Mean Square Percentage Error | rmspe | | Root Mean Squared Scaled Error | rmsse | | Root Median Squared Scaled Error | rmsse | | Root Relative Squared Error | rrse | | RSR | rsr | | Separmann correlation coefficient | spearmann_corr | | Skill Score of Murphy | skill_score_murphy | | Spectral Angle | sa | | Spectral Correlation | sc | | Spectral Gradient Angle | sga | | Spectral Information Divergence | sid | | Symmetric kullback-leibler divergence | KLsym | | Symmetric Mean Absolute Percentage Error | smape | | Symmetric Median Absolute Percentage Error | smdape | | sum of squared errors | sse | | Volume Errors | volume_error | | Volumetric Efficiency | ve | | Unscaled Mean Bounded Relative Absolute Error | umbrae | | Watterson's M | watt_m | | Weighted Mean Absolute Percent Errors | wmape | | Weighted Absolute Percentage Error | wape |

Classification Metrics

The API is same for performance metrics of classification problem.

```python import numpy as np from SeqMetrics import ClassificationMetrics

boolean array

t = np.array([True, False, False, False]) p = np.array([True, True, True, True]) metrics = ClassificationMetrics(t, p) print(metrics.calculate_all())

binary classification with numerical labels

true = np.array([1, 0, 0, 0]) pred = np.array([1, 1, 1, 1]) metrics = ClassificationMetrics(true, pred) print(metrics.calculate_all())

multiclass classification with numerical labels

true = np.random.randint(1, 4, 100) pred = np.random.randint(1, 4, 100) metrics = ClassificationMetrics(true, pred) print(metrics.calculate_all())

You can also provide logits instead of labels.

predictions = np.array([[0.25, 0.25, 0.25, 0.25], [0.01, 0.01, 0.01, 0.96]]) targets = np.array([[0, 0, 0, 1], [0, 0, 0, 1]]) metrics = ClassificationMetrics(targets, predictions, multiclass=True) print(metrics.calculate_all())

Working with categorical values is seamless

true = np.array(['a', 'b', 'b', 'b']) pred = np.array(['a', 'a', 'a', 'a']) metrics = ClassificationMetrics(true, pred) print(metrics.calculate_all())

same goes for multiclass categorical labels

t = np.array(['car', 'truck', 'truck', 'car', 'bike', 'truck']) p = np.array(['car', 'car', 'bike', 'car', 'bike', 'truck']) metrics = ClassificationMetrics(targets, predictions, multiclass=True) print(metrics.calculate_all()) ```

SeqMetrics library currently calculates following performance metrics of classification.

| Name | Name in this repository | | -------------------------- | ------------- | | Accuracy | accuracy | | Balanced Accuracy | balanced_accuracy | | Error Rate | error_rate | | Recall | recall | | Precision | precision | | F1 score | f1_score | | F2 score | f2_score | | Specificity | specificity | | Cross Entropy | cross_entropy | | False Positive Rate | false_positive_rate | | False Negative Rate | false_negative_rate | | False Discovery Rate | false_discovery_rate | | False Omission Rate | false_omission_rate | | Negative Predictive Value | negative_predictive_value | | Positive Likelihood Ratio | positive_likelihood_ratio | | Negative Likelihood Ratio | negative_likelihood_ratio | | Prevalence Threshold | prevalence_threshold | | Youden Index | youden_index | | Confusion Matrix | confusion_matrix | | Fowlkes Mallows Index | fowlkes_mallows_index | | Mathews correlation Coefficient | mathews_corr_coeff |

Web App

The SeqMetrics library is available from the webapp which is deployed used stream https://seqmetrics.streamlit.app/

You can also launch the app locally if you do not wish to use the web-based app. Make sure you follow the below steps

git clone https://github.com/AtrCheema/SeqMetrics.git
cd SeqMetrics
pip install requirements.txt
pip install streamlit
streamlit run app.py

Usage of streamlit based application app involves, 1) providing the true and predicted arrays either by pasting the data in the boxes or by uploading a file, 2) Selecting the relevant performance metric and 3) calculating the performance metric. These steps are further illustrated below.

The method to provide data from a (excel/csv) file is described in below image

Related

forecasting_metrics

hydroeval

SkillMetrics

HydroErr

Keras

SickitLearn

Torchmetrics