**Machine learning metrics for distributed, scalable PyTorch applications.** ______________________________________________________________________

What is Torchmetrics • Implementing a metric • Built-in metrics • Docs • Community • License

______________________________________________________________________ [](https://pypi.org/project/torchmetrics/) [](https://badge.fury.io/py/torchmetrics) [ ](https://pepy.tech/project/torchmetrics) [](https://anaconda.org/conda-forge/torchmetrics) [](https://github.com/Lightning-AI/torchmetrics/blob/master/LICENSE) [](https://github.com/Lightning-AI/torchmetrics/actions/workflows/ci-tests.yml) [](https://dev.azure.com/Lightning-AI/Metrics/_build/latest?definitionId=54&branchName=master) [](https://codecov.io/gh/Lightning-AI/torchmetrics) [](https://results.pre-commit.ci/latest/github/Lightning-AI/torchmetrics/master) [](https://torchmetrics.readthedocs.io/en/latest/?badge=latest) [](https://discord.gg/VptPCZkGNa) [](https://doi.org/10.5281/zenodo.5844769) [](https://joss.theoj.org/papers/561d9bb59b400158bc8204e2639dca43) ______________________________________________________________________

Installation

Simple installation from PyPI

bash pip install torchmetrics

What is TorchMetrics

TorchMetrics is a collection of 100+ PyTorch metrics implementations and an easy-to-use API to create custom metrics. It offers:

• A standardized interface to increase reproducibility
• Reduces boilerplate
• Automatic accumulation over batches
• Metrics optimized for distributed-training
• Automatic synchronization between multiple devices

You can use TorchMetrics with any PyTorch model or with PyTorch Lightning to enjoy additional features such as:

• Module metrics are automatically placed on the correct device.
• Native support for logging metrics in Lightning to reduce even more boilerplate.

Using TorchMetrics

Module metrics

The module-based metrics contain internal metric states (similar to the parameters of the PyTorch module) that automate accumulation and synchronization across devices!

• Automatic accumulation over multiple batches
• Automatic synchronization between multiple devices
• Metric arithmetic

This can be run on CPU, single GPU or multi-GPUs!

For the single GPU/CPU case:

```python import torch

import our library

import torchmetrics

initialize metric

metric = torchmetrics.classification.Accuracy(task="multiclass", num_classes=5)

move the metric to device you want computations to take place

device = "cuda" if torch.cuda.is_available() else "cpu" metric.to(device)

nbatches = 10 for i in range(nbatches): # simulate a classification problem preds = torch.randn(10, 5).softmax(dim=-1).to(device) target = torch.randint(5, (10,)).to(device)

# metric on current batch
acc = metric(preds, target)
print(f"Accuracy on batch {i}: {acc}")

metric on all batches using custom accumulation

acc = metric.compute() print(f"Accuracy on all data: {acc}") ```

Module metric usage remains the same when using multiple GPUs or multiple nodes.

Implementing your own Module metric

Implementing your own metric is as easy as subclassing an torch.nn.Module. Simply, subclass torchmetrics.Metric and just implement the update and compute methods:

```python import torch from torchmetrics import Metric

class MyAccuracy(Metric): def init(self): # remember to call super super().init() # call self.add_statefor every internal state that is needed for the metrics computations # distreducefx indicates the function that should be used to reduce # state from multiple processes self.addstate("correct", default=torch.tensor(0), distreducefx="sum") self.addstate("total", default=torch.tensor(0), distreducefx="sum")

def update(self, preds: torch.Tensor, target: torch.Tensor) -> None:
    # extract predicted class index for computing accuracy
    preds = preds.argmax(dim=-1)
    assert preds.shape == target.shape
    # update metric states
    self.correct += torch.sum(preds == target)
    self.total += target.numel()

def compute(self) -> torch.Tensor:
    # compute final result
    return self.correct.float() / self.total

my_metric = MyAccuracy() preds = torch.randn(10, 5).softmax(dim=-1) target = torch.randint(5, (10,))

print(my_metric(preds, target)) ```

Functional metrics

Similar to torch.nn, most metrics have both a module-based and functional version. The functional versions are simple python functions that as input take torch.tensors and return the corresponding metric as a torch.tensor.

```python import torch

import our library

import torchmetrics

simulate a classification problem

preds = torch.randn(10, 5).softmax(dim=-1) target = torch.randint(5, (10,))

acc = torchmetrics.functional.classification.multiclassaccuracy( preds, target, numclasses=5 ) ```

Covered domains and example metrics

In total TorchMetrics contains 100+ metrics, which covers the following domains:

• Audio
• Classification
• Detection
• Information Retrieval
• Image
• Multimodal (Image-Text)
• Nominal
• Regression
• Segmentation
• Text

Each domain may require some additional dependencies which can be installed with pip install torchmetrics[audio], pip install torchmetrics['image'] etc.

Additional features

Plotting

Visualization of metrics can be important to help understand what is going on with your machine learning algorithms. Torchmetrics have built-in plotting support (install dependencies with pip install torchmetrics[visual]) for nearly all modular metrics through the .plot method. Simply call the method to get a simple visualization of any metric!

```python import torch from torchmetrics.classification import MulticlassAccuracy, MulticlassConfusionMatrix

num_classes = 3

this will generate two distributions that comes more similar as iterations increase

w = torch.randn(num_classes) target = lambda it: torch.multinomial((it * w).softmax(dim=-1), 100, replacement=True) preds = lambda it: torch.multinomial((it * w).softmax(dim=-1), 100, replacement=True)

acc = MulticlassAccuracy(numclasses=numclasses, average="micro") accperclass = MulticlassAccuracy(numclasses=numclasses, average=None) confmat = MulticlassConfusionMatrix(numclasses=numclasses)

plot single value

for i in range(5): accperclass.update(preds(i), target(i)) confmat.update(preds(i), target(i)) fig1, ax1 = accperclass.plot() fig2, ax2 = confmat.plot()

plot multiple values

values = [] for i in range(10): values.append(acc(preds(i), target(i))) fig3, ax3 = acc.plot(values) ```

For examples of plotting different metrics try running this example file.

Contribute!

The lightning + TorchMetrics team is hard at work adding even more metrics. But we're looking for incredible contributors like you to submit new metrics and improve existing ones!

Join our Discord to get help with becoming a contributor!

Community

For help or questions, join our huge community on Discord!

Citation

We’re excited to continue the strong legacy of open source software and have been inspired over the years by Caffe, Theano, Keras, PyTorch, torchbearer, ignite, sklearn and fast.ai.

If you want to cite this framework feel free to use GitHub's built-in citation option to generate a bibtex or APA-Style citation based on this file (but only if you loved it 😊).

License

Please observe the Apache 2.0 license that is listed in this repository. In addition, the Lightning framework is Patent Pending.