Data

Tools

Indexes

Applications

Experiments

Open Source Science

recsys-metrics

An efficient PyTorch implementation of the evaluation metrics in recommender systems.

https://github.com/zuoxingdong/recsys_metrics

Science Score: 44.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (10.0%) to scientific vocabulary
Last synced: 7 months ago · JSON representation ·

Repository

An efficient PyTorch implementation of the evaluation metrics in recommender systems.

Basic Info
  • Host: GitHub
  • Owner: zuoxingdong
  • Language: Python
  • Default Branch: master
  • Size: 580 KB
Statistics
  • Stars: 27
  • Watchers: 2
  • Forks: 1
  • Open Issues: 0
  • Releases: 3
Created over 4 years ago · Last pushed about 3 years ago
Metadata Files
Readme Citation

README.md

recsys_metrics

**An efficient PyTorch implementation of the evaluation metrics in recommender systems.** ______________________________________________________________________

Overview • Installation • How to use • Benchmark • Citation •

______________________________________________________________________

Overview

Why do we need recsys_metrics?

Highlights

  • Efficient (vectorized) implementations over mini-batches.
  • Standard RecSys metrics: precision, recall, map, mrr, hr, ndcg, $\alpha$-ndcg
  • Beyond-accuracy metrics: coverage, diversity, novelty, serendipity, etc.
  • All metrics support a top-k argument.

Installation

You can install recsys_metrics from PyPI:

bash pip install recsys_metrics

Or you can also install the latest version from source:

bash pip install git+https://github.com/zuoxingdong/recsys_metrics.git@master

Note that we support Python 3.7+ only.

How to use

Hit Rate

Let us take Hit Rate (HR) to illustrate how to use this library:

```python preds = torch.tensor([ [.5, .3, .1], [.3, .4, .5] ]) target = torch.tensor([ [0, 0, 1], [0, 1, 1] ]) hit_rate(preds, target, k=1, reduction='mean')

tensor(0.5000) ``` The first example in the batch does not have a hit (i.e. top-1 item is not a relevant item) and second example in the batch gets a hit (i.e. top-1 item is a relevant item). Thus, we have a hit-rate of 0.5.

The API of other metrics are of the same format.

Serendipity

Let us calculate serendipity:

python preds = torch.tensor([ [.5, .3, .1], [.3, .4, .5] ]) indexes = torch.tensor([ [1, 2, 3], [4, 5, 6] ]) target = torch.tensor([ [0, 1, 1], [0, 1, 1] ])

Suppose we are given the following tensors for the category of items and historical interactions for each example in the mini-batch:

python item_category = torch.tensor([ [2, 1, 8], [3, 9, 4] ]) hist_seq_category = torch.tensor([ [5, 0, 1, 2, 3, 0, 4], [1, 3, 6, 0, 2, 3, 5] ])

Then, we could use a helper function provided in this library to calculate a tensor of unexpectedness (or surprise) with a window size of the most recent interactions:

python unexpectedness = category_unexpectedness( item_category, hist_seq_category=hist_seq_category[:, None, :].tile(1, 3, 1), window_size=3 )

Finally, let us calculate the serendipity:

```python serendipity(preds, target, unexpectedness, k=3, reduction='mean')

tensor(0.6667) ```

Report functions

This library also offers report functions to conveniently calculate standard IR metrics or beyond-accuracy metrics in an one-liner style.

```python rankreport( preds=preds, target=target, k=None, toitem=True, name_abbreviation=True, rounding=4, )

{'prec': 0.5, 'rec': 1.0, 'map': 0.6667, 'mrr': 0.6667, 'hr': 1.0, 'ndcg': 0.75} ```

```python beyondaccuracyreport( preds=preds, indexes=indexes, k=None, toitem=True, nameabbreviation=True, target=target, unexpectedness=unexpectedness.long(), rounding=4, )

{'catcov': 1.0, 'distcov': 1.0, 'mil': 1.0, 'serend': 0.6667} ```

Benchmark

| Metrics | Single Example | Mini-Batch | | :---: | :---: | :---: | | Precision | | | | Recall | | | | MAP | | | | MRR | | | | HR | | | | NDCG | | |

Citation

This work is inspired by Torchmetrics from PyTorchLightning Team.

Please use this bibtex if you want to cite this repository in your publications:

@misc{recsys_metrics,
      author = {Zuo, Xingdong},
      title = {recsys_metrics: An efficient PyTorch implementation of the evaluation metrics in recommender systems.},
      year = {2021},
      publisher = {GitHub},
      journal = {GitHub repository},
      howpublished = {\url{https://github.com/zuoxingdong/recsys_metrics}},
    }

Owner

  • Name: Xingdong Zuo
  • Login: zuoxingdong
  • Kind: user
  • Location: Seongnam, South Korea
  • Company: NAVER Corp

AI in well-being is my dream. Neural networks need to understand the world causally.

Citation (CITATION.cff) 

cff-version: 1.2.0
message: "If you use this software, please cite it using the following metadata"
title: recsys_metrics
authors:
- family-names: "Zuo"
  given-names: "Xingdong"
version: 0.0.2
url: "https://github.com/zuoxingdong/recsys_metrics"

GitHub Events

Total
  • Watch event: 3
Last Year
  • Watch event: 3

Committers

Last synced: about 3 years ago

All Time
  • Total Commits: 8
  • Total Committers: 1
  • Avg Commits per committer: 8.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Xingdong Zuo z****g@h****m 8

Issues and Pull Requests

Last synced: 8 months ago

All Time
  • Total issues: 0
  • Total pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Total issue authors: 0
  • Total pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 267 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 1
  • Total versions: 4
  • Total maintainers: 1
pypi.org: recsys-metrics

An efficient PyTorch implementation of the evaluation metrics in recommender systems.

  • Versions: 4
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 267 Last month
Rankings
Dependent packages count: 10.1%
Stargazers count: 14.2%
Average: 18.7%
Dependent repos count: 21.6%
Forks count: 22.6%
Downloads: 25.2%
Maintainers (1)
zuoxingdong
Last synced: 8 months ago

Dependencies

setup.py pypi
  • numpy >=1.17.2
  • torch >=1.3.1