evalica

Evalica, your favourite evaluation toolkit

Science Score: 67.0%

This score indicates how likely this project is to be science-related based on various indicators:

✓
CITATION.cff file
Found CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
✓
DOI references
Found 6 DOI reference(s) in README
✓
Academic publication links
Links to: arxiv.org
○
Committers with academic emails
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (11.6%) to scientific vocabulary

Keywords

arena bradley-terry elo evalica evals evaluation hacktoberfest leaderboard library llm pagerank pairwise-comparison pyo3 python ranking rating rust serbia statistics winrate

Keywords from Contributors

aggregations annotation crowd crowdsourcing data-mining labeling quality-control toloka truth-inference transformer

Last synced: 6 months ago · JSON representation ·

Repository

Evalica, your favourite evaluation toolkit

Basic Info

Host: GitHub
Owner: dustalov
License: apache-2.0
Language: Python
Default Branch: master
Homepage: https://dustalov.github.io/evalica/
Size: 640 KB

Statistics

Stars: 57
Watchers: 2
Forks: 6
Open Issues: 2
Releases: 16

Topics

arena bradley-terry elo evalica evals evaluation hacktoberfest leaderboard library llm pagerank pairwise-comparison pyo3 python ranking rating rust serbia statistics winrate

Created over 1 year ago · Last pushed 6 months ago

Metadata Files

Readme Funding License Citation

Evalica, your favourite evaluation toolkit

Evalica ɛˈʋalit͡sa is a Python library that transforms pairwise comparisons into ranked lists of items. It offers convenient high-performant Rust implementations of the corresponding methods via PyO3, and additionally provides naïve Python code for most of them. Evalica is fully compatible with NumPy arrays and pandas data frames.

The logo was created using Recraft.

Installation

pip: pip install evalica
Anaconda: conda install conda-forge::evalica

Usage

Imagine that we would like to rank the different meals and have the following dataset of three comparisons produced by food experts.

Given this hypothetical example, Evalica takes these three columns and computes the outcome of the given pairwise comparison according to the chosen model. Note that the first argument is the column Item X, the second argument is the column Item Y, and the third argument corresponds to the column Winner.

```pycon

from evalica import elo, Winner result = elo( ... ['pizza', 'burger', 'pizza'], ... ['burger', 'sushi', 'sushi'], ... [Winner.X, Winner.Y, Winner.Draw], ... ) result.scores pizza 1014.972058 burger 970.647200 sushi 1014.380742 Name: elo, dtype: float64 ```

As a result, we obtain Elo scores of our items. In this example, pizza was the most favoured item, sushi was the runner-up, and burger was the least preferred item.

| Item| Score | |---|---:| | pizza | 1014.97 | | burger | 970.65 | | sushi | 1014.38 |

Command-Line Interface

Evalica also provides a simple command-line interface, allowing the use of these methods in shell scripts and for prototyping.

console $ evalica -i food.csv bradley-terry item,score,rank Tacos,2.509025136024378,1 Sushi,1.1011561298265815,2 Burger,0.8549063627182466,3 Pasta,0.7403814336665869,4 Pizza,0.5718366915548537,5

Refer to the food.csv file as an input example.

Web Application

Evalica has a built-in Gradio application that can be launched as python3 -m evalica.gradio. Please ensure that the library was installed as pip install evalica[gradio].

Implemented Methods

| Method | In Python | In Rust | |---|:---:|:---:| | Counting | ✅ | ✅ | | Average Win Rate | ✅ | ✅ | | Bradley–Terry | ✅ | ✅ | | Elo | ✅ | ✅ | | Eigenvalue | ✅ | ✅ | | PageRank | ✅ | ✅ | | Newman | ✅ | ✅ |

Contributing

Evalica is a mixed Rust/Python project that uses PyO3, so it requires setting up the Maturin build system.

To set up the environment, we recommend using the uv package manager, as demonstrated in our test suite:

console $ uv venv $ uv pip install maturin $ maturin develop --uv --extras dev,docs,gradio

In case uv is not available, you can use the following workaround:

console $ python3 -m venv venv $ source venv/bin/activate $ pip install maturin $ maturin develop --extras dev,docs,gradio

We welcome pull requests on GitHub: https://github.com/dustalov/evalica. To contribute, fork the repository, create a separate branch for your changes, and submit a pull request.

Citation

Ustalov, D. Reliable, Reproducible, and Really Fast Leaderboards with Evalica. 2025. Proceedings of the 31st International Conference on Computational Linguistics: System Demonstrations. 46–53. arXiv: 2412.11314 [cs.CL].

bibtex @inproceedings{Ustalov:25, author = {Ustalov, Dmitry}, title = {{Reliable, Reproducible, and Really Fast Leaderboards with Evalica}}, year = {2025}, booktitle = {Proceedings of the 31st International Conference on Computational Linguistics: System Demonstrations}, pages = {46--53}, address = {Abu Dhabi, UAE}, publisher = {Association for Computational Linguistics}, eprint = {2412.11314}, eprinttype = {arxiv}, eprintclass = {cs.CL}, url = {https://aclanthology.org/2025.coling-demos.6}, language = {english}, }

The code for replicating the experiments is available in the coling2025 directory.

Copyright

Owner

Name: Dmitry Ustalov
Login: dustalov
Kind: user
Location: Belgrade, Serbia
Company: @JetBrains

Website: https://ustalov.com/
Repositories: 14
Profile: https://github.com/dustalov

Citation (CITATION.cff)

cff-version: 1.2.0
title: Evalica
message: If you use this software, please cite the article from preferred-citation.
type: software
authors:
- family-names: Ustalov
  given-names: Dmitry
  orcid: "https://orcid.org/0000-0002-9979-2188"
license: Apache-2.0
identifiers:
  - type: url
    value: "https://aclanthology.org/2025.coling-demos.6"
  - type: url
    value: "https://github.com/dustalov/evalica"
  - type: other
    value: "arXiv:2412.11314"
repository-artifact: "https://pypi.org/project/evalica/"
repository-code: "https://github.com/dustalov/evalica"
preferred-citation:
  type: conference-paper
  authors:
  - family-names: Ustalov
    given-names: Dmitry
    orcid: "https://orcid.org/0000-0002-9979-2188"
  title: "Reliable, Reproducible, and Really Fast Leaderboards with Evalica"
  year: 2025
  collection-title: "Proceedings of the 31st International Conference on Computational Linguistics: System Demonstrations"
  conference:
    name: "31st International Conference on Computational Linguistics"
    date-start: 2025-01-19
    date-end: 2025-01-24
  url: "https://aclanthology.org/2025.coling-demos.6"
date-released: 2024-07-09

GitHub Events

Total

Create event: 32
Release event: 2
Issues event: 8
Watch event: 38
Delete event: 31
Issue comment event: 58
Push event: 60
Pull request review comment event: 2
Pull request review event: 3
Pull request event: 69
Fork event: 3

Last Year

Create event: 32
Release event: 2
Issues event: 8
Watch event: 38
Delete event: 31
Issue comment event: 58
Push event: 60
Pull request review comment event: 2
Pull request review event: 3
Pull request event: 69
Fork event: 3

Committers

Last synced: 9 months ago

All Time

Total Commits: 356
Total Committers: 3
Avg Commits per committer: 118.667
Development Distribution Score (DDS): 0.053

Past Year

Commits: 356
Committers: 3
Avg Commits per committer: 118.667
Development Distribution Score (DDS): 0.053

Top Committers

Name	Email	Commits
Dmitry Ustalov	d**v@g**m	337
dependabot[bot]	4****]	18
Mathew Shen	d**r@g**m	1

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 7
Total pull requests: 76
Average time to close issues: 4 days
Average time to close pull requests: 8 days
Total issue authors: 4
Total pull request authors: 3
Average comments per issue: 3.86
Average comments per pull request: 0.61
Merged pull requests: 36
Bot issues: 0
Bot pull requests: 70

Past Year

Issues: 6
Pull requests: 69
Average time to close issues: 5 days
Average time to close pull requests: 7 days
Issue authors: 4
Pull request authors: 3
Average comments per issue: 4.0
Average comments per pull request: 0.59
Merged pull requests: 33
Bot issues: 0
Bot pull requests: 63

View more stats

Top Authors

Issue Authors

shenxiangzhuang (3)
dustalov (2)
cthorrez (1)
zhimin-z (1)

Pull Request Authors

dependabot[bot] (70)
shenxiangzhuang (4)
zhimin-z (2)

Top Labels

Issue Labels

bug (2) question (2) wontfix (1) enhancement (1) python (1)

Pull Request Labels

dependencies (70) github_actions (42) rust (15) python (13)

Packages

Total packages: 1
Total downloads:
- pypi 5,923 last-month

Total dependent packages: 0
Total dependent repositories: 0
Total versions: 16
Total maintainers: 1

pypi.org: evalica

Evalica, your favourite evaluation toolkit.

Homepage: https://github.com/dustalov/evalica
Documentation: https://evalica.readthedocs.io/
License: Apache-2.0
Latest release: 0.3.2
published about 1 year ago

Versions: 16
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 5,923 Last month

Rankings

Dependent packages count: 10.7%

Average: 35.4%

Dependent repos count: 60.0%

Maintainers (1)

dustalov

Last synced: 6 months ago

evalica

Science Score: 67.0%

Keywords

Keywords from Contributors

Repository

Basic Info

Statistics

Topics

Metadata Files

README.md

Evalica, your favourite evaluation toolkit

Installation

Usage

Command-Line Interface

Web Application

Implemented Methods

Contributing

Citation

Copyright

Owner

Citation (CITATION.cff)

GitHub Events

Total

Last Year

Committers

All Time

Past Year

Top Committers

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

pypi.org: evalica

Rankings

Maintainers (1)