Efficient inference of rankings from multi-body comparisons

This project provides source code for the Efficient inference of rankings from multi-body comparisons. The repository also contains the original scientific analyses developed by the Authors (see below) for the paper

• Yeung et al. 2025. Efficient inference of rankings from multi-body comparisons.

If you use this codebase, please cite our work according to the CITATION.cff.

Contents

• Efficient inference of rankings from multi-body comparisons
• Contents
• Getting Started
  • Prerequisites
  • Installing
  • Quick Start
• Usage
  • Reproducing experiments
  • Package Structure
• Documentation
• Tests
• Other Information
  • Built With
  • Contributing
  • Versioning
  • Authors
  • License
  • Acknowledgments

Getting Started

The code base for this project is written in Python with package management handled with Conda.

These instructions will give you a copy of the project up and running on your local machine for development, testing, and analysis purposes.

Prerequisites

A compatible Python install is needed to begin - the package management is handled by Conda as described below. - Python [3.10+] - GNU Make [4.2+]

A complete list of utilized packages is available in the requirements.txt file. There is, however, a package dependency hierarchy where some packages in the requirements.txt are not strictly necessary for the utilization of package infrastructure. The core requirements are listed as dependencies in the build instructions. Further instructions for creating a controlled environment from this manifest is available below, in the Installing section.

Installing

To (locally) reproduce this project, do the following:

1. Download this code base. Notice that raw data are typically not included in the git-history and may need to be downloaded independently - see Reproducing Experiments for more information.
2. (Optional) Open a terminal with Python installed and create a new virtual environment: bash python -m venv .venv source .venv/bin/activate
3. Install the package bash pip install .

This will install all necessary packages for you to be able to run the scripts and everything should work out of the box.

Quick Start

This guide provides simple instructions for running the simulation on a chosen dataset.

1. Compile source code

The core of the ranking calculations are written in an efficient C implementation. These must be compiled before the python scripts running the experiments will work. You can use the provided makefiles in the C_Prog/ subdirectories to compile this for UNIX-based machines out-of-the-box; Windows machines will need to edit some of the compiler flags within the makefiles. The compilation can be accomplished by running the following from the root directory

bash cd C_Prog/Readfile make cd ../Convergence_Readfile make cd ../..

1. Locate the Dataset

Find the ID of the dataset you want to use by checking the datasets/dataset.info file. Each dataset is assigned a unique ID that must be formatted as a 5-digit number, with leading zeroes if necessary (e.g., 00001). Ensure that the selected dataset has a file for both its edges and nodes within dataset/Real_Data/

1. Run the Model

if the dataset has the true scores set issynthetic = 1 otherwise issynthetic = 0

Run the model on the selected dataset using the following command:

bash python3 src/test.py --dataset_number=00001 --is_synthetic=0

Usage

Reproducing experiments

Synthetic

to generate synthetic results(note that this will create a large amount of files)

bash python3 datasets/utils/gen_synthetic_data

Accuracy

To run the experiments on the accuracy of all four models on the synthetic data bash python3 exp/ex01/ex01

This will download each result into the folder exp/ex01/data to preprocess and visualize these result run all cells within notebook/ex01syntheticaccuracy

Convergence

To run the experiments on the convergence of our model and zermellos bash python3 exp/ex02/ex02 This will result in a table being saved into the folder exp/ex02/results to visualize these results run all cells within the file notebook/ex02syntheticconvergence

Real Results

To run all datasets included in the paper

Accuracy

To run the experiments on the accuracy of all four models on the synthetic data bash python3 exp/ex03/ex03

This will download each result into the folder exp/ex03/data to preprocess and visualize these result run all cells within notebook/ex01realaccuracy

Convergence

To run the experiments on the convergence of our model and zermellos bash python3 exp/ex04/ex04 This will result in a table being saved into the folder exp/ex04/results to visualize these results run all cells within the file notebook/ex04realconvergence

Package Structure

text ├── C_Prog * Efficient C implementation │   ├── Convergence_Readfile * Measure convergence results │   │   ├── bt_functions.c │   │   ├── bt_functions.h │   │   ├── bt_model_data.c │   │   ├── bt_model_data.out │   │   ├── makefile │   │   ├── mt19937-64.c │   │   ├── mt64.h │   │   ├── my_sort.c │   │   └── my_sort.h │   └── Readfile * Measure accuracy │   ├── bt_functions.c │   ├── bt_functions.h │   ├── bt_model_data.c │   ├── bt_model_data.out │   ├── makefile │   ├── mt19937-64.c │   ├── mt64.h │   ├── my_sort.c │   └── my_sort.h ├── LICENSE ├── README.md ├── datasets │   ├── Real_Data * Edges and Nodes of datasets used in paper │   │   ├── 00001_edges.txt │   │   ├── 00001_nodes.txt │   │   ├── 00002_edges.txt │   │   ├── 00002_nodes.txt │   │   ├── 00003_edges.txt │   │   ├── 00003_nodes.txt │   │   ├── 00004_edges.txt │   │   ├── 00004_nodes.txt │   │   ├── 00005_edges.txt │   │   ├── 00005_nodes.txt │   │   ├── 00006_edges.txt │   │   ├── 00006_nodes.txt │   │   ├── 00007_edges.txt │   │   ├── 00007_nodes.txt │   │   ├── 00008_edges.txt │   │   ├── 00008_nodes.txt │   │   ├── 00009_edges.txt │   │   └── 00009_nodes.txt │   ├── dataset_info.csv * information on edge size, number of players, number of games, and mappings of dataset names and ids │   └── utils * preprocessing │   ├── convert_raw_files.py │   ├── dataset_info.py │   ├── extract_ordered_games.py │   ├── gen_synthetic_data.py │   └── rename_datasets.py ├── doc │   ├── experiment_descriptions.txt │   └── sketch_experiment.txt ├── exp │   ├── ex01 │   │   ├── ex01.py │   │   └── results │   │   ├── leadership_log_likelihood_summary.csv │   │   ├── log_likelihood_summary.csv │   │   ├── rho_summary.csv │   │   └── tau_summary.csv │   ├── ex02 │   │   ├── ex02.py │   │   └── results │   │   └── Convergence_Table.csv │   ├── ex03 │   │   ├── ex03.py │   │   └── results │   │   ├── leadership_log_likelihood_summary.csv │   │   └── log_likelihood_summary.csv │   └── ex04 │   ├── ex04.py │   └── results │   └── Convergence_Table.csv ├── notebook │   ├── comparison_models.ipynb │   ├── convergence_behavior.ipynb │   ├── ex01_synthetic_accuracy.ipynb │   ├── ex02_synthetic_convergence.ipynb │   ├── ex03_real_accuracy.ipynb │   ├── ex04_real_convergence.ipynb │   ├── figure_settings │   │   ├── __init__.py │   │   ├── ieee.mplstyle │   │   ├── science.mplstyle │   │   └── settings.py │   └── training_size.ipynb ├── requirements.txt ├── src │   ├── __init__.py │   ├── archive │   │   ├── weighted_bt.py │   │   └── weighted_graph_helpers.py │   ├── models * All models including comparisons to Zermello and other graph ranking algorithms │   │   ├── BradleyTerry.py * Python representation of our model │   │   ├── SpringRank.py │   │   ├── __init__.py │   │   ├── page_rank.py │   │   ├── point_wise.py │   │   └── zermello.py │   |── utils │   | ├── __init__.py │   | ├── c_operation_helpers.py * run c code │   | ├── convergence_test_helpers.py │   | ├── file_handlers.py │   | ├── graph_tools.py * building hypergraphs │   | ├── metrics.py │   | └── operation_helpers.py * run python implmentations | |_ test.py * Example Run | | └── tst ├── test_graph_tools.py ├── test_metrics.py ├── test_models.py ├── test_operation_helpers.py └── test_synthetic.py

Documentation

This repository does not maintain extensive independent documentation for its source code. We do, however, include documentation and notes on scientific experiments we've conducted throughout the project. If you are interested in seeing these notes, please email Filippo Radicchi with your inquiry.

Tests

All unit tests are written with pytest.

Tests can be run directly with the commands: bash pip install pytest pytest tst/

Other Information

Built With

• ChooseALicense - Used to choose the license

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Versioning

We use Semantic Versioning for versioning. For the versions available, see the tags on this repository.

Authors

All correspondence shoulld be directed to Filippo Radicchi.

• Jack Yeung
• Daniel Kaiser
• Filippo Radicchi

License

This project is licensed under the MIT License Creative Commons License - see the LICENSE file for details.

Acknowledgments

• Billie Thompson - Provided README and CONTRIBUTING template - PurpleBooth
• George Datseris - Published workshop on scientific code; inspired organization for reproducibility - GoodScientificCodeWorkshop