https://github.com/aida-ugent/abcfair

Science Score: 13.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
○
.zenodo.json file
○
DOI references
○
Academic publication links
○
Academic email domains
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (10.2%) to scientific vocabulary

Last synced: 10 months ago · JSON representation

Repository

Basic Info

Host: GitHub
Owner: aida-ugent
Language: Python
Default Branch: main
Size: 6.75 MB

Statistics

Stars: 3
Watchers: 3
Forks: 0
Open Issues: 0
Releases: 0

Created about 2 years ago · Last pushed about 2 years ago

Metadata Files

Readme

ABCFair: an Adaptable Benchmark approach for Comparing Fairness Methods

Code and results for the ABCFair benchmark. We evaluate on we evaluate 10 methods on 6 datasets (+ 1 from the unbiased labels in SchoolPerformance), 7 fairness notions, and 2 output formats, and 3 sensitive feature formats.

Results

Putting all results here would lead to an obscenely large README, so we provide two scripts to read out the benchmark results.

1: The Table Format

Results are presented in latex table code, with a row for each combination of sensitive feature format and maximal fairness violation values k.

The table is generated with the generate_table_results.py script. To see the command line options, run python generate_table_results.py --help. These include the dataset, the fairness notion with respect to which violation is measured, and the output format.

2: The Plot Format

Results are presented in an accuracy-fairness trade-off plot, for a range of fairness strengths. Each scatter point is the mean test performance and fairness violation, with a confidence ellipse (using the standard error) around it.

The plot is generated with the generate_tradeoff_results.py script. To see the command line options, run python generate_tradeoff_results.py --help. These include the dataset, the fairness notion with respect to which violation is measured, the output format, and the sensitive feature format.

Running the Pipeline

To generate new results, the pipeline can be run with main.py, which expects a config .yaml file as input. All logging is done using wandb (Weights and Biases), so you will need to login with your own account. Other required packages are found in requirements.txt.

Larger benchmark experiments were done using the config/sweep_config .yaml files, following standard wandb sweep practice.

Owner

Name: Ghent University Artificial Intelligence & Data Analytics Group
Login: aida-ugent
Kind: organization
Email: tijl.debie@ugent.be
Location: Ghent

Website: aida.ugent.be
Repositories: 36
Profile: https://github.com/aida-ugent

GitHub Events

Total

Watch event: 1

Last Year

Watch event: 1

Issues and Pull Requests

Last synced: about 1 year ago

All Time

Total issues: 0
Total pull requests: 0
Average time to close issues: N/A
Average time to close pull requests: N/A
Total issue authors: 0
Total pull request authors: 0
Average comments per issue: 0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

Past Year

Issues: 0
Pull requests: 0
Average time to close issues: N/A
Average time to close pull requests: N/A
Issue authors: 0
Pull request authors: 0
Average comments per issue: 0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Open Source Science