Data

Tools

Indexes

Applications

Experiments

Open Source Science

correctR

R package for computing corrected test statistics for comparing machine learning models on correlated samples

https://github.com/hendersontrent/correctr

Science Score: 49.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 1 DOI reference(s) in README
  • ✓
    Academic publication links
    Links to: pubmed.ncbi, ncbi.nlm.nih.gov, springer.com, zenodo.org
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (14.4%) to scientific vocabulary

Keywords

hypothesis-testing machine-learning statistics
Last synced: 6 months ago · JSON representation

Repository

R package for computing corrected test statistics for comparing machine learning models on correlated samples

Basic Info
Statistics
  • Stars: 14
  • Watchers: 1
  • Forks: 3
  • Open Issues: 1
  • Releases: 6
Topics
hypothesis-testing machine-learning statistics
Created about 3 years ago · Last pushed about 1 year ago
Metadata Files
Readme License

README.Rmd 

---
output: rmarkdown::github_document
---

# correctR 

[![CRAN version](https://www.r-pkg.org/badges/version/correctR)](https://www.r-pkg.org/pkg/correctR)
[![CRAN RStudio mirror downloads](https://cranlogs.r-pkg.org/badges/correctR)](https://www.r-pkg.org/pkg/correctR)
[![DOI](https://zenodo.org/badge/578642033.svg)](https://zenodo.org/badge/latestdoi/578642033)

Corrected test statistics for comparing machine learning models on correlated samples

```{r, include = FALSE}
knitr::opts_chunk$set(
  comment = NA, fig.width = 8, fig.height = 8, cache = FALSE)
```

## Installation

You can install the stable version of `correctR` from CRAN:

```{r eval = FALSE}
install.packages("correctR")
```

You can install the development version of `correctR` from GitHub:

```{r eval = FALSE}
devtools::install_github("hendersontrent/correctR")
```

## General purpose

Often in machine learning, we want to compare the performance of different models to determine if one statistically outperforms another. However, the methods used (e.g., data resampling, $k$-fold cross-validation) to obtain these performance metrics (e.g., classification accuracy) violate the assumptions of traditional statistical tests such as a $t$-test. The purpose of these methods is to either aid generalisability of findings (i.e., through quantification of error as they produce multiple values for each model instead of just one) or to optimise model hyperparameters. This makes them invaluable, but unusable with traditional tests, as [Dietterich (1998)](https://pubmed.ncbi.nlm.nih.gov/9744903/) found that the standard $t$-test underestimates the variance, therefore driving a high Type I error. `correctR` is a lightweight package that implements a small number of corrected test statistics for cases when samples are not independent (and therefore are correlated), such as in the case of resampling, $k$-fold cross-validation, and repeated $k$-fold cross-validation. These corrections were all originally proposed by [Nadeau and Bengio (2003)](https://link.springer.com/article/10.1023/A:1024068626366). Currently, only cases where two models are to be compared are supported.

Owner

  • Name: Trent Henderson
  • Login: hendersontrent
  • Kind: user
  • Location: Canberra, Australia
  • Company: Nous Group

Senior data scientist and statistics PhD student. Mostly coding in R, Julia, and Stan. Interested in genetic programming, time series, and data vis

GitHub Events

Total
  • Release event: 1
  • Watch event: 3
  • Push event: 8
  • Pull request event: 4
  • Fork event: 1
  • Create event: 1
Last Year
  • Release event: 1
  • Watch event: 3
  • Push event: 8
  • Pull request event: 4
  • Fork event: 1
  • Create event: 1

Committers

Last synced: 7 months ago

All Time
  • Total Commits: 41
  • Total Committers: 2
  • Avg Commits per committer: 20.5
  • Development Distribution Score (DDS): 0.073
Past Year
  • Commits: 3
  • Committers: 1
  • Avg Commits per committer: 3.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Trent t****1@o****m 38
Trent Henderson t****n@n****m 3
Committer Domains (Top 20 + Academic)
nousgroup.com: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 5
  • Total pull requests: 17
  • Average time to close issues: 4 months
  • Average time to close pull requests: about 2 hours
  • Total issue authors: 2
  • Total pull request authors: 1
  • Average comments per issue: 0.6
  • Average comments per pull request: 0.0
  • Merged pull requests: 17
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 4
  • Average time to close issues: N/A
  • Average time to close pull requests: 2 minutes
  • Issue authors: 0
  • Pull request authors: 1
  • Average comments per issue: 0
  • Average comments per pull request: 0.0
  • Merged pull requests: 4
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • hendersontrent (3)
  • fnmelo (1)
Pull Request Authors
  • hendersontrent (17)
Top Labels
Issue Labels
documentation (2) bug (1)
Pull Request Labels
documentation (6) enhancement (4) bug (2) CRAN (1)

Packages

  • Total packages: 1
  • Total downloads:
    • cran 324 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 0
  • Total versions: 4
  • Total maintainers: 1
cran.r-project.org: correctR

Corrected Test Statistics for Comparing Machine Learning Models on Correlated Samples

  • Versions: 4
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 324 Last month
Rankings
Forks count: 28.8%
Dependent packages count: 29.8%
Stargazers count: 31.7%
Dependent repos count: 35.5%
Average: 40.0%
Downloads: 74.3%
Maintainers (1)
then6675@uni.sydney.edu.au
Last synced: 6 months ago

Dependencies

.github/workflows/pkgdown.yaml actions
  • JamesIves/github-pages-deploy-action v4.4.1 composite
  • actions/checkout v3 composite
  • r-lib/actions/setup-pandoc v2 composite
  • r-lib/actions/setup-r v2 composite
  • r-lib/actions/setup-r-dependencies v2 composite
DESCRIPTION cran
  • R >= 3.5.0 depends
  • stats * imports
  • knitr * suggests
  • markdown * suggests
  • pkgdown * suggests
  • rmarkdown * suggests
  • testthat >= 3.0.0 suggests