CalibrationErrors
Estimation of calibration errors.
Science Score: 54.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
✓Academic publication links
Links to: zenodo.org -
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (12.2%) to scientific vocabulary
Keywords
Keywords from Contributors
Repository
Estimation of calibration errors.
Basic Info
- Host: GitHub
- Owner: devmotion
- License: mit
- Language: Julia
- Default Branch: main
- Homepage: https://devmotion.github.io/CalibrationErrors.jl/dev
- Size: 11.2 MB
Statistics
- Stars: 17
- Watchers: 3
- Forks: 3
- Open Issues: 2
- Releases: 31
Topics
Metadata Files
README.md
CalibrationErrors.jl
Estimation of calibration errors.
There are also Python and R interfaces for this package
Overview
This package implements different estimators of the expected calibration error (ECE), the squared kernel calibration error (SKCE), and the unnormalized calibration mean embedding (UCME) in the Julia language.
This package supports calibration error estimation of classification models that output vectors of class probabilities. In addition, SKCE and UCME can be estimated for more general probabilistic predictive models that output probability distributions defined in Distributions.jl such as normal and Laplace distributions.
Example
Calibration errors can be estimated from a data set of predicted probability distributions
and a set of corresponding observed targets by executing
julia
estimator(predictions, targets)
The sets of predictions and targets have to be provided as vectors.
This package implements the estimator ECE of the ECE, the estimator SKCE for the SKCE
(unbiased and biased variants with different sample complexity), and UCME for the UCME.
Related packages
CalibrationTests.jl implements statistical hypothesis tests of calibration.
pycalibration is a Python interface for CalibrationErrors.jl and CalibrationTests.jl.
rcalibration is an R interface for CalibrationErrors.jl and CalibrationTests.jl.
Talk at JuliaCon 2021
The slides of the talk are available as Pluto notebook.
Citing
If you use CalibrationErrors.jl as part of your research, teaching, or other activities, please consider citing the following publications:
Widmann, D., Lindsten, F., & Zachariah, D. (2019). Calibration tests in multi-class classification: A unifying framework. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019) (pp. 12257–12267).
Widmann, D., Lindsten, F., & Zachariah, D. (2021). Calibration tests beyond classification. International Conference on Learning Representations (ICLR 2021).
Acknowledgements
This work was financially supported by the Swedish Research Council via the projects Learning of Large-Scale Probabilistic Dynamical Models (contract number: 2016-04278), Counterfactual Prediction Methods for Heterogeneous Populations (contract number: 2018-05040), and Handling Uncertainty in Machine Learning Systems (contract number: 2020-04122), by the Swedish Foundation for Strategic Research via the project Probabilistic Modeling and Inference for Machine Learning (contract number: ICA16-0015), by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, and by ELLIIT.
Owner
- Name: David Widmann
- Login: devmotion
- Kind: user
- Location: Uppsala, Sweden
- Company: Uppsala University
- Website: widmann.dev
- Repositories: 16
- Profile: https://github.com/devmotion
Citation (CITATION.bib)
@incollection{CalibrationAnalysis.jl-2019,
title = {Calibration tests in multi-class classification: A unifying framework},
author = {Widmann, David and Lindsten, Fredrik and Zachariah, Dave},
booktitle = {Advances in Neural Information Processing Systems 32},
pages = {12236--12246},
year = {2019},
url = {http://papers.nips.cc/paper/9392-calibration-tests-in-multi-class-classification-a-unifying-framework}
}
@inproceedings{CalibrationAnalysis.jl-2021,
title={Calibration tests beyond classification},
author={Widmann, David and Lindsten, Fredrik and Zachariah, Dave},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=-bxf89v3Nx}
}
GitHub Events
Total
- Watch event: 2
- Fork event: 1
Last Year
- Watch event: 2
- Fork event: 1
Committers
Last synced: over 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| David Widmann | d****t@d****e | 78 |
| David Widmann | d****n | 61 |
| David Widmann | d****n@i****e | 26 |
| github-actions[bot] | 4****] | 14 |
| github-actions[bot] | c****y@j****g | 4 |
| Pietro Monticone | 3****e | 2 |
| dependabot[bot] | 4****] | 2 |
| Julia TagBot | 5****t | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 2
- Total pull requests: 107
- Average time to close issues: 2 days
- Average time to close pull requests: 12 days
- Total issue authors: 2
- Total pull request authors: 5
- Average comments per issue: 10.5
- Average comments per pull request: 1.71
- Merged pull requests: 20
- Bot issues: 0
- Bot pull requests: 53
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- DwaraknathT (1)
- JuliaTagBot (1)
Pull Request Authors
- github-actions[bot] (53)
- devmotion (52)
- dependabot[bot] (7)
- pitmonticone (2)
- dependabot-preview[bot] (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- julia 6 total
- Total dependent packages: 3
- Total dependent repositories: 0
- Total versions: 32
juliahub.com: CalibrationErrors
Estimation of calibration errors.
- Homepage: https://devmotion.github.io/CalibrationErrors.jl/dev
- Documentation: https://docs.juliahub.com/General/CalibrationErrors/stable/
- License: MIT
-
Latest release: 0.6.4
published over 2 years ago
Rankings
Dependencies
- actions/checkout v4 composite
- codecov/codecov-action v3 composite
- coverallsapp/github-action v2 composite
- julia-actions/cache v1 composite
- julia-actions/julia-buildpkg v1 composite
- julia-actions/julia-processcoverage v1 composite
- julia-actions/julia-runtest v1 composite
- julia-actions/setup-julia v1 composite
- actions/checkout v4 composite
- actions/checkout v4 composite
- actions/checkout v4 composite
- julia-actions/cache v1 composite
- julia-actions/julia-buildpkg v1 composite
- julia-actions/setup-julia v1 composite
- actions/checkout v4 composite
- julia-actions/setup-julia latest composite
- reviewdog/action-suggester v1 composite
- JuliaRegistries/TagBot v1 composite