trust-crisis-in-simulation-based-inference
Code related to the manuscript "Averting A Crisis In Simulation-Based Inference"
https://github.com/montefiore-institute/trust-crisis-in-simulation-based-inference
Science Score: 54.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (10.9%) to scientific vocabulary
Repository
Code related to the manuscript "Averting A Crisis In Simulation-Based Inference"
Basic Info
Statistics
- Stars: 5
- Watchers: 4
- Forks: 1
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
Abstract
We present extensive empirical evidence showing that current Bayesian simulation-based inference algorithms are inadequate for the falsificationist methodology of scientific inquiry. Our results collected through massive experimental computations show that all benchmarked algorithms -- (S)NPE, (S)NRE, SNL and variants of ABC -- may produce overconfident posterior approximations, which makes them demonstrably unreliable and dangerous if one's scientific goal is to constrain parameters of interest. We believe that failing to address this issue will lead to a well-founded trust crisis in simulation-based inference. For this reason, we argue that research efforts should now focus on theoretical and methodological developments of conservative approximate inference algorithms and present research directions towards this objective. In this regard, we show empirical evidence that ensembles are consistently more reliable.
A PDF render of the manuscript is available in this repo or on ArXiV.
Using the code
Recommended. This installs a Python 3 environment by default.
console
you@computer:~ $ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
you@computer:~ $ sh Miniconda3-latest-Linux-x86_64.sh
Next, install the necessary dependencies.
console
you@computer:~ conda env create -f environment.yml
you@computer:~ conda activate crisissbi
After the environment has been activated, there are 2 ways to execute the pipelines depending on your setup.
The first only requires your laptop. In that regard simply execute a pipeline as follows:
console
you@computer:~ cd workflows/auc_demonstration
you@computer:~ python pipeline.py
The other approach is on a Slurm enabled HPC cluster. To exploit the parallelism, execute the script as
console
you@computer:~ cd workflows/auc_demonstration
you@computer:~ python pipeline.py --slurm
The jobs will be automatically submitted to the default Slurm queue.
Citation
See CITATION.cff
License
Described the LICENSE file.
Owner
- Name: Montefiore Institute
- Login: montefiore-institute
- Kind: organization
- Location: Liège, Belgium
- Website: https://montefiore.uliege.be
- Repositories: 1
- Profile: https://github.com/montefiore-institute
Research at the Montefiore Institute of the University of Liège.
Citation (CITATION.cff)
@ARTICLE{2021arXiv211006581H,
author = {{Hermans}, Joeri and {Delaunoy}, Arnaud and {Rozet}, Fran{\c{c}}ois and {Wehenkel}, Antoine and {Louppe}, Gilles},
title = "{Averting A Crisis In Simulation-Based Inference}",
journal = {arXiv e-prints},
keywords = {Statistics - Machine Learning, Computer Science - Machine Learning},
year = 2021,
month = oct,
eid = {arXiv:2110.06581},
pages = {arXiv:2110.06581},
archivePrefix = {arXiv},
eprint = {2110.06581},
primaryClass = {stat.ML},
adsurl = {https://ui.adsabs.harvard.edu/abs/2021arXiv211006581H},
adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}
GitHub Events
Total
Last Year
Dependencies
- cloudpickle *
- palettable *
- papermill *
- sbi *
- sklearn *