Computational Reproducibility Assessments: Summary

Summary of the eight computational reproducibility assessments conducted as part of STARS Work Package 1.

Table of contents

• 👋 About the repository
• 📍 Locating tables and figures from the article
• 📖 View book locally
• 📝 Citation
• 💰 Funding

👋 About the repository

In work package 1, we assessed the computational reproducibility of eight discrete-event simulation papers with models in Python and R. The reproductions and findings are summarised at: https://pythonhealthdatascience.github.io/stars_wp1_summary/.

Relevant GitHub repositories:

| Repository | Description | | --- | --- | | stars-reproduction-protocol | Latex files for reproduction protocol | | stars-reproduce-allen-2020 |Test run of reproducibility protocol on Allen et al. 2020 | | stars-reproduction-template | Template for assessment of computational reproducibility | | stars-reproduce-shoaib-2022 | Reproduction study 1: Shoaib and Ramamohan 2022 (Python) | | stars-reproduce-huang-2019 | Reproduction study 2: Huang et al. 2019 (R) | | stars-reproduce-lim-2020 | Reproduction study 3: Lim et al. 2020 (Python) | | stars-reproduce-kim-2021 | Reproduction study 4: Kim et al. 2021 (R) | | stars-reproduce-anagnostou-2022 | Reproduction study 5: Anagnostou et al. 2022 (Python) | | stars-reproduce-johnson-2021 | Reproduction study 6: Johnson et al. 2021 (R) | | stars-reproduce-hernandez-2015 | Reproduction study 7: Hernandez et al. 2015 (Python + R) | | stars-reproduce-wood-2021 | Reproduction study 8: Wood et al. 2021 (R) |

Process followed for each study:

📍 Locating tables and figures from the article

| Figure/Table | Method | Location | | - | - | - | | Figure 1. Five standards that scientific code should strive to achieve, and the benefits of doing so | Inkscape | images/5rs.svg | | Figure 2. Time to complete items in scope for each reproduction, inspired by figure in Krafczyk et al. 2021 | Matplotlib | Created within pages/reproduction.qmd, saved as images/article_times.png | | Figure 3. Recommendations to support reproduction, with stars to highlight five recommendations considered to have the greatest potential impact. | Inkscape | images/reproduction_wheel.svg| | Figure 4. Recommendations to support troubleshooting and reuse | Inkscape | images/troubleshooting_wheel.svg | | Figure 5. Of the eight healthcare DES studies evaluated, proportion that met each recommendation in the current STARS framework. | Plotly express | Created within pages/repo_evaluation.qmd, saved as images/stars_criteria.png | | Figure 6. Of the eight healthcare DES studies evaluated, proportion that met each item in the current STRESS-DES criteria. | Plotly express | Created within pages/paper_evaluation.qmd, saved as stress_criteria.png | | Figure 7. Of the eight healthcare DES studies evaluated, proportion that met each criteria in the general reporting checklist for DES | Plotly express | Created within pages/paper_evaluation.qmd, saved as ispor_criteria.png | | Table 2. Evaluation of repositories against ACM badge criteria. | - | Created within pages/repo_evaluation.qmd, saved as data/badges_table.csv (and Table 2 is an extract from that table) | | Table 3. Proportion of applicable criteria that were fully met, from evaluation of repository or article, alongside the proportion of items reproduced from each study. | - | Combination of two tables: (1) data/applicable_stars.csv created within pages/repo_evaluation.qmd, and (2) data/applicable_report.csv created within pages/paper_evaluation.qmd | | Table D1. Evaluation of studies against badge criteria - grouped into three themes, as defined by NISO. | - | Created within pages/repo_evaluation.qmd, saved as data/badges_table.csv |

The remaining tables were created directly in the Latex article, rather than in this repository, as they are not describing results from reproduction:

• Table 1. Description of the included studies.
• Table 4. Simple checklists to assist reviewers in assessing the openness, longevity, and reproducibility of DES models during peer review.
• Table B1. Links for reproduction and evaluation.
• Table B2. Links to original study repositories.

📖 View book locally

The website is a quarto book hosted with GitHub pages. If you want to view the book locally on your own machine you will need to:

1. Clone GitHub repository

git clone https://github.com/pythonhealthdatascience/stars_wp1_summary.git

1. Create the virtual environment

virtualenv stars_wp1_summary source stars_wp1_summary/bin/activate pip install -r requirements.txt

1. Create the book

quarto render

1. Open the book in your browser (open the _book/index.html file).

📝 Citation

This repository has been archived on Zenodo and can be cited as:

Heather, A., Monks, T., & Harper, A. (2025). Computational Reproducibility Assessments: Summary. Zenodo. https://doi.org/10.5281/zenodo.14267268.

If you wish to cite this repository on GitHub, please refer to the citation file CITATION.cff, and the auto-generated alternatives citation_apalike.apa and citation_bibtex.bib. Authors:

| Member | ORCID | GitHub | | --- | --- | --- | | Amy Heather | | https://github.com/amyheather | | Thomas Monks | | https://github.com/TomMonks | | Alison Harper | | https://github.com/AliHarp |

💰 Funding

This project is supported by the Medical Research Council [grant number MR/Z503915/1].