Scripts for simulation study on obtaining average time to discovery

This repository contains the scripts for a simulation study performed on data from the systematic review by Brouwer et al. (2019).

The simulation study has the following characteristics:

• The number of runs is equal to the number of inclusions in the dataset;
• Every run starts with 1 prior inclusion and 10 prior exclusions;
• The prior inclusion is different for every run, e.g. all inclusions in the data are used as a prior inclusion once;
• The 10 prior exclusions are the same for every run, and they are randomly sampled from the dataset.

Installation

The scripts in this repository require Python 3.6+ and R. Install the dependencies with (in the command line):

```

install required software (such as ASReview)

pip install requirements.txt Rscript install.R ```

Data

The systematic review data used for this study can be found on the Open Science Framework. Download the file brouwer_deduplicated.csv and put it in the /data folder.

Execute simulation study

To reproduce the results presented in this study, run the following in your terminal:

```

run all simulations

sh run_all.sh ```

Extract results

Extract the results from the simulation output by executing the following in your CLI:

Rscript scripts/extract_simulation_results.R

The output (in the folder output/) consists of - All inclusions in the dataset, ordered by their time to discovery - the time_to_discovery.csv file. The column row corresponds to the position of the inclusion in the original dataset. Note that the position starts at 1; - A table with WSS and RRF values for every run - the wss_and_rrf_values.csv file; - A plot with recall curves for all runs of this simulation study - the recall.png file.

License

The content in this repository is published under the MIT license.

Funding

This project is funded by a grant from the Centre for Urban Mental Health, University of Amsterdam, The Netherlands

Contact

For any questions or remarks, please send an email to the ASReview-team or Marlies Brouwer.

References

Brouwer, M. E., Williams, A. D., Kennis, M., Fu, Z., Klein, N. S., Cuijpers, P., & Bockting, C. L. H. (2019). Psychological theories of depressive relapse and recurrence: A systematic review and meta-analysis of prospective studies. Clinical Psychology Review, 74, 101773. https://doi.org/10.1016/j.cpr.2019.101773