simChef

Overview

The goal of simChef is to help you quickly cook up a fully-realized, high-quality, reproducible, and transparently-documented simulation study in a flexible, efficient, and low-code manner. simChef removes many of the administrative burdens of simulation design through:

1. An intuitive tidy grammar of data science simulations
2. Powerful abstractions for distributed simulation processing backed by future
3. Automated generation of interactive R Markdown simulation documentation, situating results next to the workflows needed to reproduce them.

Installation

simChef is under active development. To install the package directly from GitHub, please use:

r devtools::install_github("Yu-Group/simChef")

Example Usage

Consider the following toy simulation experiment, where we want to study the prediction accuracy of linear regression and random forests under both linear and non-linear data-generating processes for varying signal-to-noise ratios.

Let us first code up the necessary simulation components, namely, the linear and nonlinear (here, an exclusive-or) data-generating processes as well as the linear regression and random forest models. To evaluate the methods and visualize the results, one can also write custom code, but we will leverage built-in evaluation and visualization functions (e.g., summarize_pred_err and plot_pred_err) from simChef for convenience.

```r

Generate data via linear model

lineardgpfun <- function(ntrain, ntest, p, beta, noisesd) { n <- ntrain + ntest X <- matrix(rnorm(n * p), nrow = n, ncol = p) y <- X %*% beta + rnorm(n, sd = noisesd) datalist <- list( Xtrain = X[1:ntrain, , drop = FALSE], ytrain = y[1:ntrain], Xtest = X[(ntrain + 1):n, , drop = FALSE], ytest = y[(ntrain + 1):n] ) return(datalist) }

Generate data via exclusive-or model

xordgpfun <- function(ntrain, ntest, p, thresh, beta, noisesd) { n <- ntrain + ntest X <- matrix(rnorm(n * p), nrow = n, ncol = p) xor <- (((X[, 1] > thresh) + (X[, 2] > thresh)) == 1) y <- beta * xor + rnorm(n, sd = noisesd) datalist <- list( Xtrain = X[1:ntrain, , drop = FALSE], ytrain = y[1:ntrain], Xtest = X[(ntrain + 1):n, , drop = FALSE], ytest = y[(ntrain + 1):n] ) return(datalist) }

Fit linear regression model

linearregfun <- function(Xtrain, ytrain, Xtest, ytest) { traindf <- dplyr::bindcols(data.frame(Xtrain), y = ytrain) fit <- lm(y ~ ., data = traindf) predictions <- predict(fit, data.frame(Xtest)) return(list(predictions = predictions, ytest = ytest)) }

Fit random forest model

rffun <- function(Xtrain, ytrain, Xtest, ytest, ...) { traindf <- dplyr::bindcols(data.frame(Xtrain), y = ytrain) fit <- ranger::ranger(y ~ ., data = traindf, ...) predictions <- predict(fit, data.frame(Xtest))$predictions return(list(predictions = predictions, ytest = y_test)) } ```

From here, there is minimal coding on the user's end, as simChef provides a powerful tidy grammar to instantiate, assemble, and run various configurations of the simulation experiment.

```r library(simChef)

Uncomment to run experiment across multiple processors

library(future)

plan(multisession, workers = 5)

Create simChef DGPs (data-generating processes)

lineardgp <- createdgp( .dgpfun = lineardgpfun, .name = "Linear DGP", # additional named parameters to pass to .dgpfun() ntrain = 200, ntest = 200, p = 2, beta = c(1, 0), noisesd = 1 ) xordgp <- createdgp( .dgpfun = xordgpfun, .name = "XOR DGP", # additional named parameters to pass to .dgpfun() ntrain = 200, ntest = 200, p = 2, thresh = 0, beta = 1, noisesd = 1 )

Create simChef Methods

linearreg <- createmethod( .methodfun = linearregfun, .name = "Linear Regression" # additional named parameters to pass to .methodfun() ) rf <- createmethod( .methodfun = rffun, .name = "Random Forest", # additional named parameters to pass to .methodfun() num.threads = 1 )

Create simChef Evaluators

prederr <- createevaluator( .evalfun = summarizeprederr, .name = 'Prediction Accuracy', # additional named parameters to pass to .evalfun() truthcol = "ytest", estimate_col = "predictions" )

Create simChef Visualizers

prederrplot <- createvisualizer( .vizfun = plotprederr, .name = 'Prediction Accuracy Plot', # additional named parameters to pass to .vizfun() evalname = 'Prediction Accuracy' )

Create experiment

experiment <- createexperiment(name = "Test Experiment") |> adddgp(lineardgp) |> adddgp(xordgp) |> addmethod(linearreg) |> addmethod(rf) |> addevaluator(prederr) |> addvisualizer(prederrplot) |> # vary across noise parameter in linear dgp addvaryacross( .dgp = "Linear DGP", noisesd = c(0.1, 0.5, 1, 2) ) |> # vary across noise parameter in xor dgp addvaryacross( .dgp = "XOR DGP", noise_sd = c(0.1, 0.5, 1, 2) )

Run experiment over n_reps

results <- runexperiment(experiment, nreps = 100, save = TRUE)

Render automated documentation and view results

render_docs(experiment) ```

Simulation experiment complete!

In addition, the code, narrative, and results of the simulation experiment have been automatically rendered into an interactive html document via R Markdown (see ? render_docs), such as the one shown below:

For a more detailed walkthrough of this example usage, please see vignette("simChef").

For examples of real-world case study using simChef to develop novel statistical methodology, please check out:

• Boileau et al. (2022). A Flexible Approach for Predictive Biomarker Discovery. (GitHub, simChef docs)
• Huang et al. (2025). Distilling heterogeneous treatment effects: Stable subgroup estimation in causal inference. (GitHub, simChef docs)

More examples of the rendered documentation for different simulation experiments:

• Toy Example 1
• Toy Example 2

Grammar of a simChef Simulation Experiment

The simChef API distills a simulation experiment into four modular concepts, two of which are optional (but highly recommended): data-generating processes (DGPs), methods, evaluation (optional), and visualization (optional). simChef takes an object-oriented approach to encapsulate these simulation concepts, using R6 classes to make them concrete. These four classes are:

• DGP: corresponds to the data-generating process from which to generate data.
  • DGPs simply generate data in a reproducible and flexible manner, in the size and manner that you specify. For a library of preset but highly customizable DGPs, simChef has a sibling R package, dgpoix (currently in early development).
  • Ex: In the above example usage, there are two DGPs: the linear DGP and the exclusive-or DGP.
• Method: corresponds to the method (or model) to fit to the data in the experiment.
  • Methods can be either a new method under study, a baseline comparison method, or any means by which to transform the simulated data (i.e,. the output of DGP).
  • Ex: In the above example usage, there are two methods: linear regression and random forests.
• Evaluator: corresponds to the evaluation metrics/functions to evaluate the methods' performance.
  • Evaluators receive the results of the fitted methods and summarize them to produce meaningful statistics about the experiment.
  • Ex: In the above example usage, there is one evaluation function that evaluates the test prediction accuracy.
• Visualizer: corresponds to the visualization tools/functions to visualize results.
  • These visualizations can be applied directly to the raw method outputs, the evaluation transformations/summaries, or both. Visualizers can output anything that can be rendered in an R Markdown document: static or interactive plots, tables, strings and captured output, markdown, generic HTML, etc.
  • Ex: In the above example usage, there is one visualization function that visualizes the test prediction accuracy, averaged across experimental replicates.

A fifth R6 class and concept, Experiment, unites the four concepts above. More precisely, an Experiment is a collection of DGP(s), Method(s), Evaluator(s), and Visualizer(s), which are thoughtfully composed to answer a particular question of interest. An Experiment can also include references to DGP and/or Method parameters that should be varied and combined during the simulation run (see ? add_vary_across).

Using the DGP, Method, Evaluator, and Visualizer classes, users can easily build a simChef Experiment using reusable building blocks and customizable functions.

Once an Experiment has been constructed, users can finally run the simulation experiment via the function run_experiment(). As summarized in the figure below, running the experiment will (1) fit each Method on each DGP (and for each of the varying parameter configurations), (2) evaluate the experiment according to the given Evaluator(s), and (3) visualize the experiment according to the given Visualizer(s).

Origins of simChef

Related R packages

Below, we examine the main functionality of a number of existing tools for running reproducible simulation experiments that are currently available on CRAN and have been updated within the last couple of years.

• batchtools implements abstractions for "problems" (similar to our DGP concept), "algorithms" (Method in simChef), and "experiments". In addition to shared-memory computation via the parallel and snow packages, it also provides a number of utilities for working with high performance computing batch systems such as Slurm and Torque, which simChef supports via the future.batchtools package.
• simulator provides a similar tidy human-readable framework for performing simulations such as those common in methodological statistics papers. simulator includes code for running simulations in parallel, storing simulation outputs, summarizing simulation results with plots and tables, and generating reports, among many other features.
• SimDesign provides helper functions to define experimental conditions and then pass those experimental conditions to a user-defined data generation function, analysis function, and summary function. The package also provides a number of these functions for the user to choose from. Each experimental condition can be run over many replicates, computing results in parallel via the parallel package.
• simhelpers defines functions to calculate Monte Carlo standard errors of simulation performance metrics, generate skeleton simulation code, and evaluate in parallel across simulation parameters via the future package.
• The simTool package has two main functions: expand_tibble() and eval_tibble(). The former wraps the base R function expand.grid() to create a cartesian product of simulation functions and parameters, while the latter evaluates those functions in parallel via the parallel package.
• The parSim package implements a single function of the same name which allows for parallelization of arbitrary R expressions across replicates and simulation conditions. parSim uses the snow package to setup parallel backends.
• rsimsum is an R implementation of the Stata command simsum and provides helper functions for summarizing and visualizing the results of a simulation study.

Citing simChef

To cite simChef in publications, please use:

@software{duncan2024simchef, title={simChef: High-quality data science simulations in R}, author={Duncan, James and Tang, Tiffany and Elliott, Corrine F and Boileau, Philippe and Yu, Bin}, journal={Journal of Open Source Software}, volume={9}, number={95}, pages={6156}, year={2024} }