mmiCATs

Mighty Metrika Interface to CATs

https://github.com/mightymetrika/mmicats

Repository

Mighty Metrika Interface to CATs

README.Rmd

---
output: github_document
---



```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
```

# mmiCATs




The goal of the Mighty Metrika Interface to Cluster Adjusted t-Statistics ('mmiCATs') R package is to provide ‘shiny’ web applications for CATs and to provide research tools for investigating when CATs models should be preferred over other statistical models used for cluster adjustment.

The implementations of CATs in 'mmiCATs' is based on the cluster.im.glm() function from the R package 'clusterSEs'. For more information on CATs see Esarey and Menger (2019).

## Installation

You can install the released version of 'mmiCATs' from [CRAN](https://CRAN.R-project.org):

```{r eval=FALSE}
install.packages("mmiCATs")
```


You can install the development version of 'mmiCATs' like so:

```{r, eval=FALSE}
# install.packages("devtools")
devtools::install_github("mightymetrika/mmiCATs")
```


## Shiny Application

Load the 'mmiCATs' package and call the mmiCATs() function to launch a 'shiny' web application which allows you to run the clusterSEs::cluster.im.glm() function on a csv dataset:

```{r}
library(mmiCATs)
```


```{r, eval=FALSE}
mmiCATs()
```


## CATs with Robust Models (for simulation research purposes only)

R packages such as 'robust' and 'robustbase' have functions which allow users to run robust alternatives to stats::lm() and stats::glm(). The 'mmiCATs' package has the functions cluster_im_lmRob() and cluster_im_glmRob() which take the basic algorithm used in clusterSEs::cluster.im.glm() but swaps out stats::glm() for a robust alternative. In the case of cluster_im_lmRob() either robust::lmRob() or robustbase::lmrob() get swapped in while in the case of cluster_im_glmRob() either robust::glmRob() or robustbase::glmrob() get swapped in. The example below shows some simple results comparing the methods for linear models.

```{r}
# Get common parameters
.form <- Sepal.Length ~ Petal.Length + Petal.Width
.clust <- ~ Species

# clusterSEs::cluster.im.glm()
glmout <- stats::glm(.form,
                     family = "gaussian",
                     data = iris)

clusterSEs::cluster.im.glm(glmout, dat = iris, cluster = .clust,
                           return.vcv = TRUE)

# robust::lmRob()
robustout <- robust::lmRob(.form, data = iris)

cluster_im_lmRob(robustout, .form, dat = iris, cluster = .clust,
                 engine = "robust", return.vcv = TRUE)

# robustbase::lmrob()
robustbaseout <- robustbase::lmrob(.form, data = iris)

cluster_im_lmRob(robustbaseout, .form, dat = iris, cluster = .clust,
                 engine = "robustbase", return.vcv = TRUE)
```


The simulation study in Esarey and Menger (2019) tested a few different methods for handling clustering. They found that a correctly specified mixed effects model tends to perform most efficiently; however, they found that CATs can outperform a mispecified mixed effects model. The pwr_func_lmer() function can be used to run a simulation where data is generated from a mixed effect model and results are compared between:

* The data generating mixed effects model
* A random intercept model
* A clusterSEs::cluster.im.glm(drop = TRUE, truncate = FALSE) model
* A clusterSEs::cluster.im.glm(drop = TRUE, truncate = TRUE) model
* A cluster_im_lmRob(drop = TRUE, engine = "robust") model
* A cluster_im_lmRob(drop = TRUE, engine = "robustbase") model

The models are compared on:

* Mean coefficient
* Rejection rate
* Rejection rate standard error
* Root mean square error
* Relative root mean square error
* Confidence interval coverage
* Average confidence interval width

The following example shows a simulation where both a random intercept and random slope are specified and where two of the variables (x1 and x3) are correlated. The simulation is limited to 5 reps to minimize computation time. The main variable of interest is variable x1; as such, the comparison metrics will be with respect to x1.

```{r}
pwr_func_lmer(betas = list("int" = 0, "x1" = -5, "x2" = 2, "x3" = 10),
              dists = list("x1" = stats::rnorm,
                           "x2" = stats::rbinom,
                           "x3" = stats::rnorm),
              distpar = list("x1" = list(mean = 0, sd = 1),
                             "x2" = list(size = 1, prob = 0.4),
                             "x3" = list(mean = 1, sd = 2)),
              N = 50,
              reps = 5,
              alpha = 0.05,
              var_intr = "x1",
              grp = "ID",
              mod = "out ~ x1 + x2 + x3 + (x3|ID)",
              catsmod = "out ~ x1 + x2 + x3",
              r_slope = "x3",
              r_int = "int",
              n_time = 100,
              mean_i = 0,
              var_i = 1,
              mean_s = 0,
              var_s = 1,
              cov_is = 0,
              mean_r = 0,
              var_r = 1,
              cor_mat = diag(2),
              corvars = list(c("x1", "x3")))
```


## CloseCATs

When summarizing simulation results, Esarey and Menger (2019) state, "in our simulations an accurate RE model of intra-cluster heterogeneity provides better performance than any cluster adjustment technique, but the cluster adjustment techniques perform better in the event of misspecification." Of the cluster adjustment techniques, the summary also mentions that, "Our simulation analysis finds that CATs (based on the work of Ibragimov and Muller (2010)) are the best choice among the options we examine for correcting SEs for clustering in data sets with a small number of clusters."

In practice, mixed effects models are often used to obtain cluster adjusted results. However, when the sample size is small, researchers often use a random intercept model (with no random slope) in order to obtain a model which is not too complex for the data. But if the true data generating process is consistent with a mixed effects model with a random slope then the random intercept model might be a misspecification.

CloseCATs is a card game where:

* Two cards are dealt to the computer and two to the player
* The card in row 1 represents the random slope variance
* The card in row 2 represents the random effect covariance
* The player has an option to switch their row 1 and row 2 card (the computer does not)
* Data is simulated from a mixed effects model defined by the game setup and the cards
* Simulated data is fitted to the data generating mixed effects model, a random intercept model, and CATs models
* A misspecification distance (rmse of CATs - rmse of random intercept) is computed for the computer and for the player
* The lower misspecification distance wins the game

The game was designed to help users get a better understanding of when to prefer a CATs model over a random intercept model. To play the game, call the CloseCATs() function to launch the 'shiny' application.

```{r, eval=FALSE}
CloseCATs()
```


# References

Esarey J, Menger A. Practical and Effective Approaches to Dealing With Clustered Data. Political Science Research and Methods. 2019;7(3):541-559. doi:10.1017/psrm.2017.42

Rustam Ibragimov & Ulrich K. Muller (2010) t-Statistic Based Correlation and Heterogeneity Robust Inference, Journal of Business & Economic Statistics, 28:4, 453-468, DOI: 10.1198/jbes.2009.08046 

Owner

GitHub Events

Issues and Pull Requests

Last synced: 10 months ago

Packages