You must use the concepts you learn in this course throughout the rest of your program. To help you do this, you will build a couple of tools to help you during and after you finish this course. The first is a table of what statistical tests can be used when you have independent (or predictor) variables (IV) and dependent (or outcome) variables (DV) with various levels of measurement. The second is an analysis/critique checklist of things you should consider when performing an analysis or critiquing the statistical analyses used in a research report. You must add information to your copy of these tools and turn it in each week as we go through the course.

Which descriptive statistics should I use?

| | Nominal | Ordinal | Interval | Ratio | | --- | --- | --- | --- | --- | | | Categorical | Categorical | Continuous | Continuous | | Data Summary: | Frequency (counts) and Percentages | Frequency (counts) and Percentages | Means and Standard Deviations | Means and Standard Deviations | | Data Summary (outliers present or skewed data): | NA | NA | Medians and Ranges | Medians and Ranges | | Data Display: | Bar Charts | Bar Charts | Histograms and/or Box Plots | Histograms and/or Box Plots |

Which statistical test should I use?

One Variable

| Level of Measurement | Test(s) | | --- | --- | | Nominal | Chi-Square Goodness-Of-Fit | | Ordinal | Chi-Square Goodness-Of-Fit | | Interval | one-sample t-test | | Ratio | one-sample t-test |

One Dependent Variable (DV), One Independent Variable (IV)

| | | Independent
(or Predictor)
Variable (IV)
| | | | | :---: | :---: | :---: | :---: | :---: | :---: | |


| | Nominal | Ordinal | Interval | Ratio | | Dependent
(or Outcome)
Variable (DV)

| Nominal | Chi-Square Test of Independence (Association)
(Dependent Samples) McNemar's Test | | | | |


| Ordinal | | (2 variable relationship) Spearman Rho correlation | (2 variable relationship) Spearman Rho correlation | (2 variable relationship) Spearman Rho correlation | |


| Interval | (two groups) independent t-test
(skewed data) Mann-Whitney U
(3 or more groups) one-way ANOVA
(skewed data) Kruskal-Wallis test
(prediction) Simple Regression | (3 or more groups) one-way ANOVA
(skewed data) Kruskal-Wallis test
(prediction) Simple Regression | (two paired or repeated measures) paired t-test
(skewed data) Wilcoxon Signed-Ranks test
(2 variable relationship) Pearson Correlation
(skewed data) Spearman Rho
(prediction) Simple Regression | (two paired or repeated measures) paired t-test
(skewed data) Wilcoxon Signed-Ranks test
(2 variable relationship) Pearson Correlation
(skewed data) Spearman Rho
(prediction) Simple Regression | |


| Ratio | (two groups) independent t-test
(skewed data) Mann-Whitney U
(3 or more groups) one-way ANOVA
(skewed data) Kruskal-Wallis test
(prediction) Simple Regression | (3 or more groups) one-way ANOVA
(skewed data) Kruskal-Wallis test
(prediction) Simple Regression | (two paired or repeated measures) paired t-test
(skewed data) Wilcoxon Signed-Ranks test
(2 variable relationship) Pearson Correlation
(skewed data) Spearman Rho
(prediction) Simple Regression | (two paired or repeated measures) paired t-test
(skewed data) Wilcoxon Signed-Ranks test
(2 variable relationship) Pearson Correlation
(skewed data) Spearman Rho
(prediction) Simple Regression |

One Dependent Variable (DV), More than One Independent Variable (IV)

| | | Independent
(or Predictor)
Variable (IV)
| | | | | :---: | :---: | :---: | :---: | :---: | :---: | |


| | Nominal | Ordinal | Interval | Ratio | | Dependent
(or Outcome)
Variable (DV)

| Nominal | | | | | |


| Ordinal | | | | | |


| Interval | (2 categorical IV) two-way ANOVA
(skewed data) Kruskal-Wallis test
$\textcolor{red}{Add \ this \ test:}$ (prediction) Multiple Regression | (2 categorical IV) two-way ANOVA
(skewed data) Kruskal-Wallis test
$\textcolor{red}{Add \ this \ test:}$ (prediction) Multiple Regression |
$\textcolor{red}{Add \ this \ test:}$ (prediction) Multiple Regression |
$\textcolor{red}{Add \ this \ test:}$ (prediction) Multiple Regression | |


| Ratio | (2 categorical IV) two-way ANOVA
(skewed data) Kruskal-Wallis test
$\textcolor{red}{Add \ this \ test:}$ (prediction) Multiple Regression | (2 categorical IV) two-way ANOVA
(skewed data) Kruskal-Wallis test
$\textcolor{red}{Add \ this \ test:}$ (prediction) Multiple Regression |
$\textcolor{red}{Add \ this \ test:}$ (prediction) Multiple Regression |
$\textcolor{red}{Add \ this \ test:}$ (prediction) Multiple Regression |

Statistical Analysis/Critique Steps

Hopefully, the statistics you calculate are more than just a collection of random facts about the data you have. Most of the time, individuals and organizations seek to answer a question or solve a problem using statistical data. As such, the statistics you calculate should be part of the supporting evidence of a well-crafted argument (Booth et al., 2016). Validity is an overarching concern of science and refers to the level of truth we ascribe to inferences (Shadish et al., 2002). This checklist aims to help you gather information relevant to establishing the validity of the inferences you seek to make with the statistics you calculate. The answers you provide to the prompts in this checklist are the logical foundation upon which any arguments you desire to make with the results of your analyses rest. Ensuring the integrity of your data and analyses is part of ethical statistical practice (American Statistical Association, 2022). Background information should not be overlooked when conducting and interpreting statistical analyses. It’s never a bad idea to consult a statistician during the planning stages of a study or analysis to ensure the data collection and analysis plan is sound (Berman & Gullion, 2007). This checklist will also help you gather and prepare the information needed to work with a statistician or critique the work of others.

| Analysis/Critique Checklist | | --- | | | | --------------------Section #1: Study Information (The Big Picture)-------------------- | | What is the source of the data you’ve gathered? Is the mechanism by which you’ve collected your data subject to bias? The questions in this section are geared toward data collected as part of a study, but the same concepts apply when analyzing data from an existing data source. If you are extracting data from an existing data source, you might consider by what mechanisms data were collected and entered into that data source and what query was used to extract the data from the data source. Are the data collection and entry methods subject to bias? Is the wording of the query sound, or are there potential errors or omissions that might lead to some of the relevant data being excluded from the dataset used for analysis? | | What is the general purpose of this study/analysis? (exploration, hypothesis confirmation, quality improvement, learning, other?) What are the implications for how test-wise and study-wise error rates, inferences from tests, and generalizability of results are handled? | |



| | What are the research questions for this study? Have the investigators hypothesized any outcomes? | |



| | What is the study design? Is the study qualitative, quantitative, or mixed methods? If the study has a quantitative design, is it experimental (manipulation if an independent variable, control group, random assignment), quasi-experimental (manipulation of an independent variable, either a control group or random assignment but not both), or pre-experimental (observational or descriptive studies that lack manipulation of an independent variable, lack random assignment, lack control group) (Shadish et al., 2002)? What threats to validity may be applicable to this study design? Are there any procedures that can be put in place to address these threats (Shadish et al., 2002)? What type of sampling was used for the study? What data collection methods were used? | |



| | What were IRB status and procedures? Are there any other ethical considerations necessary for this study? (American Statistical Association, 2022) | |



| | When was the study data collected? | |



| | How was the study data collected? | |



| | What were the data handling procedures? | |



| | --------------------Section #2: Dataset Information (Data Screening & Data Cleaning)-------------------- | | | | What are the variables in the dataset, and what is the level of measurement for each one? | |



| | Describe data accuracy. Any issues? If so, how will they be addressed? (Some potential things to check about data accuracy might include: data types match levels of measurement, typographical errors in data entry, values that don’t make sense for the phenomena represented by the variable, categories that don’t make sense for the phenomena represented by the variable, decisions about how to address problems identified) | |



| | Describe any additional data manipulation needed before analysis. (Do any variables need to be reverse-coded? Do summary scores for instruments need to be calculated? Do z-scores need to be calculated? Are there any other data transformations that need to be done to help meet statistical assumptions?)? | |



| | Describe missing data. Any issues? | |



| | Describe outliers. Any issues? | |



| | After initial data cleaning and review, have needed descriptive statistics been calculated before proceeding with analysis? What descriptive statistics will be needed for the study report? | |



| | Will any participant information be included in the study report? (APA Methods Participants section) | |



| | --------------------Section #3: Individual Test Information (Multiple Regression)-------------------- | | | | Test selection & Power analysis: Before a statistical test can be conducted, the correct statistical test must first be selected. The test selected depends mainly on the levels of measurement of the data you have available and the characteristics of those data. You may also need to meet certain thresholds for the amount of data you have to be able to conduct a meaningful test. (Determining the amount of data needed is often accomplished by conducting a power analysis.) What statistical test(s) will be conducted? Which variable(s) will be used in those test(s)? Do you have the amount of data needed to get a meaningful result? | | Statistical test:

Adequate sample size (Power analysis - simulate samples of data based on expected model parameters):

| | What are the independent (or predictor) and dependent (or outcome) variables for the test, and what is the level of measurement for each one? | | IV:

DV:

For multi-level models:

What is the hierarchical structure of the data?

Level 3:

Level 2:

Level 1

Considerations for fixed and random effects (Fixed effects - interested in differences between levels. Random effects - interested in quantifying variability between levels, interested in generalizing beyond current observed levels, unbalanced design, categorical nuisance variable)

Will a random intercept be present? (Check intercepts in plot)

Will random slopes be present? (Check slopes in plot.)

Model to be fit, including the fixed effects in the model and random effects in the model:

Do variables need to be centered for the model to converge (grand mean centering, group mean centering)?

| | What are the assumptions for the statistical test used? Do the data in the dataset meet those assumptions? If not, should an alternative test be selected or are any adjustments necessary to conduct a meaningful test? | | $\textcolor{red}{Update \ this \ section:}$
sample is randomly selected from population
(not an assumption for multi-level models)Observations are independent
data are normally distributed (residuals are normally distributed)
linear relationship between predictor and outcome
homoscedasticity
| | NHST steps: (for the regression model overall) | | For regression analyses, these steps are essentially followed twice. Once for evaluating the model overall, and once for evaluating the model parameters individually.
| | 1. State the null and alternative hypothesis. | | $\textcolor{red}{Update \ this \ section:}$
H0: There is no difference between the null model (intercept only) and the model with predictors.

H1: There is a difference between the null model (intercept only) and the model with predictors.

| | 2. Establish the criteria for rejection (alpha level). | | Typically alpha = .05



| | 3. Calculate the test statistic. | | $\textcolor{red}{Update \ this \ section:}$
Plot the main variables split by groups (Do intercepts and slopes vary across contexts?)
Fit the model to the complete data
Fit the full model with random effects
Interpret overall F statistic for fixed effects
Check if random effects intercepts and slopes are non-zero and what the relationship is between intercepts and slopes
Interpret the size and direction of the parameter estimates for fixed effects

Fit a fixed-effects model first

F(df 1, df 2) = [F-value], p = [p-value]

Assessing model fit and comparing multilevel models:

AIC (goodness-of-fit corrected for model complexity, smaller values mean better fit for full ML estimation for nested models):

BIC (more conservative than AIC, corrects harshly for number of parameters, use when sample size is large and number of parameters is small):

| | 4. Make a decision about the null hypothesis. (Reject or fail to reject?) | | $\textcolor{red}{Update \ this \ section:}$
reject or fail to reject null model?

| | NHST steps: (for the model parameters) | | For regression analyses, these steps are essentially followed twice. Once for evaluating the model overall, and once for evaluating the model parameters individually.
| | 1. State the null and alternative hypothesis. | | $\textcolor{red}{Update \ this \ section:}$
H0: b0 = 0

H1: b0 ≠ 0

| | 2. Establish the criteria for rejection (alpha level). | | Typically alpha = .05



| | 3. Calculate the test statistic. | | $\textcolor{red}{Update \ this \ section:}$
t (standard error) = [t-value], p = [p-value]

| | 4. Make a decision about the null hypothesis. (Reject or fail to reject?) | | $\textcolor{red}{Update \ this \ section:}$
reject or fail to reject H0?

| | Are any post-hoc or follow-up analyses needed? | | $\textcolor{red}{Update \ this \ section:}$
Regression diagnostics:

Model checking:
Autocorrelation test (Durbin-Watson) should NOT be significant
Collinearity statistics (VIF) should be under 5 and are problematic if 10 or higher
Normality test (Shapiro-Wilk) should not be significant, Q-Q plot of standardized residuals and theoretical quantiles should show normal distribution
Residual plots for predictors should be evenly distributed around 0. This helps show the linearity, independence, and homoscedasticity assumptions are met

Identify influential cases using Cook's distance:
Values for Cook's distance greater than 1 are potentially problematic, and it should be explored why those cases are different and how they impact the analysis.
| | What is the effect size? What is the interpretation of the effect size? | | $\textcolor{red}{Update \ this \ section:}$
Model overall:
R squared =

Each parameter:
standardized coefficient
| | Can a Bayes Factor be calculated? What is the interpretation of the Bayes Factor? | | B01 (null supported) or B10 (alternative supported) = (1-3 negligible evidence, 3-20 moderate evidence, 20-150 strong evidence, >150 very strong evidence)



| | How will you report results? | | $\textcolor{red}{Update \ this \ section:}$ A report about multiple regression should include these elements: Model summary (R squared, Adjusted R squared, F-statistics and p-value); Regression coefficients (Unstandardized coefficients B, standard error, t-value and p-value, standardized coefficients beta, confidence intervals); Interpretation of key findings; Assumption checks.
A multiple regression was performed to examine the effect of [predictors] on [outcome]. The overall model was statistically significant, F(df1, df2) = [F], p = [p], R² = [value], Adjusted R² = [value]. [List of significant predictors], with [interpret results]. Assumptions were [met/violated] based on [tests/checks].
| | --------------------References-------------------- | | American Statistical Association. (2022, February 1, 2022). Ethical guidelines for statistical practice. Retrieved October 10 from https://www.amstat.org/your-career/ethical-guidelines-for-statistical-practice

Berman, N., & Gullíon, C. (2007). Working with a Statistician. Topics in Biostatistics, 489-503.

Booth, W. C., Colomb, G. G., Williams, J. M., Bizup, J., & Fitzgerald, W. T. (2016). The craft of research (Fourth ed.). University of Chicago Press. https://doi.org/10.7208/chicago/9780226239873.001.0001

Field, A. (2018). Discovering statistics using IBM SPSS statistics (Fifth ed.). SAGE Publications, Inc.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Wadsworth, Cengage Learning.

Tabachnick, B. G., & Fidell, L. S. (2019). Using multivariate statistics (Seventh ed.). Pearson Education, Inc. |

Useful Prompts

See more prompts at https://github.com/cmcntsh/NURS6671ChecklistPrompts

Write results:

``` Please help me report the results of a . Please report some relevant descriptive statistics first, followed by the name of the test performed and the null hypothesis, followed by the relevant statistical test information with the effect size, and interpret the results for the reader. Here is the relevant information:

```