You must use the concepts you learn in this course throughout the rest of your program. To help you do this, you will build a couple of tools to help you during and after you finish this course. The first is a table of what statistical tests can be used when you have independent (or predictor) variables (IV) and dependent (or outcome) variables (DV) with various levels of measurement. The second is an analysis/critique checklist of things you should consider when performing an analysis or critiquing the statistical analyses used in a research report. You must add information to your copy of these tools and turn it in each week as we go through the course.

Which descriptive statistics should I use?

| | Nominal | Ordinal | Interval | Ratio | | --- | --- | --- | --- | --- | | | Categorical | Categorical | Continuous | Continuous | | Data Summary: | Frequency (counts) and Percentages | Frequency (counts) and Percentages | Means and Standard Deviations | Means and Standard Deviations | | Data Summary (outliers present or skewed data): | NA | NA | Medians and Ranges | Medians and Ranges | | Data Display: | Bar Charts | Bar Charts | Histograms and/or Box Plots | Histograms and/or Box Plots |

Which statistical test should I use?

| | | Independent
(or Predictor)
Variable (IV)
| | | | | :---: | :---: | :---: | :---: | :---: | :---: | |


| | Nominal | Ordinal | Interval | Ratio | | Dependent
(or Outcome)
Variable (DV)

| Nominal | | | | | |


| Ordinal | | | | | |


| Interval | | | | | |


| Ratio | | | | |

Statistical Analysis/Critique Steps

Hopefully, the statistics you calculate are more than just a collection of random facts about the data you have. Most of the time, individuals and organizations seek to answer a question or solve a problem using statistical data. As such, the statistics you calculate should be part of the supporting evidence of a well-crafted argument (Booth et al., 2016). Validity is an overarching concern of science and refers to the level of truth we ascribe to inferences (Shadish et al., 2002). This checklist aims to help you gather information relevant to establishing the validity of the inferences you seek to make with the statistics you calculate. The answers you provide to the prompts in this checklist are the logical foundation upon which any arguments you desire to make with the results of your analyses rest. Ensuring the integrity of your data and analyses is part of ethical statistical practice (American Statistical Association, 2022). Background information should not be overlooked when conducting and interpreting statistical analyses. It’s never a bad idea to consult a statistician during the planning stages of a study or analysis to ensure the data collection and analysis plan is sound (Berman & Gullion, 2007). This checklist will also help you gather and prepare the information needed to work with a statistician or critique the work of others.

| Analysis/Critique Checklist | | --- | | | | --------------------Section #1: Study Information (The Big Picture)-------------------- | | What is the source of the data you’ve gathered? Is the mechanism by which you’ve collected your data subject to bias? The questions in this section are geared toward data collected as part of a study, but the same concepts apply when analyzing data from an existing data source. If you are extracting data from an existing data source, you might consider by what mechanisms data were collected and entered into that data source and what query was used to extract the data from the data source. Are the data collection and entry methods subject to bias? Is the wording of the query sound, or are there potential errors or omissions that might lead to some of the relevant data being excluded from the dataset used for analysis? | | What is the general purpose of this study/analysis? (exploration, hypothesis confirmation, quality improvement, learning, other?) What are the implications for how test-wise and study-wise error rates, inferences from tests, and generalizability of results are handled? | |



| | What are the research questions for this study? Have the investigators hypothesized any outcomes? | |



| | What is the study design? Is the study qualitative, quantitative, or mixed methods? If the study has a quantitative design, is it experimental (manipulation if an independent variable, control group, random assignment), quasi-experimental (manipulation of an independent variable, either a control group or random assignment but not both), or pre-experimental (observational or descriptive studies that lack manipulation of an independent variable, lack random assignment, lack control group) (Shadish et al., 2002)? What threats to validity may be applicable to this study design? Are there any procedures that can be put in place to address these threats (Shadish et al., 2002)? What type of sampling was used for the study? What data collection methods were used? | |



| | What were IRB status and procedures? Are there any other ethical considerations necessary for this study? (American Statistical Association, 2022) | |



| | When was the study data collected? | |



| | How was the study data collected? | |



| | What were the data handling procedures? | |



| | --------------------Section #2: Dataset Information (Data Screening & Data Cleaning)-------------------- | | | | What are the variables in the dataset, and what is the level of measurement for each one? | |



| | Describe data accuracy. Any issues? If so, how will they be addressed? (Some potential things to check about data accuracy might include: data types match levels of measurement, typographical errors in data entry, values that don’t make sense for the phenomena represented by the variable, categories that don’t make sense for the phenomena represented by the variable, decisions about how to address problems identified) | |



| | Describe any additional data manipulation needed before analysis. (Do any variables need to be reverse-coded? Do summary scores for instruments need to be calculated? Do z-scores need to be calculated? Are there any other data transformations that need to be done to help meet statistical assumptions?)? | |



| | Describe missing data. Any issues? | |



| | Describe outliers. Any issues? | |



| | After initial data cleaning and review, have needed descriptive statistics been calculated before proceeding with analysis? What descriptive statistics will be needed for the study report? | |



| | Will any participant information be included in the study report? (APA Methods Participants section) | |



| | --------------------Section #3: Individual Test Information (Complete this section for each test run.)-------------------- | | | | $\textcolor{red}{Add \ this \ section:}$ Test selection & Power analysis: Before a statistical test can be conducted, the correct statistical test must first be selected. The test selected depends mainly on the levels of measurement of the data you have available and the characteristics of those data. You may also need to meet certain thresholds for the amount of data you have to be able to conduct a meaningful test. (Determining the amount of data needed is often accomplished by conducting a power analysis.) What statistical test(s) will be conducted? Which variable(s) will be used in those test(s)? Do you have the amount of data needed to get a meaningful result? | | Statistical test:

Adequate sample size (Power analysis):

| | What are the independent (or predictor) and dependent (or outcome) variables for the test, and what is the level of measurement for each one? | | IV:

DV:

| | NHST steps: | | These steps should be followed for each statistical test conducted as part of a study. | | 1. State the null and alternative hypothesis. | |



| | 2. Establish the criteria for rejection (alpha level). | |



| | 3. Calculate the test statistic. | |



| | 4. Make a decision about the null hypothesis. (Reject or fail to reject?) | |



| | --------------------References-------------------- | | American Statistical Association. (2022, February 1, 2022). Ethical guidelines for statistical practice. Retrieved October 10 from https://www.amstat.org/your-career/ethical-guidelines-for-statistical-practice

Berman, N., & Gullíon, C. (2007). Working with a Statistician. Topics in Biostatistics, 489-503.

Booth, W. C., Colomb, G. G., Williams, J. M., Bizup, J., & Fitzgerald, W. T. (2016). The craft of research (Fourth ed.). University of Chicago Press. https://doi.org/10.7208/chicago/9780226239873.001.0001

Field, A. (2018). Discovering statistics using IBM SPSS statistics (Fifth ed.). SAGE Publications, Inc.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Wadsworth, Cengage Learning.

Tabachnick, B. G., & Fidell, L. S. (2019). Using multivariate statistics (Seventh ed.). Pearson Education, Inc. |