Beware of testing too many hypotheses; the more you torture the data, the more likely they are to confess, but confession obtained under duress may not be admissible in the court of scientific opinion.
—Stigler (as cited in Mark & Gamble, 2009, p. 210)
Misinterpretation and abuse of statistical tests, confidence intervals, and statistical power have been decried for decades, yet remain rampant.
—Greenland, Senn, Rothman, Carlin, Poole, Goodman, & Altman, 2016, p. 337
In This Chapter
Common types of statistics used for quantitative data analysis are defined, along with methods for choosing among them. Computer software for quantitative analysis are discussed.
Interpretation issues relevant to quantitative data analysis are discussed, including randomization, sample size, statistical versus practical significance, cultural bias, generalizability, and options for reporting quantitative results, such as effect sizes and variance accounted for, replication, use of nonparametric statistics, exploration of competing explanations, recognition of a study’s limitations, and a principled discovery strategy.
Effect sizes as a part of statistical synthesis (i.e., meta-analysis) as a literature review method are explained.
Options for qualitative analysis are described, along with selected computer programs that are available.
Interpretation issues related to qualitative data analysis are discussed, including use of triangulated data, audits, cultural bias, and generalization of results.
Mixed methods analysis and interpretation issues are addressed.
Development of a management plan for conducting a research study is described as a tool to be included in the research proposal.
Writing research reports is described in terms of dissertation and thesis requirements, alternative reporting formats (including performance), and publication issues. Digital reporting and dissemination strategies are discussed.
Strategies are discussed for improving the probability of the utilization of your research results.
Strategies are discussed for improving the probability of the utilization of your research results.
By reading and studying this book, you have moved through the steps of preparing a research proposal or critiquing a research study to the point of data analysis. If you are preparing a research proposal, your next step is to describe the data analysis strategies that you plan to use. In most research proposals, this section is followed by a management plan that specifies what tasks you will complete within a specified time frame and what resources will be required to complete the research project. Then, you would be in a position to complete the research study itself and to write up the results. Thus, the organizing framework for this chapter is designed to take you through the data analysis and interpretation decisions, the design of a management plan, and ideas concerning writing and disseminating research. If your goal is to critique research (rather than conduct it yourself), you will find guidelines that will help you identify the strengths and weaknesses of this portion of a research study.
A final section addresses the utilization of research results. Although this section appears at the end of this text, ideas to enhance utilization have been integrated throughout the descriptions of the research planning process in this text. If you wait until after the research is finished to consider utilization, chances are that your research could become a “dust catcher” on someone’s shelf, an unused computer file, or an unvisited website. That would not be a happy ending after all your work, so it is important to build in strategies for utilization during your planning process.
Quantitative Analysis Strategies
Will struggling first-grade readers who participate in the Reading Recovery program make greater gains in their reading achievement than struggling readers who do not participate in that program (Sirinides, Gray, & May 2018)? How do experiences of discrimination relate to thoughts of dropping out among Latina/o students (McWhirter, Garcia, & Bines, 2018)? These are the types of questions for which researchers use quantitative research methods to investigate. Brief descriptions of two studies that explored answers to these questions are provided in Box 13.1. The analytic and interpretive strategies used in these studies are provided as examples of the various concepts described in this section of the chapter.
Commonly Used Quantitative Data Analysis Techniques
It is not possible to explain all the different types of statistics, the derivation of their formulas, and their appropriate uses in this chapter. The reader is referred to general statistics books for more specific information on this topic (see, e.g., Carlson, & Winquist, 2017; Field, 2017; Vogt, Vogt, Gardner, & Haeffele, 2014). First, I take you on a little side trip into scales of measurement territory, as the scale of measurement has implications for decisions about which statistic to us. Second, I discuss computer software programs used for quantitative data analysis and define and give examples of some of the more commonly used quantitative data analysis techniques. Then, I provide you with a model to aid you in making decisions about the most appropriate data analysis techniques. Finally, I discuss issues related to the interpretation of quantitative data analysis results.
Scale of Measurement
Before presenting definitions of commonly used statistics and explaining in detail the decision strategy for choosing a statistical procedure, I wish to digress for a moment to describe one concept on which the basis for choosing a statistical procedure rests—the scale of measurement. As a researcher you need to ask: What is the scale of measurement for the data for both the independent and dependent variables?
Box 13.1 Brief Descriptions of Two Quantitative Studies
Study 1: The Impacts of Reading Recovery at scale: Results from the 4-year i3 external evaluation (Sirindes et al., 2018)
The researchers wanted to test the effectiveness of Reading Recovery, a pull-out program that provides daily 30-minute one-to-one instruction that is in addition to the regular classroom instruction. They compared students in the Reading Recovery condition with students who received reading instruction “as usual” with supplemental supports. “Means in the treatment group are one third to one half of a standard deviation larger than the control group means (p. 13).” The authors conclude that this is evidence that Reading Recovery is “an effective intervention that can help reverse struggling readers’ trajectories of low literacy” (p. 16).
Study 2: Discrimination and other education barriers, school connectedness, and thoughts of dropping out among Latina/o students (McWhirter, Garcia, & Bines, 2018)
The researchers studied the relationship between Latina/o adolescents’ experiences with discrimination and their thoughts of dropping out of high school. Experience with discrimination was measured by rating 16 experiences that reflect discrimination (e.g., teachers think you are less smart) using a scale from 1 = never to 5 = daily. Thoughts of dropping out of school were measured with two items (e.g., I might drop out of school) using a scale from 1 = not at all true to 5 = very true. The researchers reported a significant relationship between experiences of discrimination and thoughts of dropping out of school. However, they also noted that the effect of discrimination could be reduced by enhancing a sense of school connectedness.
The four scales of measurement are defined and examples of each are provided in Table 13.1. The scale of measurement is important because it determines which type of statistical procedure is appropriate. As you will see later, this has an influence on deciding between parametric or nonparametric statistics as well as on the appropriate choice of correlation coefficient.
The choice of a statistical procedure is outlined in Table 13.2. Your choice will depend on the following factors:
Your research question, which can be descriptive, concerns the extent of relationships between variables, determines significance of group differences, makes predictions of group membership, or examines the structure of variables
The type of groups that you have (i.e., independent, dependent, repeated measures, matched groups, randomized blocks, or mixed groups)
The number of independent and dependent variables you have
The scale of measurement
Your ability to satisfy the assumptions underlying the use of parametric statistics
Each type of research question leads you to a different statistical choice; thus, this is the most important starting point for your decisions.
You are almost ready to dive into the different types of statistics, but I suggest that before jumping into complex statistical analysis, it is important to really understand what your data look like. Statisticians recommend that you always graph your data before you start conducting analyses. This will help you in several respects. First, you will be closer to your data and know them better in terms of what they are capable of telling you. Second, they will help you determine if you have met the appropriate assumptions required for different types of analyses. Third, you will be able to see if you have any “outliers”—that is, values for variables that are very different from the general group response on your measure.
Table 13.1 Scales of Measurement
Scale of Measurement
Definition
Example
Nominal
Categorical data
Color: red, green, blue
Label: male, female
Ordinal
Ranked data organized according to increasing or decreasing presence of a characteristic
Tallest to shortest
Sweetest to sourest
Heaviest to lightest
Interval
Equal intervals, but zero is arbitrary
Temperature
Ratio
Equal intervals, and zero is defined as meaning the absence of the characteristic
Weight, age, IQ, many personality and educational tests
Computers and Quantitative Analysis
It would be highly unusual for researchers to analyze their quantitative data by hand; there are many statistical packages that are available for this purpose. I grew up using SPSS; current versions are very intuitive (as opposed to versions from the 1960s). Many students will make their decision about the software package to use based on what their university supports. There are other criteria to consider in making this decision, such as the comprehensiveness of the package, the cost, the learning curve required, the training provided by the developer, the platform it runs on (Mac or Windows), and the types of data that can be analyzed. If you do a Google search for the top 10 statistical softwares, you will get thousands of hits. The top 10 software packages are listed as SPSS, SAS, Stata, Minitab, JMP, NCSS, SYSTAT, ANALYSE-IT, PSPP, and MedCalc. If you search for the top 10 free statistical packages, you get this list: SAS University Edition, GNU PSPP, Statistical Lab, Shogun, DataMelt, GNU Octave, Zelig, Develve, Dataplot, and SOFA Statistics. As each of these packages has evolved to be more user friendly, the processes for doing various analyses have become more intutitive. Most software packages provide video training at their websites with opportunities to practice with the software before purchase or download.
Statistics can be thought of as being descriptive (i.e., they describe characteristics of your sample), correlational (i.e., they describe the strength and direction of relationships), and inferential (i.e., they allow you to make group comparisons). Box 13.2 provides definitions of the most commonly used descriptive, correlational, and inferential statistics.
Box 13.2 Definitions of Commonly Used Statistics
Descriptive Statistics: Statistics whose function it is to describe or indicate several characteristics common to the entire sample. Descriptive statistics summarize data on a single variable (e.g., mean, median, mode, standard deviation).
Measures of Central Tendency
Mean: The mean is a summary of a set of numbers in terms of centrality; it is what we commonly think of as the arithmetic average. In graphic terms, it is the point in a distribution around which the sum of deviations (from the mean point) is zero. It is calculated by adding up all the scores and dividing by the number of scores. It is usually designated by an X with a bar over it (–X) or the capital letter M.
Median: The median is the midpoint in a distribution of scores. This is a measure of central tendency that is equidistant from low to high; the median is the point at which the same number of scores lies on one side of that point as on the other.
Mode: The mode is a measure of central tendency that is the most frequently occurring score in the distribution.
Measures of Variability
Range: The range is a measure of variability that indicates the total extension of the data; for example, the numbers range from 1 to 10. It gives the idea of the outer limits of the distribution and is unstable with extreme scores.
Standard Deviation: The standard deviation is the measure of variability—that is, the sum of the deviations from the mean squared. It is a useful statistic for interpreting the meaning of a score and for use in more sophisticated statistical analyses. The standard deviation and mean are often reported together in research tables because the standard deviation is an indication of how adequate the mean is as a summary statistic for a set of data.
Variance: The variance is the standard deviation squared and is a statistic used in more sophisticated analyses.
Correlational Statistics: Statistics whose function it is to describe the strength and direction of a relationship between two or more variables.
Simple Correlation Coefficient: The simple correlation coefficient describes the strength and direction of a relationship between two variables. It is designated by the lowercase letter r.
Coefficient of Determination: This statistic is the correlation coefficient squared. It depicts the amount of variance that is accounted for by the explanatory variable in the response variable.
Multiple Regression: If the researcher has several independent (predictor) variables, multiple regression can be used to indicate the amount of variance that all of the predictor variables explain.1
Inferential Statistics: Statistics that are used to determine whether sample scores differ significantly from each other or from population values. Inferential statistics are used to compare differences between groups.
Parametric Statistics: Statistical techniques used for group comparison when the characteristic of interest (e.g., achievement) is normally distributed in the population; randomization is used in sample selection (see Chapter 11) and/or assignment (see Chapter 4), and the interval or ratio-level of measurement is used (e.g., many test scores).
t tests: Inferential statistical tests are used when you have two groups to compare. If the groups are independent (i.e., different people are in each group), the t test for independent samples is used. If two sets of scores are available for the same people (or matched groups), the t test for correlated samples is used.
ANOVA: The analysis of variance is used when you have more than two groups to compare or when you have more than one independent variable.
ANCOVA: The analysis of covariance is similar to the ANOVA, except that it allows you to control for the influence of an independent variable (often some background characteristic) that may vary between your groups before the treatment is introduced.
MANOVA: The multivariate analysis of variance is used in the same circumstances as ANOVA, except that you have more than one dependent variable.
Structural Equation Modeling: SEM is used to test complex theoretical models or confirm factor structures of psychological instruments. It can assess relationships among both manifest (observed) and latent (underlying theoretical constructs) variables. For further information, see Vogt et al. (2014).
Nonparametric Statistics: Statistical techniques used when the assumption of normality cannot be met with small samples sizes and with ordinal (rank) or nominal (categorical) data.
Chi-Square: Used with nominal-level data to test the statistical independence of two variables.
Wilcoxon Matched Pairs Signed-Ranks Test: Used with two related samples and ordinal-level data.
Mann-Whitney U Test: Used with two independent samples and ordinal-level data.
Friedman Two-Way Analysis of Variance: Used with more than two related samples and ordinal-level data.
Kruskal-Wallis One-Way Analysis of Variance: Used with more than two independent samples and ordinal-level data.
Descriptive Statistics
Researchers commonly report means and standard deviations for the descriptive statistics portion of their report. The usual format is to first state the mean and then show the standard deviation in parentheses immediately following the mean. Sirindes et al. (2018) used the Iowa Tests of Basic Skills to measure reading achievement. The results were as follows—experimental group: a mean of 138.8 with a standard deviation of 7.5; for Control Group 1: 135.4 (7.2). Sample size is usually indicated by the letter n and in this case, the sample sizes for the experimental and control groups in this study were identical: 3,444 in each group. In the McWhirter et al.’s (2018) study, they reported descriptive statistics (means and standard deviations) for the different measures: discrimination experience 1.80 (.69), educational barriers 1.69 (.48), and school connectedness 3.85 (.72). Their sample size was 819.
Correlational Statistics
McWhirter et al. (2018) wanted to test the strength of the relationship between their predictor variables and thoughts of dropping out of high school. They reported simple correlation coefficients between the variables:
Correlation analyses indicate that students with greater levels of perceived discrimination and educational barriers were more likely to have thoughts of dropping out (r = .20 and r = .22, respectively) and had lower levels of school connectedness (r = −.19 and r = −.31, respectively). Those with higher levels of school connectedness were less likely to have thoughts of dropping out (r = −.32). (p. 335)
The letter r is used to stand for the correlation coefficient statistic. They also chose to use a hierarchical regression technique that allowed them to test the relationship of individual predictor variables in the same statistical analysis. They found that each of the predictor variables contributed to the thoughts of dropping out. However, they also found that students with a higher level of school connectedness, despite their experience with discrimination, were less likely to think about dropping out (beta = .11, SE = .05, p < .01, F = 43.18, p < .001, and R2 = .07).
In English, this parenthetical expression would be read: Beta equals .11, standard error equals .05, and significance level of p is less than .01 for the racial discrimination variable. The F value is a test of the statistical significance of the full model of prediction of thoughts of dropping out when other barriers and school connectedness are considered. In English, this reads: F equals 43.18 and a significance level of p less than .001.
Beta is a standardized regression coefficient obtained by multiplying the regression coefficient by the ratio of the standard deviation of the explanatory variable to the standard deviation of the response variable. Thus, a standardized regression coefficient is one that would result if the explanatory and response variables had been converted to standard z scores prior to the regression analysis. This standardization is done to make the size of beta weights from regression analysis easier to compare for the various explanatory variables.
Researchers use the symbol R2 to indicate the proportion of variance in the response variable (in this case, thoughts of dropping out) explained by the explanatory variable (in this case, experience with racial discrimination) in this multiple regression. F is the statistic used to determine the statistical significance of this result. That is, is the contribution of the explanatory variable to the prediction of the response variable statistically significant? And p is the level of statistical significance associated with F. (Statistical significance is explained in the next section.)
Degrees of freedom indicate the appropriate degrees of freedom for determining the significance of the reported F statistic. F distributions are a family of distributions with two parameters—the degrees of freedom in the numerator of the F statistic (based on the number of predictor variables or groups) and those associated with the denominator (based on the sample size). If you know the number of explanatory variables and the sample size, the computer program will calculate the appropriate degrees of freedom and will use the appropriate sampling distribution to determine the level of statistical significance.
On the basis of these results, McWhirter et al. (2018) conclude that experiences of racial discrimination may increase students’ thoughts of dropping out of high school. They recognize the power of interventions that promote a feeling of connectedness with the school. They hypothesize that discrimination could be reduced by training peers, teachers, and staff to recognize and interrupt overt forms of discrimination. This might also include training to recognize covert discrimination in the form of lower expectations and reduction of referrals for disciplinary issues.
As noted in Chapter 5 on causal comparative and correlational approaches, researchers should not interpret correlation as meaning causation, as a correlational statistic can be calculated between any two variables. If a strong positive correlation was found between shoe size and income, the researcher could not conclude a causal relationship. It would be erroneous to conclude that increasing shoe size would result in higher incomes or that higher incomes would increase shoe size. However, finding a strong positive correlation between two variables does not mean that they are not causally related (e.g., number of cigarettes smoked and incidence of lung cancer). [Please excuse my double negative in the previous sentence; it seems to make the point clearly.]