CHAPTER 3 LITERATURE REVIEW AND FOCUSING THE RESEARCH

CHAPTER 3 LITERATURE REVIEW AND FOCUSING THE RESEARCH

Although previous works have documented some of the challenges that African American males encounter in schools, there is a pressing need to examine the larger body of research on Black males and develop a comprehensive account of what we know about this population, what general trends exist in the literature about them, identify some of the strengths and problems with the current literature base, and be able to offer some recommendations for future research, theory, and practice on this population.

—Howard, 2013, p. 56

How does it feel to be a problem?

—W. E. B. Du Bois, 1903/2003, p. 7

In This Chapter

Two major reasons for conducting a literature review are explained: as a basis for conducting your own research or as an end in itself.

A nine-step process for conducting a literature review is outlined:

Development of the focus of your research

Review of secondary sources to get an overview of the topics

Development of a search strategy, including identification of preliminary sources and primary sources, and accessing personal networks

Conduct the search

Obtain full-text resources (e.g., journal articles or books)

Read and prepare reference information and notes on each resource; organize resources

Evaluate the research reports

Synthesize your findings

Use the literature review to develop a conceptual framework, solve a practical problem, and/or formulate research questions, hypotheses

Issues and questions related to the critical analysis of literature reviews are presented.

Why would you be interested in doing or reading a literature review? Isn’t it easier just to run down the hall and ask someone what they would do about a problem you encounter? Suppose you are a teacher with students who are immigrants or children of immigrants labeled in the school system as English language learners? How would you attempt to improve teacher-student-peer relationships? DeNicolo, Yu, Crowley, and Gabel (2017) took up the challenge of looking for solutions to these problems through conducting a review of research on school belonging for immigrant and immigrant-origin youth. They focused on literature that identified factors that enabled students to build on the cultural and linguistic knowledge that they brought with them to school and how that knowledge could contribute to fostering a sense of belonging in school.

When asked, Why do a literature review? a somewhat cynical answer may have popped into some of your minds: “Why do a literature review? It is required for my research class,” or “I have to do a thesis or dissertation.” Then, again, some of you may have more socially redeeming motivations, such as wanting to change the world or improve practice in your profession.

Literature reviews are important as research tools, especially in emerging areas, with populations that typically yield small samples (e.g., special education research often does) or in areas that represent value-laden positions adopted by advocacy groups. Literature reviews can serve to provide a more balanced understanding of what is known and what is not known about a topic, especially if you seek out diverse schools of thought on a topic. Literature reviews are also valuable in light of the knowledge explosion and the consequent impossibility of reading everything. Therefore, it is good that someone does literature reviews.

A few definitions will make your progress through this chapter more meaningful:

Preliminary sources: Databases that contain information about research articles that are published on the topic of interest to you

Secondary sources: Literature reviews that are published on your topic of interest consisting of a synthesis and analysis of previous research published on that topic (e.g., Chappell and Cahnmann-Taylor’s (2013) review of what is known about arts education)

Primary empirical research: Reports of studies that are conducted by the researcher(s) that include a description of the methods, sampling and data collection strategies, and data analysis and results

Grey literature: Unpublished or informally published work such as master’s theses, press releases, and Web documents

Reason Literature Reviews for Planning Primary Research

Almost every primary research study begins with a review of the literature. The purpose of the literature review section of a research article is to provide the reader with an overall framework for where this piece of work fits in the “big picture” of what is known about a topic from previous research. Thus, the literature review serves to explain the topic of the research and to build a rationale for the problem that is studied and the need for additional research. Boote and Beile (2005) eloquently explain the purpose of a literature review in planning primary research:

As the foundation of any research project, the literature review should accomplish several important objectives. It sets the broad context of the study, clearly demarcates what is and what is not within the scope of the investigation, and justifies those decisions. It also situates an existing literature in a broader scholarly and historical context. It should not only report the claims made in the existing literature but also examine critically the research methods used to better understand whether the claims are warranted. Such an examination of the literature enables the author to distinguish what has been learned and accomplished in the area of study and what still needs to be learned and accomplished. Moreover, this type of review allows the author not only to summarize the existing literature but also to synthesize it in a way that permits a new perspective. Thus a good literature review is the basis of both theoretical and methodological sophistication, thereby improving the quality and usefulness of subsequent research. (p. 4)s for Doing Literature Reviews

There are two major reasons for conducting a literature review: to get guidance on how to conduct primary research oneself (or as a part of a team) or as an end in itself.

Researchers use the literature review to identify a rationale for the need for their own study. Some of the specific rationales for your research that might emerge from your literature review include the following:

1. You may find a lack of consistency in reported results across the studies you have chosen to review and undertake research to explore the basis of the inconsistency. For example, Berliner, Barrat, Fong, and Shirk (2008) noted inconsistencies in research on high school dropouts; they suggested that the problem might be that researchers were not differentiating between high school dropouts who reenrolled and those who did not.
2. You may have uncovered a flaw in previous research based on its design, data collection instruments, sampling, or interpretation. For example, G. D. Borman et al. (2007) reviewed research on the Success for All literacy program and found that no randomized control studies had been conducted on its effectiveness. The quasi-experimental designs from past research left the findings open to possible criticism based on uncontrolled extraneous variables.
3. Research may have been conducted on a different population from the one in which you are interested, thus justifying your work with a different population. For example, Strassman and Schirmer (2013) reported that an evidence-based schema existed for research on reading based on the work of the National Early Literacy Panel. However, no such schema exists for research on effective writing instruction. In addition, the extant research on writing was primarily conducted with hearing students. Therefore, they “assess[ed] the evidence base for current approaches to writing instruction with deaf students and … compare[d] results to the body of research on hearing writers without disabilities” (p. 166).
4. You may document an ongoing educational or psychological problem and propose studying the effect of an innovative intervention to try to correct that problem. For example, DeNicolo et al. (2017) wanted to explore innovative pedagogical practices to engage students who were facing challenges stemming from poverty, class, race, religion, linguistic and cultural heritage, or gender. In particular, they were interested in how teachers engaged and supported students whose first language was not English.
5. Uncertainty about the interpretation of previous studies’ findings may justify further research. For example, prior research with people with schizophrenia indicated that participants sometimes continued to feel bewildered about their condition and treatment, even after meeting with a health care professional. A literature review might reveal how people with mental illness determine what contributed to their perceptions of effective and ineffective relations with professionals.

As mentioned previously, a literature review can be used at the beginning of the study to explain what is known about your topic and provide a rationale for the study you are planning. In addition, the literature review can be used to help in the design of the study by providing guidance as to appropriate sample size or identifying promising data collection practices or instruments that can be used in your study. Familiarity with the literature is useful for quantitative, qualitative, and mixed methods studies no matter what the researcher’s paradigm. Everyone who prepares a literature review should do so with a critical eye: What are the strengths and weaknesses of the prior research? What is missing from the formal body of scholarly literature that might be necessary in order to formulate an appropriate research focus and method of investigation?

When your purpose is to plan your own research study, the number of studies that you actually cite in your literature review may be fairly limited because of space limitations (for authors who publish in journals) or because the review is considered a learning activity (in your own course work). Typically, primary research articles published in journals contain 20 to 30 references to primary research. The number of citations may be quite limited for a course activity or more extensive if you are preparing a proposal for a thesis or dissertation. The exact number varies, depending on the purpose of the literature review and the extant literature. The primary criterion for inclusion should be centrality to your topic, within whatever constraints are imposed by instructors, advisers, or publishers.

Use of the literature review to plan and conduct a study requires that you critically evaluate the research that you read. This critical analysis can form the basis for your rationale or for your choice of data collection procedures; this is a topic discussed later in this chapter.

Review of Literature as an End in Itself

The review of literature can be seen as an end in itself, either to inform practice or to provide a comprehensive understanding about what is known about a topic. The process for conducting this type of literature review varies, depending on your purpose. If your purpose is to improve your professional practice, you will want to base your literature review on the problem you encountered in your profession. Therefore, when you look to the literature for a solution, you may rely on other people’s literature reviews, or you may seek out primary research reports until you find one that seems to fit your situation. For example, Mayo (2007) reviewed literature from the LGBTQ community with a specific focus on the act of “coming out” as it is researched in schools from the perspective of obstacles that the youth encounter as well as in terms of the agency and resiliency demonstrated by some youth. Mayo uses the literature review to suggest promising strategies for school leaders, youth, and researchers to make progress on this issue.

When a literature review is conducted to provide a comprehensive understanding of what is known about a topic, the process is much longer. For example, Howard (2013) included nearly 200 references in his literature review of the experiences of African American males in the educational system. The review was designed to

develop a comprehensive account of what we know about this population, what general trends exist in the literature about them, identify some of the strengths and problems with the current literature base, and be able to offer some recommendations for future research, theory, and practice. (p. 56)

Extending Your Thinking Literature Review Uses

When writing a literature review for the purposes of planning a research study, what are some of the uses that the literature review can serve for you?

Why is a literature review especially important in areas that (a) are emerging, (b) typically have small samples (e.g., special education research), or (c) represent value-laden positions adopted by advocacy groups (e.g., gender differences)?

Students receive different kinds of advice about how much literature to review and at what stage of the research process this should occur. What is your reaction to the following pieces of advice?

Cease your literature review:

When you have enough sense of the conversation to argue persuasively that the target for your proposed study is sound and that the methods of inquiry are correct; you know enough for the purpose of the proposal.

OR

When you feel familiar with the literature and generally what has been done on your topic, but you are not immersed in the details or chasing obscure references.

He framed his work by recognizing the deficit perspective of Black males (as reflected in the question from W. E. B. Du Bois, 1903/2003, at the beginning of this chapter) and sought literature that would disrupt these deficit-based perceptions. Chappell and Cahnmann-Taylor (2013) included over 250 references in their review of arts education in order to examine the changing place of the arts in education through a lens of power, culture, and representation. They draw conclusions for researchers and educators in terms of future directions suggested by the current body of scholarly knowledge in this area. Lee, Horvath, and Hunsley (2013) asked the very important question: Does it work in the real world? The “it” referred to the effectiveness of treatments for psychological problems in children and adolescents. They limited themselves to 20 effectiveness studies for the treatment of anxiety disorders, depression, and disruptive behavior problems. Later in this chapter, you will learn why they limited themselves to this small number of studies to answer such an important question.

The Search Process

No matter what the reason for the literature review or the paradigm within which the researcher is working, many aspects of the literature review process are the same. A general outline for conducting a literature review is provided in Box 3.1. Some of the differences in the process that emanate from paradigm choice include the following:

With the postpositivist paradigm, the researcher who plans to conduct experimental research needs to be able to develop a hypothesis (a best guess as to the outcome of the planned research) based on previous research. Quantitative researchers examine research in order to build a knowledge base of a topic that is sufficient to develop a hypothesis that can be tested and to benefit from guidance in terms of methodology found in prior studies.

With a constructivist orientation, the researcher should have a good understanding of previous research but remain open to possible emerging hypotheses that would require examination of additional literature during the study (Marshall & Rossman, 2011). Qualitative researchers (Corbin & Strauss, 2008) note that a literature review can be useful in order to decide on a research topic, to formulate a research plan, and to enhance the researcher’s awareness of subtleties uncovered in previous research. They do caution both novice and experienced researchers to be careful so that their perceptions of their findings emanate from their own data and not from expectations generated by reading extant literature. Some qualitative researchers actually suggest that the research questions be formulated before the literature review is conducted in order to guide the literature process. The wording of the questions can then be modified as necessary based on the literature.

In addition to a review of scholarly literature, researchers working within the transformative paradigm should consult with persons who have experienced oppression and seek out literature that represents their viewpoints (Mertens, 2009). This can include searching for grey literature, as minoritized communities may not have access to publishing in mainstream venues or they may choose to share their knowledge in other formats. In order to do this, researchers need to develop an understanding of themselves as individuals with potential biases as well as understand themselves in terms of their relationships with the community of interest. Hence, transformative researchers are more inclined to work with community members to develop the focus of the research rather than rely solely on extant literature.

Extending Your Thinking Literature Reviews and Qualitative Research

When conducting qualitative research, some texts advise against conducting a comprehensive literature review because it may bias the researcher to see “what others say they saw” instead of looking with fresh eyes. What do you think?

What are the advantages and disadvantages of engaging with members of the community you plan to work with as part of the process of developing the focus of your study?

Extending Your Thinking Literature Reviews and Qualitative Research

When conducting qualitative research, some texts advise against conducting a comprehensive literature review because it may bias the researcher to see “what others say they saw” instead of looking with fresh eyes. What do you think?

What are the advantages and disadvantages of engaging with members of the community you plan to work with as part of the process of developing the focus of your study?

In the following sections that describe the steps in the literature review process, the commonalities in the search process are described, along with recognition of appropriate caveats that differentiate work within alternative paradigms.

Step 1: Identify Research Topic

A few pieces of advice should guide (novice) researchers as they begin their literature review process. They should be flexible in their conceptualization of the research problem being investigated, and they should begin with a broad idea and be prepared to narrow it down as they progress through the search. Sometimes, students choose topics for research that turn out to be not very researchable (in that no one else has conceptualized the problem quite that way), and as they begin reading and seeing what is available, their ideas change as to what they want to investigate. Also, if the topic definition is too narrow, it may not be possible to identify any previous research that addressed that specific topic. Therefore, be flexible and start broadly. In my experience with students who are beginning a literature review, their topics shift as they become more familiar with the topic. Some students write me desperate emails explaining that they want to change their topics and they hope that this is OK. In most cases, I write them back to assure them that this is a normal part of an evolutionary process of developing the topic. (Only rarely do I think, what in the world is that student thinking!)

Sources of Research Topics

A research topic can emerge from a wide variety of sources, including the researcher’s interests, knowledge of social conditions, articles in newspapers, observations of educational and psychological problems, challenges that arise in one’s professional practice, readings in other courses, talking to other researchers, and the availability of funds to conduct research on a specific topic (sponsored research). Any of these is appropriate as a source to help identify the primary research topic. For researchers interested in conducting a comprehensive review of literature for its own sake, another criterion must be met: They must study topics that appear in the literature.

As an educator, psychologist, or other type of social researcher or service provider, you should be aware of the major issues that are challenging your field. For example, I conducted an extensive review of literature to determine what should be changed in the fifth edition of this text. Some of the topics that I saw repeatedly include (a) bullying and violence prevention; (b) preventing and treating addictions related to smoking, alcohol, and drug use; (c) the achievement gap in reading, writing, math, and science on the basis of race/ethnicity, gender, and disability; (d) the contribution of arts-based education; (e) strategies for assessment; (f) data-driven reform; (g) implementation of the Core Curriculum in the United States; (h) health disparities; (i) prevention and treatment of HIV/AIDS; (j) cultural and linguistic diversity in schools and therapy; (k) discrimination on the basis of sexual identity; (l) teacher evaluation based on student test scores; (m) school dropouts; (n) use of technology; and (o) early intervention. If you do not have a burning issue that you can identify from other sources, these topics clearly are of interest to our professional and grassroots communities. Here is an excerpt for a study that justifies the need for research on the topic of violence against teachers:

Violence directed against K–12 teachers is a serious problem that demands the immediate attention of researchers, providers of teacher pre-service and in-service training, school administrators, community leaders, and policymakers. Surprisingly, little research has been conducted on this growing problem despite the broad impact teacher victimization can have on schooling, recruitment, and retention of highly effective teachers and on student academic and behavioral outcomes. Psychologists should play a leadership role in mitigating school violence, including violence directed toward teachers. There is a need for psychologists to conduct research accurately assessing the types and scope of violence that teachers experience; to comprehensively evaluate the individual, classroom, school, community, institutional, and cultural contextual factors that might predict and/or explain types of teacher violence; and to examine the effectiveness and sustainability of classroom, school, and district-wide prevention and intervention strategies that target teacher violence in school systems. Collectively, the work of psychologists in this area could have a substantial impact on schooling, teacher experience and retention, and overall student performance. (Espelage et al., 2013, p. 75)

One important caveat that you should take into consideration: Members of many marginalized communities are tired of researchers coming into their communities to once again document their problems and then leaving without implementing any solutions that make things better. I refer you to the W. E. B. Du Bois (2003) quote and the sentiment expressed in the Cram, Chilisa, and Mertens (2013) quote at the beginning of this text. This desire for constructive, appropriate, research-supported solutions has been echoed by many different communities (Cram, 2009; Harris, Holmes, & Mertens, 2009; LaFrance & Crazy Bull, 2009). It is also exemplified in this quote from Europe: “We, the Roma, are the most analyzed people in history. We are tired of being the object of these studies that only benefit those who say always the same thing” (Garcia, Melgar, & Sorde, 2013, p. 367).

Thus, when you consider your research topic, ask yourself, how would I like to be considered the problem?

For sponsored research, the researcher needs to clarify with the funding agency what the research problem is from their perspective. Often, students can apply for funding to support their own research, usually with a faculty sponsor. When applying for funds, it is important to know what the agency is interested in sponsoring and to tailor one’s research interests to match those of the agency. Other students might work as research assistants to faculty members who have received financial support from an outside agency.

Scholars working in the transformative paradigm have been instrumental in stimulating research on a variety of topics that had previously received little attention, such as spousal abuse, sexual abuse, sexual harassment, homophobia, unpaid labor, violence, and motherhood and childcare. For transformative research, S. Harding (1993) recommends beginning with marginalized lives. To develop the research focus, the researcher might want to involve persons affected by the research through informal or formal means such as focus groups (Mertens, 2009). The following quotation from Schneider et al.’s (2004) study of people with mental illness illustrates the transformative effect of engaging participants in the process of developing a research topic.

There was also a real transformation in group members’ sense of themselves as people who could accomplish something. They had all been subjects in many research projects and, at the beginning of the project, could not conceive of themselves as people who could do research. By the end of the project, they had taken on a sense of themselves as researchers. They saw that they could articulate problems, come up with ways to investigate the problems, and produce solutions. This experience increased their awareness of themselves as people with resources and strengths who could make a significant contribution to society. (p. 575)

Step 2: Review Secondary Sources to Get an Overview

A good literature review written by someone else can provide you with an overview of what is known about your chosen topic. Specific places that you can look for literature reviews include journals that typically publish literature reviews, such as the Review of Educational Research, Harvard Educational Review, and the Psychological Bulletin, and books that contain literature reviews, such as the following:

Review of Research in Education (RRE): This series is published annually by the American Educational Research Association. Each volume contains a series of chapters on various topics—for example, Volume 40 (2016) provides a historical perspective on educational research and uses that as a foundation to identify the most challenging issues for educational researchers. Additional volumes address equity of school funding, literacy development for ELLs, and use of digital tools with young children.

The Annual Review of Psychology contains literature reviews on topics of interest in psychology and education, such as counselling or learning theory.

Research in Race and Ethnic Relations is published annually to address race relations and minority and ethnic group research.

Other handbooks have been published on specific topics:

Extending Your Thinking Selecting Your Research Topic and Setting

Students of research are sometimes given conflicting advice about the topic and site for their own research. The following pieces of advice exemplify such conflicts. Where do you stand on these two issues (i.e., choice of a research topic and setting) and why?

Select a research topic based on your personal experience and something that you feel passionately about. You will work on this topic for a long time, so it is important that you feel strong motivation to continue working on it.

Don’t pick a topic about which you have strong personal feelings because you may not be able to manage your own biases. The people you interview may be influenced by your personal experiences and biases.

A trend in research is now for members of marginalized communities to demand that the research be conducted by themselves, with themselves, and for themselves (Mertens, 2009). This is exemplified in Chilisa’s (2012) statement that research needs to start “with a call for recognition of the conceptual frameworks, theoretical frameworks, and data collection and analysis methods derived from the researched’s frames of reference and Indigenous knowledge” (p. 171).

Butcher, J. N. (Ed.). (2018). APA handbook of psychopathology. Washington, DC: Author.

Collins, P. H., & Solomos, J. (Eds.). (2010). The SAGE handbook of race and ethnic studies. Thousand Oaks, CA: Sage.

Florian, L. (Ed.). (2014). The SAGE handbook of special education (2nd ed.). Thousand Oaks, CA: Sage.

Fouad, N. A. (Ed.). (2012). APA handbook of counseling psychology. Washington, DC: Author.

Kitayama, S., & Cohen, D. (Eds.). (2007). Handbook of cultural psychology. New York, NY: Guilford.

Miller, L., Cameron, C., Dalli, C., & Barbour, N. (2018). The Sage handbook of early childhood policy. Thousand Oaks, CA: Sage.

Travis, C. B., & White, J. W. (Eds.). (2017). APA handbook of the psychology of women. Washington, DC: Author.

Step 3: Develop a Search Strategy

Four paths for search strategies are described in this section: (a) identify preliminary sources, (b) identify primary research journals, (c) access personal networks, and (d) involve community members. These are explained below. Decide which strategies are the best for you to follow in your search process, and remember, stay flexible.

Box 3.2 Most Frequently Used Preliminary Sources

ERIC: The Educational Resources Information Center (ERIC) is sponsored by the U.S. Department of Education and contains abstracts of journal articles and grey literature, such as research reports, conference presentations, and government reports. The database can be searched by going to www.eric.ed.gov or by using ERIC in commercial databases provided in many libraries. Most ERIC documents are available electronically, in print, or on microfiche in libraries. Many nonjournal materials are available, at no charge, as PDF documents or via links to publisher websites. Check with your local library (academic, public, etc.) to find out if they can provide journal articles or documents that are not available online. If the library cannot do this for you, print copies of journal articles can be purchased through such article reprint services as Ingenta (www.ingentaconnect.com).

ProQuest® has a number of databases, including Career and Technical Education, Education Journals, Psychology Journals, Social Science Journals, ComDisDom (related to hearing, speech, and language), and Dissertations and Theses. The database includes thousands of journals, many with full texts of the articles. ProQuest is available at many libraries.

JSTOR: This is a database of academic journals, monographs, and other academic papers from multiple disciplines, including the social sciences, humanities, and the sciences. It is available from libraries that subscribe to the service. Individuals in the United States can subscribe for a modest amount; JSTOR made special arrangements for individuals in Africa to have access to this database for free (www.jstor.org).

PsycINFO: This is a product of the American Psychological Association (APA) that contains indexes and abstracts from journals, books, and book chapters related to psychology, education, and linguistics. Both members and nonmembers can search the database and purchase articles. As with other databases, you can check on its availability at your local library.

PsycARTICLES: This is another product of the APA, but it includes the full-text articles of many journals related to psychology that APA or affiliated organizations publish. The database can be searched by APA members and by nonmembers (for a small fee).

Google Scholar is a free search engine that indexes full-text scholarly literature from a wide variety of disciplines. One nice feature of Google scholar citations is that they include a link to articles that have cited the article that you are reading.

Many university libraries also provide access to searchable databases that cover a wide range of journals and disciplines such as Academic Search Premier and Academic Search Complete.

World Wide Web (WWW) sites are easily and pervasively available to assist you in your literature searching. There are many search sites on the Web, and new ones appear with some regularity. In March 2018, the top three choices for search engines were Google, Bing, and Yahoo. These three search engines were recognized because of the size of their databases, ability to search HTML and PDF files, accuracy of results, and advanced searching power using Boolean logic. The search process on websites generally employs Boolean logic (explained later in this chapter) but can differ a bit from site to site. Because this is such a dynamic area, it is best to check PC Magazine or some other computer source to find out what sites are recommended for appropriate searching on your topic and to determine appropriate search strategies for those sites. One word of caution: The WWW sites do not have a peer review system to screen what is accepted (as most professional journals do); therefore, scholars raise questions about the quality of information available from those sources. In addition, the websites are not designed to contain information specifically about research in education and psychology as are the other databases described in this chapter. Many .com websites are intended to sell particular products; however, .edu, .gov, and .org websites can contain valuable content for a literature review. Students should be aware of the need to be skeptical of websites that promote a particular position and seek out sources that provide alternative points of view.

The computerized databases are a tremendous resource for the researcher in the literature review phase of a project. A researcher can identify thousands of references by only a few keystrokes on the computer. Because of the tremendous coverage provided by the databases, the researcher should plan to include a search of appropriate databases in the literature review process. Box 3.3 provides examples of how research teams described their methods of searching the literature. Notice that they did use the academic databases, but they also used many other sources to identify relevant studies.

Identify Preliminary Sources

Preliminary sources include databases and indexes that contain a compilation of bibliographic information, abstracts, and sometimes full-text articles for a wide range of topics and are accessible in print form, on compact disks (CD-ROM), or through online services.1 Examples of the most frequently used preliminary sources are listed in Box 3.2. Additional abstract and index services that specifically target marginalized groups include African Urban and Regional Science Index, Women’s Studies Abstracts, and Women’s Studies Index.

Box 3.3 Description of Literature Review Search Method

We began by examining prior systematic reviews, including those cited in the literature review as well as reviews of interventions, targeting reductions in related problem behaviors, such as delinquency, drug abuse, and mental health problems. We also reviewed program evaluations identified on web-based and published lists of best practices related to violence prevention, including the Blueprints for Violence Prevention database (Center for the Study and Prevention of Violence, 2010), the Campbell Collaboration (C2; http://www.campbellcollaboration.org/), the Communities That Care Prevention Strategies Guide (http://www.sdrg.org/ctcresource/), the Model Programs Guide (Office of Juvenile Justice and Delinquency Prevention, 2010), the Office of Justice Programs (CrimeSolutions.gov), and the National Registry of Evidence-based Programs and Practices (Center for Substance Abuse Prevention, 2010).

A final step involved searching the Medline, PsycINFO, Social Sciences Citation Index, and Web of Sciences databases with key words including aggression, crime, delinquency, prevention, youth, and violence. The reference lists of all studies reviewed were used to identify additional evaluations until a saturation point was reached.

—Fagan and Catalano (2013, p. 144)

We examined the research published from the end of March 2006 to the end of December 2011…. First, a PsycINFO search was conducted of journal articles published in English using the terms “treatment effectiveness evaluation,” “treatment effectiveness,” “effectiveness trials,” and “psychotherapy.” In addition, we used the general terms “parent training,” as well as naming three specific treatments that have been frequently evaluated in efficacy trials of treatments for child and adolescent disorders, “Parent-Child Interaction Therapy,” “Incredible Years,” and “Triple P.” Second, we searched the tables of contents of 31 journals that frequently publish treatment research…. In addition, as our third search strategy, we reviewed the reference lists of articles that were found to meet our inclusion criteria to obtain additional effectiveness studies that we had not identified with our initial two search strategies. As our fourth and final strategy, we also searched the references of recent reviews of child and adolescent treatment … for effectiveness studies.

—Lee et al. (2013, p. 82)

One important limitation should be noted about the available databases. You can get out of them only what was put into them. In other words, the databases are selective about the journals they include. For example, some feminist journals (listed in Box 3.4) are not listed in ERIC; they might not be considered appropriate for inclusion in ERIC because their content is not directly related to education or psychology. However, readers who are not familiar with feminist journals might find the list helpful as a way of broadening their resource base. For example, Hypatia publishes mainly philosophical work and thus would be of interest to those who want to delve more deeply into that aspect of feminist theory. Camera Obscura publishes work that could be of interest to scholars in educational media or social learning theory (e.g., the study of the power of media to shape cultural expectations by gender).

Box 3.4 Feminist Journals

Camera Obscura: A journal of feminist perspectives in film, television, and visual media.

Feminist Studies: The first feminist academic journal (started in 1972) is based at the University of Maryland. It publishes an interdisciplinary body of feminist knowledge and theory that makes visible assumptions about gender and sexual identity and the experiences and contributions of women across a wide range of difference.

Gender & Society: This journal emphasizes theory and research from micro- and macrostructural perspectives. It aims to advance both the study of gender and feminist scholarship.

Genders: Based at the University of Colorado, Genders publishes innovative work about gender and sexuality in relation to social, political, artistic, and economic concerns.

Hypatia: This journal publishes scholarly research at the intersection of philosophy and women’s studies.

Journal of Women’s History: An international journal that covers new research on women’s history, it includes scholarship about women in all time periods that is broadly representative of national, racial, ethnic, religious, and sexual grouping.

Meridians: A feminist, interdisciplinary journal with the goal of providing a forum for scholarship and creative work by and about women of color in U.S. and international contexts.

Signs: Journal of Women in Culture and Society: An international journal in women’s studies that publishes articles from a wide range of disciplines in a variety of voices—articles engaging gender, race, culture, class, sexuality, and/or nation.

Women’s Studies International Forum: The goal of this journal is to aid the distribution and exchange of feminist research in the multidisciplinary, international area of women’s studies and in feminist research in other disciplines.

Women’s Studies Quarterly: This journal focuses on teaching in women’s studies. Thematic issues include such features as course syllabi, discussions of strategies for teaching, and bibliographies.

Identify Primary Research Journals

Additional primary research articles can be identified by examining the reference lists found at the end of relevant journal articles or books. You can also go directly to journals that you know publish articles related to your topic. This is especially important in light of the selectivity of the databases discussed in the previous section. Researchers who are working from a transformative paradigm should be aware of the journals that deal with issues specific to marginalized groups, such as those in Box 3.4 for feminists, as well as journals such as Latin American Perspectives, Hispanic Journal of Behavioral Sciences, Journal of Multicultural Counseling and Development, Journal of Negro Education, and Journal of Black Studies. Disability, Handicap & Society is a journal that frequently focuses on the transformative paradigm in research with people with disabilities, and The Counseling Psychologist (2008) devoted an entire issue to the topic of multicultural counseling for psychologists and educators (Vol. 36, No. 2). A more extensive list of special education journals can be found in Box 3.5.

Box 3.5Selected Journals Containing Special Education Resource Information

American Annals of the Deaf

American Journal of Intellectual & Developmental Disabilities

Annals of Dyslexia

Australasian Journal of Special Education

British Journal of Special Education

Career Development for Exceptional Individuals

Disability and Society

Exceptional Children

International Journal of Disability, Development and Education

Journal of Autism and Developmental Disorders

Journal of Deaf Studies and Deaf Education

Journal of Learning Disabilities

Journal of Special Education

Journal of Speech, Language, and Hearing Research

Learning Disability Quarterly

Remedial and Special Education

Research in Developmental Disabilities

Volta Review (deafness)

Personal Networking

Additional resources can be found by talking to people who are doing work in areas related to your interest. This can include people at your own institution or those you meet through professional associations, such as the American Educational Research Association, the American Evaluation Association, the American Psychological Association, the Council for Exceptional Children, or the National Association of the Deaf. Talking to people who have completed related work can reveal sources that you were unaware of, such as unpublished research reports, and provide you with leads from work that is in progress for that researcher.

Two examples of well-planned and documented searches are provided in Boxes 3.6 and 3.7. As a researcher, it is always a good idea to carefully document your search strategy. In this way, you can backtrack to helpful strategies if you need additional information and indicate to the reader how thorough you were in your search process.

Involvement of Community Members

The combination of self-knowledge with cultural knowledge and skills in effective partnering facilitates the development of the research or evaluation focus and identification of questions, development of interventions, and making decisions about design, measures, samples, data collection, analysis, interpretation, and use that are in keeping with the philosophical assumptions of the transformative paradigm (Mertens, 2009). Following proper channels to enter a community is important, and strategies for doing this will vary by context. Some Native American Indian communities have developed specific protocols for anyone who wants to conduct research in their communities (LaFrance & Crazy Bull, 2009); Maori people from New Zealand have also developed similar protocols for research in their community (Cram, 2009). Deaf researchers have adapted the Indigenous Terms of Reference to suggest a protocol for research in that community (Harris et al., 2009). These protocols will be discussed in more depth in the chapter on sampling. However, it is important for researchers to know how to enter a community with respect, to communicate their intentions to members of the community in the appropriate way, and to make clear what benefits will accrue to themselves and to the community. Schneider et al.’s (2004) method of involving people with mental illness is one example of how community members can be involved in the decision process about what topics to study in the research.

Box 3.6Method Used in Literature Review: Example 1 Social Skills and Autism

The primary purpose of the literature review was to identify research that addressed the effectiveness of group-based social skill interventions for adolescents with higher functioning autism spectrum disorder (ASD).

Method

The researchers did not want to do a comprehensive review of this topic. They chose to do a selective review that used only the PsycINFO database. They also added other studies that were not found in PsychINFO that they knew about because of their expertise in the field. Several search terms were used, including these: Autism or autistic or pervasive developmental or Asperger, and Intervention or training or treatment, and Social.

A decision was made to limit the selection of publications to peer-reviewed journals written in English and published after 1990. The participants in the studies had to be school-age adolescents or young-adult age groups (ages 10–20).

1. The researchers used specific inclusion/exclusion criteria:
2. The study specifically focused on individuals with ASD.
3. The study assessed the efficacy of an intervention (i.e., reviews, meta-analyses, longitudinal follow-ups, theoretical papers, etc., were not included).
4. The intervention focused on social skill improvement (i.e., interventions focused on challenging behaviors, comorbid symptoms, communication, etc., were not included).
5. The intervention was delivered in a group-based format, and groups were primarily composed of other individuals with ASD.
6. The intervention included at least some participants within the 10- to 20-year-old age range.
7. The intervention data were not primarily analyzed using a single-subject design (p. 24).

Step 4: Conduct the Search

In conducting the search, you should make a plan to search preliminary sources; check the table of contents, abstracts, and lists of references in primary research journals; access your personal network; and involve community members as appropriate. The remainder of this section focuses on the search strategy as it applies to accessing preliminary sources.

Box 3.7Method Used in Literature Review : Example 2 African American males and achievement in school

A literature review of interventions related to improving achievement of African American males was conducted by using the following electronic databases:

EBSCO HOST (Academic Search Premier SocINDEX with full text), JSTOR, Google Scholar, Wilson Web Social Sciences Full Text, and ERIC.

1. He used the following search terms: African American males or boys, Black males or boys.
2. He used this review checklist to select studies:
3. was conducted in the United States;

used quantitative, qualitative, or mixed methods study designs;

was conducted within the last 12 years;

• reported disaggregated data on or about successful or high-achieving African American males, or effective pedagogies with them;
• was published in a peer-reviewed journal, scholarly book; and
• specially addressed African American/Black males (not males of color) (p. 58)

He eliminated studies with a deficit perspective that focused only on problems with Black males. He then selected those studies that focused on literacy, mathematics, discipline/and/or punishment, and teacher education or teacher practice.

SOURCE: Howard (2013).

Prepare to Search Preliminary Sources

Select the preliminary sources that you think contain the best information on your topic (see Box 3.2). Then identify key terms that will help you locate the literature included in the database of choice. One way that researchers select key terms is to find one primary research article that is “exactly” on target and identify the terms used to describe that article.

A search strategy based on using the ERIC online system is used to illustrate this process. The search strategy is similar when using other databases and indexes, such as PsycARTICLES and PsycINFO. Most databases give you many choices for searching, such as title, author, abstract, subject, or full text. The title, author, and abstract choices are fairly self-explanatory. Author and title are not usually used in the beginning of a literature review because you usually are not seeking a specific article during the early stages of searching. The subject choice needs a bit of explanation. Subject words are those that were used by the people who work for the database to categorize that item. These words are contained in a thesaurus, usually available in the online system. Each item in the database has a field associated with it that contains subject words that an indexer selected, and that is the field that is searched when you choose a subject word strategy. Full-text searchers, on the other hand, allow researchers to choose words that reflect their own vocabulary in the description of the topic. Full Text means the computer searches for these terms using a “free text” strategy; that is, it searches anywhere in the document for the words that you enter. Advantages and disadvantages accrue to whichever search strategy is chosen.

The easiest way to start is to use a key word strategy to determine if the words that you think are appropriate produce references that match your conceptualization of the problem. For example, for the topic of sexual abuse of Deaf students, I started in ERIC using sex abuse deaf as key words. The computer said there were no articles available that combined those three terms. I took a few minutes to read the directions in ERIC and found that I could use a multifield search strategy, separating the terms. So I used sex? AND abuse AND deaf?. (There is a good reason, explained later, for the inclusion of the ? and the word and in this search specification.) This resulted in 19 entries. One of the entries was Black and Glickman’s (2006) “Demographics, Psychiatric Diagnoses, and Other Characteristics of North American DeaIf you have difficulty finding references using your own key word vocabulary, check a thesaurus of terms to determine how the indexers might have conceptualized your topic. Use of subject descriptors can be helpful in narrowing down a search, as long as the descriptors are defined in a way that is compatible with your topic. They can also be helpful in broadening a search by suggesting other terms that could prove fruitful in searching.

Now, why include a? in the search terms, and what is the importance of the and in the list? You can refine your search in the following ways:

1. Truncate the terms you use. This has the effect of broadening the search to include any terms that begin with the letters that you enter, no matter how they end. In ERIC (the Educational Resources Information Center), the truncating symbol is a ? Therefore, entering sex? would include sex, sexual, sexes, and so on, and deaf? would include deaf, deafness, deafened, and so on.
2. Use Boolean or positional operators to combine terms. Boolean logic allows you to use the words and, or, not, and nor (one but not both words are in a record). Thus, asking for sex? and abuse and deaf? yields references in which the three terms appear in the same record. The or operator yields references that have either or both words in the same record. So, I could have asked for sex? abuse or child abuse and deaf? This would have given me all the records that contain sex? abuse or child abuse and deafness. In addition, I could have broadened my search by including deaf? or hearing-imp? This would have resulted in all references that had either deaf, hearing-impaired, or hearing impairment in their records.

Positional operators include same, with, adj, and near, and they limit retrieval by specifying how near key words must be to each other. Same means that both words must be in the same field of the same record (e.g., title or abstract); with means both words are in the same section of the same field of the same record; adj requires that the words must be next to one another (adjacent) in the order specified; and near finds references in which words are next to one another in any order (e.g., sex abuse or abuse sex).

• There are other ways to limit the search, such as by year of publication or limiting the field that is searched (e.g., title only). Certain stop words are not allowed to be used as key words (e.g., about, all, its), but all of these things can be learned by reading the online instructions. As you get into using a database, it is always a good idea to read the online instructions to see what can 4. Obviously, the search process can be broadened by inclusion of additional databases or indexes. For example, when I searched PsycARTICLES using the same descriptors (i.e., sex? and abuse and deaf?), I identified 57 additional references that did not overlap with those found in ERIC.
• Box 3.8

Box 3.8 Journal Citation Entry From ERIC

Black, Patricia A. Glickman, Neil S. Demographics, Psychiatric Diagnoses, and Other Characteristics of North American Deaf and Hard-of-Hearing Inpatients [Journal Articles. Reports—Research] Journal of Deaf Studies and Deaf Education. v11 n3 p303–321 2006 AN: EJ738331. By permission of Oxford University Press.

Abstract: This study examined demographic and clinical data from a specialty deaf inpatient unit so as to better understand characteristics of severely and chronically mentally ill deaf people. The study compares deaf and hearing psychiatric inpatients on demographic variables, psychiatric discharge diagnoses, a language assessment measure, a cognitive ability measure, and a measure of psychosocial functioning and risk of harm to self and others. Overall, findings indicate a broader range of diagnoses than in past studies with posttraumatic stress disorder being the most common diagnosis. Compared with hearing patients in the same hospital, deaf patients were less likely to be diagnosed with a psychotic or substance abuse disorder and more likely to be diagnosed with a mood, anxiety, personality, or developmental disorder. Psychosocial functioning of the deaf patients was generally similar to hearing psychiatric patients. Deaf patients presented significantly higher risks than hearing patients in areas of self-harm and risk of sexual offending. Cognitive scores show that both the deaf and hearing inpatient population is skewed toward persons who are lower functioning. An additional surprising finding was that 75% of deaf individuals fell into the nonfluent range of communication in American Sign Language. (Author)

A final word of advice: Cultivate a good relationship with your librarian. I invite the research librarian to my research classes to make a presentation on databases, search strategies, and documentation of findings from the search. Students report that visiting the research librarian is extremely helpful.

Interpret What You See

You can locate at least two types of documents in ERIC: journal articles and other references that are available through ERIC. Journals may be obvious because they have the name of the journal, but if you are not sure, look for an EJ code in the ERIC citation. Other references are noted with the abbreviation ED (education document) and are typically presentations made at professional meetings, curriculum guides, research reports, or other similar materials. An example of an ERIC full-text abstract for a journal article was presented in Box 3.8 and one for an ERIC document is in Box 3.9.

Box 3.9ED Document Citation From ERIC

Accession Number

ED477969

Author

Karcher, Michael J.

Title

The Hemingway: Measure of Adolescent Connectedness—Validation Studies.

Page Count

59

Peer Reviewed

No

Date of Publication

2001

ERIC Subject Headings

*Adolescent Development

Adolescents

Age Differences

Behavior Problems

Delinquency

*Interpersonal Relationship

*Psychometrics

*Social Environment

Substance Abuse

Test Validity

Identifiers

*Social Connectedness.

Abstract

This investigation reports the development of a measure of adolescent connectedness and estimates of its psychometric properties. A measure was developed to assess the ecological and developmental dimensions of adolescent connectedness, defined as adolescents’ caring for and involvement in specific relationships and contexts within their social ecology. Exploratory and confirmatory factor analyses in studies one and two yielded theoretically consistent factor solutions. These models were cross-validated in studies three and four with three geographically and ethnically diverse adolescent samples totaling 1454 adolescents. The measure of adolescent connectedness demonstrated satisfactory inter-item and test-retest reliability and convergent validity across samples. Consistent with social control and problem-behavior therapy, two higher order factors emerged across all of these samples: conventional vs. unconventional connectedness. These two dimensions of connectedness were found to differentially explain substance use for delinquent and non-delinquent adolescents. Using this ecological assessment, adolescent connectedness appears to differ as a function of age, sex, and problem-behavior status; varies across relationships and contexts; reflects either conventional or unconventional behaviors and attitudes; and can explain engagement in risk-taking behaviors. (Contains 62 references and 6 tables.) (Author)

Notes

Paper presented at the Annual Conference of the American Psychological Association (109th, San Francisco, CA, August 24–28, 2001).

Level of Availability

1

Publication Type

Information Analyses. Reports—Research. Speeches/Meeting Papers.

Language

English

Entry Month

200402

Select Titles

Most databases provide an abstract of the articles listed. By scanning these abstracts, you can make a decision as to the worth of obtaining the complete article. Advances in technology now also make it possible to view many full-text articles while you are engaged in the search process. Hence, researchers are faced with a bit of a paradox concerning the amount of time it takes to do a literature review. If you only have the abstract, you read it quickly and make a determination if it is what you want. If you think it is and full text is not available, then you need to go to the library or order the article through interlibrary loan or some other mechanism. If the full text is available, you may find yourself (like me) reading many articles because they are interesting and then you wonder where the day has gone and you have not made the progress that you expected. Intellectual curiosity is good; focus is also good.

Step 5: Obtain Full-Text Resources

As mentioned previously, many journal articles and books are now available online in full-text versions. If you cannot obtain the article in this manner, then it would be good to check the list of holdings at your library. If the journal you seek is held by your library, you are in luck: Go to the shelves (or the librarian at the help desk) and read the article. However, if your library does not have the item, you may avail yourself of an interlibrary loan service. If you provide complete bibliographic information, the librarian can determine which other library has the article and make a request to have it sent to you. There is often a small charge for this service. In some libraries, obtaining a copy of the article is available by an online request as you are doing your searcExtending Your Thinking Primary and Secondary Sources

What is the difference between a primary source and a secondary source? When and why would you choose to use one or the other? Have you been able to locate a secondary source on your topic of interest?

What search strategies have you found to be particularly effective in locating research on your topic of interest? Have you used networking? Professional associations? Why would these be important resources? What computerized databases have you used? Do you feel comfortable in using the computer to search for articles of interest for your research topic? Have you sought out journals or other sources of information (e.g., direct dialogue with individuals who are experiencing oppression) that represent the “transformative perspective” in research?

Some researchers recommend that you limit your literature review efforts to “mainstream” journals on the grounds that these represent the “cream of the crop” of research efforts. Transformative researchers might contend that this would result in a bias because the viewpoints of oppressed people might not be represented in those journals. Where do you stand on this issue?h. The computer may ask you if you want to order the document, and then it will tell you how much it costs to obtain. You have the option of agreeing to pay the cost, and if you agree, the library that holds the reference is electronically contacted and asked to transmit the article to your library. Amazing! (The researcher’s equivalent of the Home Shopping network.

If you have chosen to review an ED document from an ERIC search, that document may also be available in full text online. However, if it is not, then the document should be available for your review on microfiche in the library. The need to access documents on microfiche is becoming less and less common. The microfiche are organized in ascending order according to their ED numbers, so they are usually easy to find.

Step 6: Read and Prepare Bibliographic Information and Notes

Once you have the article in hand, read the document to determine if it is really what you want. You will want to record bibliographic information and notes on each article that you decide is relevant to your topic. This can be done electronically or manually, using old-fashioned note cards.

Bibliographic Information

If you are searching such databases as ERIC or PsycARTICLES, you can use a new digital resource called RefWorks to electronically save the bibliographic information about all the references that you select. When you are ready, RefWorks will print out a reference list in APA format (or the format that you select). That is not all: If the article is available in full text, you can save it in RefWorks with the bibliographic information. If you do not have access to this electronic resource, then you can save the bibliographic information on note cards, a word processing document, or a database file such as Excel. The important thing is to make sure you get ALL the information you need when you are working with the document so you do not have to try to find it later when you are writing up your literature review. Words of wisdom—painfully learned.

The most important thing to remember in recording bibliographic information is to be complete and accurate. Some of the problems associated with recording bibliographic information have been reduced because of the ability to print such information directly from the computer screen. However, if you have hundreds of printouts, you may want to record the information on index cards or in some other easily retrievable electronic format. (I do not always have a computer with me when I want to record bibliographic information, so index cards are handy, although the memo function on a mobile phone can also be useful. My secret at the moment: I cut and paste from the full-text documents and then type up the references later; next time I write a book, I plan to use RefWorks.)

Although several options are available for the format of recording bibliographic information, the most common style for education and psychology is based on the American Psychological Association’s (2009) Publication Manual (6th ed.). This is the basic format for a journal citation:

Book chapter:

Author’s Last Name, Initials. (date of publication). Title of journal article. Title of Journal, volume number(issue number), page numbers.

For example:

Black, P. A. & Glickman, N. S. (2006). Demographics, psychiatric diagnoses, and other characteristics of North American Deaf and hard-of-hearing inpatients. Journal of Deaf Studies and Deaf Education, 11(3), 303–321.

Book

Mertens, D. M. (2009). Transformative research and evaluation. New York, NY: Guilford.

Book Chapter

Author’s Last Name, Initials. (date of publication). Title of chapter. In Name of Editor (Ed.), Title of book (page numbers of chapter). Place of publication: Publisher.

For example:

LaFrance, J., & Crazy Bull, C. (2009). Researching ourselves back to life: Taking control of the research agenda in Indian Country. In D. M. Mertens & P. Ginsberg (Eds.), Handbook of social research ethics (pp. 135–149). Thousand Oaks, CA: Sage.

There are differences in citation style associated with different types of documents (e.g., books, chapters in books, government reports, etc.), so you are advised to obtain a copy of the APA Publication Manual (2009) to guide the compilation of bibliographic information. In addition, APA has added a great deal of information about how to handle Web-based information. Some of that is reflected in the Publication Manual, but you can also go to the APA’s website (www.apa.org) for an update on changes that they recommend or go to the APA style blog at http://blog.apastyle.org/.

The Purdue Online Writing Lab (https://owl.english.purdue.edu/owl/section/2/10/) also gives excellent examples of the many types of references you might encounter, including the digital object identifier (DOI) number when it is available for documents published online.

Notes on Each Study

Exactly what notes to write for each study varies greatly and depends on the nature of the study, the purpose of the review, and the intended use of the data. If the researcher intends to conduct a comprehensive literature review of studies that report their results in statistical form, the use of coding forms and computerized databases is recommended.

For empirical research studies (quantitative, qualitative, or mixed methods), the following outline can be helpful:

1. Area of interest; literature cited; need addressed; theoretical framework; research questions/hypothesis
2. Paradigm of researcher(s)
3. Design, including
4. Specific research approach
5. Sampling strategy
6. Characteristics of participants
7. Data collection instruments and procedures
8. Data analysis strategy
9. Results
10. Conclusions
11. Your own evaluation (including strengths and weaknesses and ideas for your own research, such as promising methodological or conceptual suggestions)

The evaluation of research reports is Step 7 (discussed next). Once you have evaluated the research report, you should return to your note cards or files and enter your own assessment of the strengths and weaknesses of the research.

Step 7: Evaluate the Research Reports

The criteria that are used to evaluate the research studies will depend on the design and data collection methods used in the study. You will be learning how to evaluate research as you progress through this text. A listing of critical analysis questions for evaluating primary research is provided at the end of each chapter. The questions are organized according to the sections of the research report (e.g., introduction, method, etc.), with additional specific questions relevant to each approach to research (e.g., experimental, quasi-experimental, etc.).

The APA Publications and Communications Board Task Force issued two reports (available at this textbook’s website) that are useful in this regard: one addressed criteria for reporting qualitative and mixed methods research (Levitt, Bamberg, Creswell, Frost, Josselson, & Suárez-Orozco, 2018) and another for quantitative research (Appelbaum, Cooper, Kline, Mayo-Wilson, Nezu, & Rao, 2018). The criteria for quantitative research distinguished between research that involved experimental manipulation, research that did not involve experimental manipulation, research with single subjects, and research in which data were collected on more than one occasion. Given that you have not yet studied designs, sampling, and so forth, you are not yet ready to apply the detailed guidance they provide when reading research studies. However, I provide you with a brief review checklist for quantitative, qualitative, and mixed methods studies based on the APA task force’s recommendations in Box 3.10. You will deepen your understanding of how to critically evaluate each part of a research study as you continue through this text and answer the questions that are relevant at the end of each chapter.

Step 8: Synthesize the Studies

The Harvard University Graduate School of Education (2018) defines synthesis in a literature review this way: To synthesize is to combine two or more elements to form a new whole. In the literature review, the “elements” are the findings of the literature you gather and read; the “new whole” is the conclusion you draw from those findings.

Before you actually begin the synthesis of the research, there are a few things to keep in mind. Organization is a plus. If you can develop a flexible framework for organizing the studies as you find them, it will be easier for you to approach the synthesis stage. I say flexible because the framework might add, delete, or redefine categories as you move through the review process. For example, for the revisions to the fifth edition of this book, I had categories for relevant studies I found in the early stages that related to each chapter title. As I found studies, I saved them into those files. As I began writing, I moved some of the files to more differentiated categories; for example, what started as the Chapter 1 introduction became paradigms, and paradigms became postpositivist, constructivist, transformative, and pragmatic. So as I approached writing, I had studies that were relevant to each part of the chapter. Of course, you sometimes run into problems in that one study may have applicability for more than one topic, but no one said this would be simple. I found using an Excel file with the author, date, approach, topic of the study, and the chapter number in which the article is cited helped me keep straight the many references in this book as I moved through the revisions. Some of my students like to keep tables of the studies they find and then organize the studies in tables under headings that make sense for their writing; some like to keep note cards on their studies that they can rearrange manually. Other students like to use a qualitative software package to save the articles as “data files” that can be coded and searched when it is time to write. This approach has merit but demands technical skills with the software and diligence to do the coding. However, once you get to the search-and-retrieve part, it may all seem worth it.

Box 3.10 Checklist for Evaluating Research Studies

Quantitative Studies

Introduction

Problem: Importance is clear

Literature review: Reports previous work and differences between that and current work

Hypothesis/aims/objectives: Specifically stated; based on literature/theory; connects to design

Method

Participants: Inclusion/exclusion criteria and demographic characteristics provided

Sampling: Sampling method clearly explained along with settings and locations; IRB approval; clearly explain decision basis for sample size and power of sample size

Data collection: Describe all instruments used; reliability; validity; administration procedures

Design: Specify design: experimental or not; randomization or not; longitudinal design; n-of-1

Data analysis, interpretation, results, discussion: Methods used to insure integrity of the data; analytic strategies used are appropriate; provide support for interpretations based on data; discuss relation to previous research, generalizability, and implications for future research

Qualitative Studies

Introduction

Problem/question: Literature is used to frame the problem or question and its context; applicable literature is reviewed and critiqued to clarify knowledge gaps or practical needs.

Objectives/aims/goals: Purpose/aims/goals are reported along with target audience; provides rationale for design and approach to inquiry.

Method

Research design: Summarize research design (data collection, analysis, approaches, etc.)

Researcher/participants/data sources: Describe researcher’s relevant background for conducting this study; provide numbers and characteristics of participants/data sources (e.g., newspapers, Internet); describe nature of relationships and interactions between researcher and participants; describe recruitment and selection processes.

Data collection: Identify form of data collected; alterations made over the course of the study; amount of time spent in data collection; provide questions asked during data collection; identify recording methods for data.

Analysis/interpretation/integrity/results: Describe methods used to analyze data, codes, coding process, software used.

Describe strategies used to insure the integrity of the methods and claims made based on the analysis. Support importance of findings, similarities and differences from prior research; limits of transferability.

Mixed Methods Studies

Introduction

Problem/question: Same as for quantitative (quant) and qualitative (qual) research

Method

Design: See quant/qual standards and explain why mixed methods (MM) is appropriate and the MM design used.

Participants: Same as quant/qual; clearly identify sources of each type of data and where they fit in the MM design; describe samples for quant and qual in separate sections.

Data collection: Same as quant/qual

Data analysis/interpretation/results: Same as quant/qual but have separate sections for quant and qual data reporting; be specific about integration of findings from each part of the study. Show integration of findings via discussion, joint table displays, or graphs.

SOURCE: Adapted from Appelbaum et al. (2018); Levitt et al. (2018).

Gingerich and Peterson (2013) conducted a review of literature on the effectiveness of solution-focused brief therapy (SFBT), an approach that focuses on strengths and solutions rather than on deficits and problems; it is therapy that is conducted in a brief time frame (usually less than six sessions). They developed a form to organize information associated with each study that

recorded problem type, setting (including country if outside the United States), SFBT techniques used, modality and duration of SFBT, type of comparison group and treatment used, sample size, key features of the study design, outcomes and measures used, pre-post change in the SFBT groups, and comparison of SFBT with the control group. (p. 268)

They found that their final set of 43 studies fell into six distinct groups:

Child academic and behaviour problems (14 studies)

Adult mental health (10 studies)

Marriage and family (6 studies)

Occupational rehabilitation (5 studies)

Health and aging (5 studies)

Crime and delinquency (4 studies) (p. 269)

They then organized their writing of the results using these categories.

Two main options exist for the synthesis of research studies: narrative and statistical methods. The choice of the type of synthesis depends on the type of extant research literature on a topic and on the purpose of the researcher. In this chapter, I focus on the narrative approach to synthesizing literature. The statistical approach (meta-analysis) is explained later in this chapter.

Narrative Synthesis

The narrative approach to literature synthesis is most commonly used in primary research studies. It is appropriate for studies that use qualitative, quantitative, and mixed methods designs. In a narrative synthesis, the writer must organize the studies in a conceptually logical order and provide sufficient detail about the studies to support relevant critical analysis of them. The amount of detail provided (as well as the number of studies cited) will be influenced by the purpose of the literature review:

Typically, the literature review section of a journal article includes a limited number of references that are selected on the basis of relevancy to the problem at hand, presenting a balanced picture, and establishing a rationale for the reported research.

A literature review for a research proposal is usually more extensive. If the research proposal is for a thesis or dissertation, it is expected to be quite comprehensive in most universities.

If you organized your literature into meaningful categories as you collected it, then this makes your writing easier. Provide an overview of your topic and describe the methods you used to search the literature. Then provide an advance organizer for the reader of the subtopics that you will address. For each study, make a determination if it is important to report details of its strengths and weaknesses in order to establish the overall picture of knowledge in the field or to provide support for your choice of methods. It is possible to explain several studies in detail and then cite other studies that agree or disagree with the findings of those studies rather than providing a detailed critique of every study in your literature review. Sometimes literature reviews include a separate section on the proposed study’s theoretical framework based on prior research. The literature review should lead to a statement of the need and purpose for the study, research questions, and hypotheses.

Statistical Synthesis: Meta-Analysis

Statistical syntheses are appropriate for experimental, quasi-experimental, causal comparative, and correlational research. However, even within these types, constraints apply. A statistical summary is not appropriate as a way of explaining the historical course of events; a narrative approach is needed for that purpose. Meta-analyses cannot in and of themselves take into account the heterogeneity of studies in terms of samples, diversity, and quality of methods, as such aggregation can obscure important differences.

Meta-analysis is one statistical approach to research synthesis that uses additional analyses to aggregate the findings of studies and reach conclusions about the overall effect of an independent variable. Meta-analysis is a quantitative synthesis of the existing research that uses effect sizes (see Chapter 13 for more explanation of the meaning of effect sizes and other statistical terms used in this chapter) as a way to standardize outcomes and facilitate comparisons across studies. Using primary source information from the literature, the meta-analyst establishes empirical evidence to increase understandings about a particular theory or intervention. In meta-analysis, the researcher addresses studies that have attempted to establish a causal relationship between some intervention and an outcome based on a review of available literature.

Wang, Jiao, Young, Brooks, and Olson (2008) conducted a meta-analysis to determine the effect of using computer-based testing (CBT) as compared to paper-and-pencil testing (PPT) in reading. They described their literature search process in detail and explained their inclusion and exclusion. Each study had to meet the following inclusion criteria:

The study had to be conducted between 1980 and 2005.

The samples of study had to be drawn from the K–12 student population, and the within-group sample size had to be larger than 25.

The study should have quantitative outcome measures (mean and standard deviation) of one of student achievement, aptitude, or ability of reading on both CBT and PPT.

The study should have the design to compare the scores from both CBT and PPT.

The test language in the study must be English because the major target population (U.S. K–12 students) in this report uses English. (p. 11)

Wang et al. (2008) calculated the effect size for each study in their sample. For studies that use means to compare an experimental and control group, the effect size is defined as the distance between the two-group means in terms of their common standard deviation (Lipsey & Wilson, 2000). It is calculated as follows:

Wang et al. (2008) calculated the effect size for each study in their sample. For studies that use means to compare an experimental and control group, the effect size is defined as the distance between the two-group means in terms of their common standard deviation (Lipsey & Wilson, 2000). It is calculated as follows:

Other

�

=

�

1

−

�

2

�

where

M1 = the mean or average for one group (e.g., experimental group)

M2 = the mean or average for the other group (e.g., control group)

S = the within-group standard deviation (measure of variability within groups)

d = the effect size or the difference between the two means

Thus, an effect size of 0.5 means that the two means are separated by half a standard deviation. This is a way of describing how well the average student who received the treatment performed relative to the average student who did not receive the treatment. For example, if an experimental group of students with behavioral problems received a drug treatment and the control group received a placebo, an effect size of 0.8 would indicate that the experimental group’s mean was 0.8 standard deviation above the control group.

Once the effect size for each study has been computed, an average effect size is calculated by combining the ds for each study. For example, Wang et al.’s (2008) meta-analysis of computer-based testing compared to paper-and-pencil testing resulted in an effect size of −.004 with a 95% confidence interval of [−.031, .023], which is not statistically significant at the alpha level of .01 (p = .782). Thus, the researchers concluded that examinees’ performance on paper-and-pencil tests was not significantly better than their performance on computer-based tests.

Interpretation of effect sizes is not clear-cut. J. Cohen (1988) proposes the following (somewhat arbitrary) guideline:

0.20 is small

0.50 is medium

0.80 is large

B. Thompson (2002b) argues that researchers should not use these benchmarks as rigid judgments about the significance of an effect size. Rather, he recommends that researchers present their effect sizes within the context of other studies to illustrate how replicable or stable the effect sizes are in a given area of inquiry.

The overly rigid use of fixed benchmarks for small, medium, and large effects fails to consider the possibility that small, replicable effects involving important outcomes can be noteworthy, or that large effects involving trivial outcomes may not be particularly noteworthy. (p. 30)

There are many pros and cons for deciding to use meta-analysis. It does offer a systematic and objective method of aggregating the results of a large number of studies on a specific topic. On the other hand, this approach suffers from the same threats faced in other research efforts—the exhaustive nature of the retrieval process and the diversity of research quality in the studies selected.

Mixed Methods and Literature Reviews

Some literature reviews, such as those using meta-analysis, focus only on quantitative studies, and some are limited only to research studies that used experimental or quasi-experimental designs. This was the practice of the Cochrane Collaboration (www.cochrane.org) and the Campbell Collaboration (www.campbellcollaboration.org), two major players in the conduct of systematic reviews of literature. However, limiting the analysis to quantitative studies of this nature means that a lot of information in the studies is lost. Therefore, mixed methods researchers have explored ways to integrate the findings of both quantitative and qualitative data into literature reviews (Mertens, 2018). In this process, both quantitative and qualitative data are extracted from the research studies. The quantitative data might provide insights into the effectiveness of interventions, but the qualitative data can answer questions such as what is the nature of the problem, what causes it, how is the intervention being implemented, what contextual factors influence its effectiveness, what is the reaction of the participants, and what are the strengths and weaknesses of the intervention? The qualitative data can also be used to address heterogeneity amongst the studies in terms of context, participants, and methods used.

Step 9: Use the Literature Review

The narrative or statistical synthesis serves as a basis for the literature section of a research proposal or report. The appendix contains an outline for a research proposal for a thesis or dissertation. It is important for the proposal writer to realize that each institution and sponsoring agency has its own requirements for proposal writing, so it is best to check with those sources before proceeding with writing. Proposal writers must also realize that in this synthesis of research they are “selling” their ideas to a research committee, institutional review board, or funding agency. So above all, make it clear why the research is important (based on what is known from the extant literature or community voices).

Extending Your Thinking Research Synthesis Approaches

1. What are the advantages and disadvantages of selecting either a narrative or statistical option for synthesizing research results? Discuss in terms of (a) quality of research studies used to reach conclusions, (b) the size of differences between groups, and (c) the limitations of meta-analysis. (Is the concept of a meta-analysis clear to you?)
2. Explore articles about mixed methods literature reviews. What do you learn about how to analyze the information in the articles and how to integrate the findings?
3. Lee et al. (2013) answered their question, Is treatment effective for psychological problems in children and adolescents? by using 20 carefully selected articles. The studies selected met these criteria: examined a psychological treatment designed to reduce behavioral or emotional problems in children or adolescents, was explicitly designed to test treatment effectiveness in the real world, cited at least one published efficacy study prior to the effectiveness study, provided data on the number of participants who started and finished the treatment, provided data on the improvement of the participants, and participants were either referred for treatment or met the diagnostic criteria for treatment. Their original search identified 148 studies, but only 20 met their criteria. Based on a meta-analysis of these studies, they concluded that completion rates for treatment were fairly high and that evidence does exist that treatments are effective for different diagnostic categories.

What is your reaction to this meta-analysis? How can 20 studies support the hypothesis that treatment is effective for children and adolescents? What are the strengths in this study?

Conceptual Framework and Program Theory

In some ways, the conceptual framework is like the chicken-or-the-egg controversy. A researcher’s original conceptual framework influences the planning and conducting of the literature review. However, if a researcher keeps an open mind throughout the literature review process, a more sophisticated and (often greatly) modified conceptual framework should emerge. Table 3.1 displays the influence of the theoretical framework on the choice of research questions and its implications for action. On the basis of work by Villegas (1991) on theoretical frameworks used to explain differential achievement by ethnic minority students, four different research questions are used to illustrate this point. The IQ deficit theory and the cultural deficit theory reflect a theoretical stance that suggests the problem is either “in the child” or “in the cultural group from which the child comes.” The cultural difference theory reflects the constructivist paradigm, and the power inequity theory reflects the transformative paradigm.

Table 3.1 Influences of Different Theoretical Frameworks on Research

Theory

Sample Research Question

Recommendations for Action

IQ deficit theory

Are minorities genetically inferior to White students?

Remedial education, but the problem is really “in” the child.

Cultural deficit theory (sociocultural deficits in home life)

Is there a higher rate of single-parent families among minorities? How do Black and White parents compare in discipline techniques?

Remedial education, but the problem is really “in” the family.

Cultural difference theory

What is the nature of language use at home and at school in terms of asking and answering questions or in seeking help?

Build on students’ prior experiences; increase their language use structures.

Power inequities (school failure is rooted in a struggle for power; schools play a role in the preservation of the socioeconomic order)

How can we teach minority students so they do not continue to be oppressed?

Explicitly teach minority children the means to access power, including linguistic forms and ways of talking, writing, and interacting. Teach them to value ethnic characteristics and that the culture of the dominant group is not necessarily superior.

These various explanations for poor academic achievement by ethnic minority children exemplify alternative theories that might be held by the researcher or by the research sponsor or participants. Researchers must be aware of their own personal theoretical base as well as that of the sponsors and the participants. For example, J. E. Davis (1992) noted that research on African American families often depicts them as deviant, pathological social organizations unable to fulfill the major responsibilities of socializing their members for productive roles in society (the deficit model). The conclusion based on this model, then, is that this undersocializationleads to negative outcomes, such as low academic achievement, juvenile delinquency, drug abuse, and teenage pregnancy. This conclusion is reached by ignoring the data that “inform us of the unique and often precarious position of African Americans” (p. 59). More than one third of the African American population in the United States lives at or near the poverty level. It is the economic condition and its implications (e.g., inadequate housing and food, poor sanitation, overcrowding) that bring about negative consequences, such as poor health, family violence, and delinquency. Ladson-Billings (2006) presents data that suggest that conditions have not improved for African American students since Davis wrote her work in 1992. Ladson-Billings suggests that there is a more insidious reason that underlies both economic and education deprivation: racism. Thus, the use of a theoretical framework that starts with the marginalized lives allows researchers to understand the experiences of oppressed groups.

In the past, much of educational and psychological research on racial or ethnic minorities, women, people with disabilities, and other marginalized groups derived from a deficit perspective that located the problem in individuals and focused on the negative reasons that they did not achieve or perform certain functions or activities (see Table 3.1). More recently, researchers have shifted to a social-cultural perspective that focuses on the dynamic interaction between the individual and environment over the life span (Howard, 2013). This focus on strengths and modifications of contextual factors has emerged under a variety of names such as positive psychology (Seligman, Steen, Park, & Peterson, 2005) and resilience theory (Liebenberg & Ungar, 2009). Such a theoretical framework has led to reframing research questions to focus on strengths. The following two questions are examples:

What are the positive aspects of parenting a deaf child? (Szarkowski, 2002)

What are the variables that contribute to successful transition from high school to college for deaf African American students? (Williamson, 2002)

Morse (2003) also notes that the theoretical framework in qualitative research is used to focus the inquiry and give it boundaries rather than to serve as the guide for data collection and analysis. Deductive analysis based on a static theoretical framework violates the assumption of constructivist qualitative inquiry. The theoretical framework should be viewed as a conceptual template with which to compare and contrast results, not seen as establishing a priori categories for data collection and analysis.

Research Questions and Hypotheses

The literature review serves as a foundation for forming research questions, operationalizing the objectives of the proposed research, focusing the research hypotheses, and clarifying what information needs to be collected from what sources under what conditions.

Framing the research questions can be a difficult task for beginning researchers. Four categories of questions can provide some guidance in this process: descriptive, normative, correlational, and impact. Each is briefly discussed in the following paragraphs.

Descriptive research questions are designed to produce information about what is or has been happening in relation to the target of the research. For example, the researcher might want to describe certain characteristics of the participants in an intervention. Alternatively, the researcher might be interested in describing the prevalence of a particular disability within an identified domain (e.g., What is the prevalence of the diagnosis of learning disabilities in Black middle school children?). Qualitative researchers often use descriptive questions, such as the following: What is the nature of the experience of Black males in urban high schools? What are the dynamics of change when a school is under state control for reform? What do people who are addicted to drugs experience when they are accepted into a rehabilitation program? In Milner’s (2012) study of the teaching styles of Black teachers, he formulated the following questions:

How does Ms. Shaw think about her role and teaching at Bridge Middle School? How and why does Ms. Shaw succeed with her students? In what ways does Ms. Shaw’s identity shape her perceptions and practices at Bridge Middle School? In what ways do or might Ms. Shaw’s perceptions and practices be consistent and inconsistent with current discourses about Black teachers? (p. 28)

Normative research questions go beyond description and require that the information generated in response to the descriptive research question be compared with some standard or expected observation. For example, in special education, there are minimum requirements regarding most aspects of the service delivery system. A normative research question might ask, Were Individualized Education Programs (IEPs) in place before the placement was made, in accordance with the minimal service delivery requirements? A qualitative question might ask, How do the services that are delivered compare with those specified in the procedures manual or by the accrediting agency?

Correlative research questions are used to identify relationships to enable the explanation of phenomena. Data derived in response to such questions indicate the strength and direction of a relationship between two or more variables, not causality. For example, the special education researcher might ask, What is the relationship between the size of family and the presence of emotional disturbance in siblings? If a strong, positive relationship is found, this would not lead to the conclusion that large families cause emotional disturbance in siblings. Such a relational finding would suggest the need for further study to uncover the causal relationships. A qualitative research question might ask, What is the nature of the relationship between psychologists who do not know how to use sign language and their Deaf clients? Reason and Bradbury (2006) suggest the following qualitative questions for action researchers:

1. What is the quality of interactions that has been developed in the inquiry and the political forms that have been developed to sustain the inquiry?
2. How have the values of democracy been actualized in practice?
3. What is the relationship between initiators and participants?
4. What are the implications for infrastructure and political structures?

Impact research questions are used when the researcher’s aim is to identify effects, to establish causal links between an independent variable (the intervention) and a dependent variable (the anticipated change). The researcher might investigate two types of effects: simple and relative. Research on the impact of an intervention (literacy intervention) on a behavior (reading) is one example of an impact study. The research question might ask, Do students who participate in the literacy intervention perform better on end-of-year assessments in reading than students who do not participate? If the researchers choose (and this is good practice) to explore the impact of the intervention on other related outcomes (e.g., math, self-confidence), additional questions could address relative effects.

Impact questions can then be reformulated and stated as hypotheses. A hypothesis is an “if …, then … ” statement. For example, a hypothesis might state this: “If students are exposed to a particular intervention, they will behave in a certain, predictable manner.” A sample hypothesis for the literacy study cited above might read this way: “If students participate in the literacy intervention, then their scores on the end-of-year reading assessments will be higher than the scores for students who do not participate.” This is known as a directional hypothesis because it is stated in the direction of the expected outcome. A researcher could choose to state a null hypothesis—that is, a statement that did not specify the expected direction of the outcome. The previous hypothesis could be restated as a null hypothesis: “There will be no difference in end-of-year reading assessments for students who do participate in the literacy intervention compared to those who do not.”

Qualitative researchers can also explore impact questions in several forms. For example, they might ask, What is the nature of change in individuals who participated in particular programs? Or, What aspects of the program contribute to the changes that were observed when participating in a program? Reason and Bradbury (2006) suggest the following types of questions for action researchers:

A. Questions about emergence and enduring consequence: What are the consequences associated with the current course of action in the community? What is the potential or realization of a sustainable change in the community?

B. Questions about outcomes and practice: What are the outcomes of the research? Does the intervention work? What are the processes of inquiry? Are the interventions authentic/life enhancing? What dimensions of an extended epistemology are emphasized in the inquiry and are these appropriate?

C. Questions about plural ways of knowing: What are the validity claims for different forms of knowing and the relationship between different ways of knowing?

D. Questions about significance: What is worthwhile? What values have been actualized in the inquiry? To what extent have we created an inquiry process that is truly worthy of human aspirations?

Transformative researchers often use a critical theoretical lens to stimulate the development of questions that interrogate injustice and seek equitable solutions. Examples include the following:

What is the impact of the zero-tolerance policies on Black males’ educational experiences and outcomes through the lens of critical race theory (CRT) using the counter-narrative storytelling tenet (Caton, 2012, p. 1057)? Counter-storytelling is a way of listening to previously silenced voices to get a better understanding of the world as it is experienced by those who experience discrimination and oppression; it is discussed further in Chapters 9 and 13.

Boddy (2009) used a strengths-based feminist framework to generate these research questions: How do women in poor economic communities structure their experiences around work at home and work for pay? What are their strengths that they bring to the opportunity to improve their living conditions?

Using a transformative lens and resilience theory, Fourie and Theron (2012) used the following research question in a case study of a young woman with a genetic condition associated with mental impairment and physical, emotional, and behavioural challenges: What contributes to positive adjustment in this woman with this syndrome? (p. 1355).

As a wrap-up to this section on research questions and hypotheses, I present you with an example of these elements of a research study about disproportional representations of racial/ethnic minorities in special education (meaning that racial/ethnic minorities are identified for special education in higher proportion than their representation in the general population). The research questions developed by A. Sullivan and Artiles (2011) were aligned with specific hypotheses, as you can see in Table 3.2. Their first question asks about the extent of disproportionality for the different disability categories and for the levels of educational agency (local and state). The second research question asks about the variables that might predict disproportional identification. To find out the results of their study, you can read about it in Chapter 5.

Table 3.2 Example of Research Questions and Hypotheses

Research Questions

To what extent are racial minority students disproportionately represented in special education across levels (state versus local districts) and disability categories? (p. 1532)

To what extent is disproportionality for the different race-disability groupings (e.g., Latino students identified as MRb) predicted by the structural factors of LEAs? (p. 1533)c

Hypotheses

Variations in disproportionality would be observed across the two levels of analysis (i.e., state and LEAa), with evidence of disproportionality primarily in the more subjective categories because they are the most susceptible to the influence of racial ideologies and norms regarding behaviour and ability (p. 1533).

The proportion of students receiving free/reduced-price lunch would be negatively related to disproportionality. (p. 1534)

The proportion of racial minority teachers would have negative relationships with disproportionality. (p. 1534)

SOURCE: A. Sullivan and Artiles (2011).

a. Local Education Agency.

b. Mentally retarded.

c. Structural factors include proportion of students receiving free/reduced-price lunch, district size, student-teacher ratio, proportion of racial minority students, and the proportion of racial minority teachers.

Summary: Purposes of Literature Reviews

In summary, the literature review serves many purposes. It establishes a historical perspective on the intended research, provides a vision of the need for additional research, and enables the researcher to develop a conceptual framework for the research. This framework allows the researcher to generate research questions and hypotheses to guide the design and conduct of the research. In qualitative research, typically, the researcher will refine, modify, add, and even discard questions throughout the progress of the study (Marshall & Rossman, 2011). Therefore, qualitative researchers are advised to begin with broader questions that can be modified in response to discoveries made during the study. No matter which research paradigm or approach is used, the literature review is an essential ingredient in the research process.

Critical Analysis of Literature Reviews

The criteria for critically analyzing literature reviews depends (again) on the nature of the review being analyzed. A literature review that serves as an introduction to a primary research study reported in a journal would be subject to a different type of scrutiny than would a comprehensive literature review on a topic. Nevertheless, Boote and Beile (2005) provide a framework to assess the quality of a literature review. (Their rubric of these categories is included as Table 3.3.) Boote and Beile included five categories for their framework

Coverage refers to the adequacy of the coverage of the topic as well as making explicit criteria for exclusion and inclusion of studies for the review. Does the reviewer include relevant works and exclude irrelevant ones? Writing a dissertation does not mean citing every study ever written on your topic. Coverage should be judged in terms of comprehensiveness, breadth, exclusion, relevance, currency, availability, and authority. Researchers can bias the results of a literature review by excluding data that is methodologically questionable, based on their own personal, subjective judgment. Or they may present conclusions that are firmer and more clear-cut than are justified because of the exclusion of studies with “murky” results. Without a clear specification of the method used to search for research and of the criteria used for inclusion or exclusion, it is difficult to judge the quality of a review.

Synthesis is the second category, and it refers to how well the author summarized, analyzed, and synthesized the selected literature on a topic. The criteria include how well the author

distinguished what has been done in the field from what needs to be done,

placed the topic or problem in the broader scholarly literature,

placed the research in the historical context of the field,

acquired and enhanced the subject vocabulary,

articulated important variables and phenomena relevant to the topic, and

synthesized and gained a new perspective on the literature. (Boote & Beile, 2005, p.

Table 3.3 Literature Review Scoring Rubric

Category

Criterion

1

2

3

1. Coverage

a. Justified criteria for inclusion and exclusion from review

b. Did not discuss the criteria inclusion or exclusion

c. Discussed the literature included and excluded

d. Justified inclusion and exclusion of literature

2. Synthesis

b. Distinguished what has been done in the field from what needs to be done

c. Placed the topic or problem in the broader scholarly literature

d. Placed the research in the historical context of the field

e. Acquired and enhanced the subject vocabulary

f. Articulated important variables and phenomena relevant to the topic

g. Synthesized and gained a new perspective on the literature

Did not distinguish what has and has not been done

Topic not placed in broader scholarly literature

History of topic not discussed

Key vocabulary not discussed

Key variables and phenomena not discussed

Accepted literature at face value

Discussed what has and has not been done

Some discussion of broader scholarly literature

Some mention of history of topic Key vocabulary defined

Reviewed relationships among key variables and phenomena

Some critique of literature

Critically examined the state of the field

Topic clearly situated in broader scholarly literature

Critically examined history of topic

Discussed and resolved ambiguities in definitions

Noted ambiguities in literature and proposed new relationships

Offered new perspective

3. Methodology

h. Identified the main methodologies and research techniques that have been used in the field and their advantages and disadvantages

i. Related ideas and theories in the field to research methodologies

Research methods not discussed

Research methods not discussed

Some discussion of research methods used to produce claims

Some discussion of appropriateness of research methods to warrant claims

Critiqued research methods

Critiqued appropriateness of research methods to warrant claims

Introduced new methods to address problems with predominant methods

4. Significance

j. Rationalized the practical significance of the research problem

k. Rationalized the scholarly significance of the research problem

Practical significance of research not discussed

Scholarly significance of research not discussed

Practical significance discussed

Scholarly significance discussed

Critiqued practical significance of research

Critiqued scholarly significance of research

5. Rhetoric

l. Was written with a coherent, clear structure that supported the review

Poorly conceptualized, haphazard

Some coherent structure

Well developed, coherent

SOURCE: Boote & Beile, 2005.

NOTE: The column-head numbers represent scores for rating dissertation literature reviews on a 3-point scale.

To satisfy these criteria, the writer needs to identify tensions and inconsistencies in the literature, provide clarity discussing the strengths and weaknesses of the individual studies as factors that influence the interpretation of their results, and use the extant knowledge base to suggest directions and topics for additional empirical investigations.

Methodology as a criterion for judging a literature review refers to the author’s accurate inclusion of details about method that have relevance for identification of methodologies and research techniques—and their strengths and weaknesses—and discussion of the relationship between theories and ideas in the field to the research methodologies (Boote & Beile, 2005).

Literature reviews should not be simple summaries of findings of previous research; they should be critical analyses of previous research. In order to critically analyze the strengths and weaknesses of prior research, several skills are necessary. One is the ability to accurately identify the methodologies; a second is the ability to identify strengths and weaknesses in the methodologies and how they impact the interpretation of results. Unless you have had prior experience critiquing research, you are probably wondering how you can do this type of critical analysis. You can continue through this book, and by the end you’ll be able to critically analyze the major approaches to research in education and psychology. At the present moment, we will focus on your being able to critically analyze the literature review section of a research study, and then you can add to your skill set as you progress through subsequent chapters. What you can do for the moment is know that when you take notes about your studies, you want to include information about the methodology, not just the results.

Significance is the fourth category and includes establishing both the practical and the scholarly significance of the research problem (Boote & Beile, 2005). While some research studies will focus more or less on one of these aspects, it is useful to provide implications for both the practical and scholarly significance of research.

Rhetoric is the final category, and it refers to the writers’ ability to organize and write cogently about the literature in such a way that they can articulate and support their claims about the knowledge in the field (Boote & Beile, 2005).

Extending Your Thinking Choices in Literature Reviews

How much and what kind of information should you include about each study in your literature review? About your literature review search method?

What are your rules for inclusion and exclusion of studies in your literature review?

Several caveats are in order at this point:

The points made in this chapter are relevant to almost all types of educational and psychological research, whether it is a single empirical study, cyclical study, master’s thesis, or doctoral dissertation.

The act of reviewing the literature does not just occur in the beginning of a research study and is not completed once the introduction to the article or the proposal for the research is completed. Literature review should be an ongoing process, and the results of that review should be integrated into the body of the report at appropriate points—but especially in the discussion and conclusions sections.

The researcher should be aware of potential biases in literature reviews. There is a greater tendency for research with statistically significant results (i.e., those showing group differences larger than chance) to be published. Research studies that show no differences either are not submitted by the authors or are rejected more frequently by journal editors (P. B. Campbell, 1989). Campbell suggested that this publication bias leads to an exaggerated concept of differences. Authors should make clear whether or not they tracked down unpublished studies on the topic to correct for this bias. Keep in mind that the quality of the unpublished data may be suspect because they have not been through a review process.

Researchers should include a discussion of threats to inference from research syntheses based on the quality (or lack thereof) of the primary research studies included in the review. In assessing the conclusions reached in any literature review, the reader should be cognizant of the quality of the studies included.

Questions for Critically Analyzing Literature Reviews

The following questions can be used to determine if a literature review is satisfactory. In preparing your answers to these questions, cite evidence in the article to support your answers.

1. The purpose of the literature review is to place the current research into the “big picture” of what is known and not known about a specific topic. What is the big picture into which this study fits? What is the central topic? How is the researcher conceptualizing the problem?
2. What is the nature of the literature cited?
3. Is the review current, using research that is recent enough to have applicability to the proposed research? Some researchers recommend inclusion of studies within the last ten years, unless a study is being included for historical purposes. This timeframe might be even shorter for areas in which rapid developments are occurring, such as educational law and theory and technology-related research.
4. Is the review based predominately on primary research rather than on secondary or opinion pieces?
5. Does the review provide a critical analysis of existing literature, recognizing the strengths and weaknesses of previous research? Or is the review just a summary of prior research?
6. Is the literature review well balanced, presenting evidence on both (or all) sides of the issue?
7. Is the review free from the biases of the reviewer? Is there any evidence in terms of emotional language, institutional affiliation, funding source, and so on to suggest that the reviewer might be biased?
8. To what extent does the review establish a need for the study? What is the author’s rationale for why this study is needed? What do we know? What do we need to know? Why is this study important (practically and in terms of scholarship)?
9. What is the theoretical framework and what are the research questions? Does the review provide enough information to support the researcher’s theoretical framework and research questions posed?
10. Does the review provide sufficient information to guide the research procedures, including the identification of subject participants, selection of data collection and analysis processes, and use of appropriate reporting strategies? After you read the review and you see what research questions and methods are used, do you think they are logically connected? Does what the researchers do in terms of method make sense in terms of what is presented in the literature review?
11. Are sources cited inclusive of “marginalized” voices? Are citations made that reference viewpoints of those with the least power?

To really have a basis for critically analyzing research, it is helpful to have broad experience with different types of research as well as with a number of studies that represent the same research approach. Of course, such breadth and depth take time to achieve. Nevertheless, a long journey begins with a single step. Throughout this text, you will be encouraged to identify full-text research articles that relate to your area of interest and to critically analyze those studies. The ability to critically analyze research is also a skill that becomes more holistic with experience.

Extending Your Thinking Critically Analyzing Literature Reviews

Locate several empirical research studies. Identify the following features of the studies: (a) the paradigm that the researchers used, (b) the research problem, (c) the theoretical framework that underlies the study, and (d) the research questions or hypotheses.

Using the questions at the end of Chapter 3 for critically analyzing literature reviews, critique literature reviews in several different literature studies, identifying their strengths and weaknesses and supporting your claims with evidence from the articles.

When you are in the beginning stages of learning critical analysis, it is helpful to look at each section of the research study. So in this chapter, we focus on the introductory section that includes the literature review and research problem, hypothesis, questions, or objectives. Later, you will be able to look at other aspects of the article, such as how the author handled certain aspects of data collection, analysis, credibility building, or ethics. You can then do comparisons across studies on these dimensions, analyzing how and why texts differ, how they relate to theoretical readings, whether the authors are justified in their methods or presentations, and how they can help you in your own decisions about research. With each research article that you review, you will increase your ability to determine the quality of the author’s work and the validity of the findings.2

Summary of Chapter 3: Literature Review and Focusing the Research

A review of scholarly literature provides information that can be used to investigate a topic of importance to learn what is known about that topic for its own sake (i.e., to improve teaching or therapy) or as a basis for designing a research study. The formulation of a research topic is enabled by reading about research that has already been conducted because the reader can figure out what is already known as well as become acquainted with the strengths and weaknesses of methods used in prior research. Multiple sources exist for the conduct of literature reviews, including secondary sources that provide an overview of past research and primary sources that report original research. Primary sources can be identified through several different electronic means that are described in this chapter. Persons conducting literature reviews can summarize their results in narrative form or a quantitative form known as meta-analysis. A literature review can be used to develop research questions of different types, such as descriptive, correlational, or impact. Researchers can also benefit by looking outside published scholarly research to community members to provide a different perspective on what needs to be studied and how it should be studied. You are now ready to consider which specific research approach is appropriate to answer the research questions.