A good literature review written by someone else can provide you with an overview of what is known about your chosen topic. Specific places that you can look for literature reviews include journals that typically publish literature reviews, such as the Review of Educational Research, Harvard Educational Review, and the Psychological Bulletin, and books that contain literature reviews, such as the following:
Review of Research in Education (RRE): This series is published annually by the American Educational Research Association. Each volume contains a series of chapters on various topics—for example, Volume 40 (2016) provides a historical perspective on educational research and uses that as a foundation to identify the most challenging issues for educational researchers. Additional volumes address equity of school funding, literacy development for ELLs, and use of digital tools with young children.
The Annual Review of Psychology contains literature reviews on topics of interest in psychology and education, such as counselling or learning theory.
Research in Race and Ethnic Relations is published annually to address race relations and minority and ethnic group research.
Other handbooks have been published on specific topics:
Extending Your Thinking Selecting Your Research Topic and Setting
Students of research are sometimes given conflicting advice about the topic and site for their own research. The following pieces of advice exemplify such conflicts. Where do you stand on these two issues (i.e., choice of a research topic and setting) and why?
Select a research topic based on your personal experience and something that you feel passionately about. You will work on this topic for a long time, so it is important that you feel strong motivation to continue working on it.
Don’t pick a topic about which you have strong personal feelings because you may not be able to manage your own biases. The people you interview may be influenced by your personal experiences and biases.
A trend in research is now for members of marginalized communities to demand that the research be conducted by themselves, with themselves, and for themselves (Mertens, 2009). This is exemplified in Chilisa’s (2012) statement that research needs to start “with a call for recognition of the conceptual frameworks, theoretical frameworks, and data collection and analysis methods derived from the researched’s frames of reference and Indigenous knowledge” (p. 171).
Butcher, J. N. (Ed.). (2018). APA handbook of psychopathology. Washington, DC: Author.
Collins, P. H., & Solomos, J. (Eds.). (2010). The SAGE handbook of race and ethnic studies. Thousand Oaks, CA: Sage.
Florian, L. (Ed.). (2014). The SAGE handbook of special education (2nd ed.). Thousand Oaks, CA: Sage.
Fouad, N. A. (Ed.). (2012). APA handbook of counseling psychology. Washington, DC: Author.
Kitayama, S., & Cohen, D. (Eds.). (2007). Handbook of cultural psychology. New York, NY: Guilford.
Miller, L., Cameron, C., Dalli, C., & Barbour, N. (2018). The Sage handbook of early childhood policy. Thousand Oaks, CA: Sage.
Travis, C. B., & White, J. W. (Eds.). (2017). APA handbook of the psychology of women. Washington, DC: Author.
Step 3: Develop a Search Strategy
Four paths for search strategies are described in this section: (a) identify preliminary sources, (b) identify primary research journals, (c) access personal networks, and (d) involve community members. These are explained below. Decide which strategies are the best for you to follow in your search process, and remember, stay flexible.
Box 3.2
Most Frequently Used Preliminary Sources
ERIC: The Educational Resources Information Center (ERIC) is sponsored by the U.S. Department of Education and contains abstracts of journal articles and grey literature, such as research reports, conference presentations, and government reports. The database can be searched by going to www.eric.ed.gov or by using ERIC in commercial databases provided in many libraries. Most ERIC documents are available electronically, in print, or on microfiche in libraries. Many nonjournal materials are available, at no charge, as PDF documents or via links to publisher websites. Check with your local library (academic, public, etc.) to find out if they can provide journal articles or documents that are not available online. If the library cannot do this for you, print copies of journal articles can be purchased through such article reprint services as Ingenta (www.ingentaconnect.com).
ProQuest® has a number of databases, including Career and Technical Education, Education Journals, Psychology Journals, Social Science Journals, ComDisDom (related to hearing, speech, and language), and Dissertations and Theses. The database includes thousands of journals, many with full texts of the articles. ProQuest is available at many libraries.
JSTOR: This is a database of academic journals, monographs, and other academic papers from multiple disciplines, including the social sciences, humanities, and the sciences. It is available from libraries that subscribe to the service. Individuals in the United States can subscribe for a modest amount; JSTOR made special arrangements for individuals in Africa to have access to this database for free (www.jstor.org).
PsycINFO: This is a product of the American Psychological Association (APA) that contains indexes and abstracts from journals, books, and book chapters related to psychology, education, and linguistics. Both members and nonmembers can search the database and purchase articles. As with other databases, you can check on its availability at your local library.
PsycARTICLES: This is another product of the APA, but it includes the full-text articles of many journals related to psychology that APA or affiliated organizations publish. The database can be searched by APA members and by nonmembers (for a small fee).
Google Scholar is a free search engine that indexes full-text scholarly literature from a wide variety of disciplines. One nice feature of Google scholar citations is that they include a link to articles that have cited the article that you are reading. Many university libraries also provide access to searchable databases that cover a wide range of journals and disciplines such as Academic Search Premier and Academic Search Complete.
World Wide Web (WWW) sites are easily and pervasively available to assist you in your literature searching. There are many search sites on the Web, and new ones appear with some regularity. In March 2018, the top three choices for search engines were Google, Bing, and Yahoo. These three search engines were recognized because of the size of their databases, ability to search HTML and PDF files, accuracy of results, and advanced searching power using Boolean logic. The search process on websites generally employs Boolean logic (explained later in this chapter) but can differ a bit from site to site. Because this is such a dynamic area, it is best to check PC Magazine or some other computer source to find out what sites are recommended for appropriate searching on your topic and to determine appropriate search strategies for those sites. One word of caution: The WWW sites do not have a peer review system to screen what is accepted (as most professional journals do); therefore, scholars raise questions about the quality of information available from those sources. In addition, the websites are not designed to contain information specifically about research in education and psychology as are the other databases described in this chapter. Many .com websites are intended to sell particular products; however, .edu, .gov, and .org websites can contain valuable content for a literature review. Students should be aware of the need to be skeptical of websites that promote a particular position and seek out sources that provide alternative points of view.
The computerized databases are a tremendous resource for the researcher in the literature review phase of a project. A researcher can identify thousands of references by only a few keystrokes on the computer. Because of the tremendous coverage provided by the databases, the researcher should plan to include a search of appropriate databases in the literature review process. Box 3.3 provides examples of how research teams described their methods of searching the literature. Notice that they did use the academic databases, but they also used many other sources to identify relevant studies.
Identify Preliminary Sources
Preliminary sources include databases and indexes that contain a compilation of bibliographic information, abstracts, and sometimes full-text articles for a wide range of topics and are accessible in print form, on compact disks (CD-ROM), or through online services.1 Examples of the most frequently used preliminary sources are listed in Box 3.2. Additional abstract and index services that specifically target marginalized groups include African Urban and Regional Science Index, Women’s Studies Abstracts, and Women’s Studies Index.
Description of Literature Review Search Method
We began by examining prior systematic reviews, including those cited in the literature review as well as reviews of interventions, targeting reductions in related problem behaviors, such as delinquency, drug abuse, and mental health problems. We also reviewed program evaluations identified on web-based and published lists of best practices related to violence prevention, including the Blueprints for Violence Prevention database (Center for the Study and Prevention of Violence, 2010), the Campbell Collaboration (C2; http://www.campbellcollaboration.org/), the Communities That Care Prevention Strategies Guide (http://www.sdrg.org/ctcresource/), the Model Programs Guide (Office of Juvenile Justice and Delinquency Prevention, 2010), the Office of Justice Programs (CrimeSolutions.gov), and the National Registry of Evidence-based Programs and Practices (Center for Substance Abuse Prevention, 2010).
A final step involved searching the Medline, PsycINFO, Social Sciences Citation Index, and Web of Sciences databases with key words including aggression, crime, delinquency, prevention, youth, and violence. The reference lists of all studies reviewed were used to identify additional evaluations until a saturation point was reached.
—Fagan and Catalano (2013, p. 144)
We examined the research published from the end of March 2006 to the end of December 2011…. First, a PsycINFO search was conducted of journal articles published in English using the terms “treatment effectiveness evaluation,” “treatment effectiveness,” “effectiveness trials,” and “psychotherapy.” In addition, we used the general terms “parent training,” as well as naming three specific treatments that have been frequently evaluated in efficacy trials of treatments for child and adolescent disorders, “Parent-Child Interaction Therapy,” “Incredible Years,” and “Triple P.” Second, we searched the tables of contents of 31 journals that frequently publish treatment research…. In addition, as our third search strategy, we reviewed the reference lists of articles that were found to meet our inclusion criteria to obtain additional effectiveness studies that we had not identified with our initial two search strategies. As our fourth and final strategy, we also searched the references of recent reviews of child and adolescent treatment … for effectiveness studies.
—Lee et al. (2013, p. 82)
One important limitation should be noted about the available databases. You can get out of them only what was put into them. In other words, the databases are selective about the journals they include. For example, some feminist journals (listed in Box 3.4) are not listed in ERIC; they might not be considered appropriate for inclusion in ERIC because their content is not directly related to education or psychology. However, readers who are not familiar with feminist journals might find the list helpful as a way of broadening their resource base. For example, Hypatia publishes mainly philosophical work and thus would be of interest to those who want to delve more deeply into that aspect of feminist theory. Camera Obscura publishes work that could be of interest to scholars in educational media or social learning theory (e.g., the study of the power of media to shape cultural expectations by gender).
Box 3.4Feminist Journals
Camera Obscura: A journal of feminist perspectives in film, television, and visual media.
Feminist Studies: The first feminist academic journal (started in 1972) is based at the University of Maryland. It publishes an interdisciplinary body of feminist knowledge and theory that makes visible assumptions about gender and sexual identity and the experiences and contributions of women across a wide range of difference.
Gender & Society: This journal emphasizes theory and research from micro- and macrostructural perspectives. It aims to advance both the study of gender and feminist scholarship.
Genders: Based at the University of Colorado, Genders publishes innovative work about gender and sexuality in relation to social, political, artistic, and economic concerns.
Hypatia: This journal publishes scholarly research at the intersection of philosophy and women’s studies.
Journal of Women’s History: An international journal that covers new research on women’s history, it includes scholarship about women in all time periods that is broadly representative of national, racial, ethnic, religious, and sexual grouping.
Meridians: A feminist, interdisciplinary journal with the goal of providing a forum for scholarship and creative work by and about women of color in U.S. and international contexts.
Signs: Journal of Women in Culture and Society: An international journal in women’s studies that publishes articles from a wide range of disciplines in a variety of voices—articles engaging gender, race, culture, class, sexuality, and/or nation.
Women’s Studies International Forum: The goal of this journal is to aid the distribution and exchange of feminist research in the multidisciplinary, international area of women’s studies and in feminist research in other disciplines.
Women’s Studies Quarterly: This journal focuses on teaching in women’s studies. Thematic issues include such features as course syllabi, discussions of strategies for teaching, and bibliographies.
Identify Primary Research Journals
Additional primary research articles can be identified by examining the reference lists found at the end of relevant journal articles or books. You can also go directly to journals that you know publish articles related to your topic. This is especially important in light of the selectivity of the databases discussed in the previous section. Researchers who are working from a transformative paradigm should be aware of the journals that deal with issues specific to marginalized groups, such as those in Box 3.4 for feminists, as well as journals such as Latin American Perspectives, Hispanic Journal of Behavioral Sciences, Journal of Multicultural Counseling and Development, Journal of Negro Education, and Journal of Black Studies. Disability, Handicap & Society is a journal that frequently focuses on the transformative paradigm in research with people with disabilities, and The Counseling Psychologist (2008) devoted an entire issue to the topic of multicultural counseling for psychologists and educators (Vol. 36, No. 2). A more extensive list of special education journals can be found in Box 3.5.
Box 3.5 Selected Journals Containing Special Education Resource Information
American Annals of the Deaf
American Journal of Intellectual & Developmental Disabilities
Annals of Dyslexia
Australasian Journal of Special Education
British Journal of Special Education
Career Development for Exceptional Individuals
Disability and Society
Exceptional Children
International Journal of Disability, Development and Education
Journal of Autism and Developmental Disorders
Journal of Deaf Studies and Deaf Education
Journal of Learning Disabilities
Journal of Special Education
Journal of Speech, Language, and Hearing Research
Learning Disability Quarterly
Remedial and Special Education
Research in Developmental Disabilities
Volta Review (deafness)
Personal Networking
Additional resources can be found by talking to people who are doing work in areas related to your interest. This can include people at your own institution or those you meet through professional associations, such as the American Educational Research Association, the American Evaluation Association, the American Psychological Association, the Council for Exceptional Children, or the National Association of the Deaf. Talking to people who have completed related work can reveal sources that you were unaware of, such as unpublished research reports, and provide you with leads from work that is in progress for that researcher.
Two examples of well-planned and documented searches are provided in Boxes 3.6 and 3.7. As a researcher, it is always a good idea to carefully document your search strategy. In this way, you can backtrack to helpful strategies if you need additional information and indicate to the reader how thorough you were in your search process.
Involvement of Community Members
The combination of self-knowledge with cultural knowledge and skills in effective partnering facilitates the development of the research or evaluation focus and identification of questions, development of interventions, and making decisions about design, measures, samples, data collection, analysis, interpretation, and use that are in keeping with the philosophical assumptions of the transformative paradigm (Mertens, 2009). Following proper channels to enter a community is important, and strategies for doing this will vary by context. Some Native American Indian communities have developed specific protocols for anyone who wants to conduct research in their communities (LaFrance & Crazy Bull, 2009); Maori people from New Zealand have also developed similar protocols for research in their community (Cram, 2009). Deaf researchers have adapted the Indigenous Terms of Reference to suggest a protocol for research in that community (Harris et al., 2009). These protocols will be discussed in more depth in the chapter on sampling. However, it is important for researchers to know how to enter a community with respect, to communicate their intentions to members of the community in the appropriate way, and to make clear what benefits will accrue to themselves and to the community. Schneider et al.’s (2004) method of involving people with mental illness is one example of how community members can be involved in the decision process about what topics to study in the research.
Box 3.6 Method Used in Literature Review: Example 1 Social Skills and Autism
The primary purpose of the literature review was to identify research that addressed the effectiveness of group-based social skill interventions for adolescents with higher functioning autism spectrum disorder (ASD).
Method
The researchers did not want to do a comprehensive review of this topic. They chose to do a selective review that used only the PsycINFO database. They also added other studies that were not found in PsychINFO that they knew about because of their expertise in the field. Several search terms were used, including these: Autism or autistic or pervasive developmental or Asperger, and Intervention or training or treatment, and Social.
A decision was made to limit the selection of publications to peer-reviewed journals written in English and published after 1990. The participants in the studies had to be school-age adolescents or young-adult age groups (ages 10–20).
The researchers used specific inclusion/exclusion criteria:
The study specifically focused on individuals with ASD.
The study assessed the efficacy of an intervention (i.e., reviews, meta-analyses, longitudinal follow-ups, theoretical papers, etc., were not included).
The intervention focused on social skill improvement (i.e., interventions focused on challenging behaviors, comorbid symptoms, communication, etc., were not included).
The intervention was delivered in a group-based format, and groups were primarily composed of other individuals with ASD.
The intervention included at least some participants within the 10- to 20-year-old age range.
The intervention data were not primarily analyzed using a single-subject design (p. 24).
SOURCE: McMahon, Lerner, and Britton (2013).
Step 4: Conduct the Search
In conducting the search, you should make a plan to search preliminary sources; check the table of contents, abstracts, and lists of references in primary research journals; access your personal network; and involve community members as appropriate. The remainder of this section focuses on the search strategy as it applies to accessing preliminary sources.
Box 3.7Method Used in Literature Review : Example 2 African American males and achievement in school
A literature review of interventions related to improving achievement of African American males was conducted by using the following electronic databases:
EBSCO HOST (Academic Search Premier SocINDEX with full text), JSTOR, Google Scholar, Wilson Web Social Sciences Full Text, and ERIC.
He used the following search terms: African American males or boys, Black males or boys.
He used this review checklist to select studies:
was conducted in the United States;
used quantitative, qualitative, or mixed methods study designs;
was conducted within the last 12 years;
reported disaggregated data on or about successful or high-achieving African American males, or effective pedagogies with them;
was published in a peer-reviewed journal, scholarly book; and
specially addressed African American/Black males (not males of color) (p. 58)
He eliminated studies with a deficit perspective that focused only on problems with Black males. He then selected those studies that focused on literacy, mathematics, discipline/and/or punishment, and teacher education or teacher practice.
SOURCE: Howard (2013).
Prepare to Search Preliminary Sources
Select the preliminary sources that you think contain the best information on your topic (see Box 3.2). Then identify key terms that will help you locate the literature included in the database of choice. One way that researchers select key terms is to find one primary research article that is “exactly” on target and identify the terms used to describe that article.
A search strategy based on using the ERIC online system is used to illustrate this process. The search strategy is similar when using other databases and indexes, such as PsycARTICLES and PsycINFO. Most databases give you many choices for searching, such as title, author, abstract, subject, or full text. The title, author, and abstract choices are fairly self-explanatory. Author and title are not usually used in the beginning of a literature review because you usually are not seeking a specific article during the early stages of searching. The subject choice needs a bit of explanation. Subject words are those that were used by the people who work for the database to categorize that item. These words are contained in a thesaurus, usually available in the online system. Each item in the database has a field associated with it that contains subject words that an indexer selected, and that is the field that is searched when you choose a subject word strategy. Full-text searchers, on the other hand, allow researchers to choose words that reflect their own vocabulary in the description of the topic. Full Text means the computer searches for these terms using a “free text” strategy; that is, it searches anywhere in the document for the words that you enter. Advantages and disadvantages accrue to whichever search strategy is chosen.
The easiest way to start is to use a key word strategy to determine if the words that you think are appropriate produce references that match your conceptualization of the problem. For example, for the topic of sexual abuse of Deaf students, I started in ERIC using sex abuse deaf as key words. The computer said there were no articles available that combined those three terms. I took a few minutes to read the directions in ERIC and found that I could use a multifield search strategy, separating the terms. So I used sex? AND abuse AND deaf?. (There is a good reason, explained later, for the inclusion of the ? and the word and in this search specification.) This resulted in 19 entries. One of the entries was Black and Glickman’s (2006) “Demographics, Psychiatric Diagnoses, and Other Characteristics of North American Deaf and Hard-of-Hearing Inpatients” (see Box 3.8).
If you have difficulty finding references using your own key word vocabulary, check a thesaurus of terms to determine how the indexers might have conceptualized your topic. Use of subject descriptors can be helpful in narrowing down a search, as long as the descriptors are defined in a way that is compatible with your topic. They can also be helpful in broadening a search by suggesting other terms that could prove fruitful in searching.
Now, why include a? in the search terms, and what is the importance of the and in the list? You can refine your search in the following ways:
Truncate the terms you use. This has the effect of broadening the search to include any terms that begin with the letters that you enter, no matter how they end. In ERIC (the Educational Resources Information Center), the truncating symbol is a ? Therefore, entering sex? would include sex, sexual, sexes, and so on, and deaf? would include deaf, deafness, deafened, and so on.
Use Boolean or positional operators to combine terms. Boolean logic allows you to use the words and, or, not, and nor (one but not both words are in a record). Thus, asking for sex? and abuse and deaf? yields references in which the three terms appear in the same record. The or operator yields references that have either or both words in the same record. So, I could have asked for sex? abuse or child abuse and deaf? This would have given me all the records that contain sex? abuse or child abuse and deafness. In addition, I could have broadened my search by including deaf? or hearing-imp? This would have resulted in all references that had either deaf, hearing-impaired, or hearing impairment in their records.
Positional operators include same, with, adj, and near, and they limit retrieval by specifying how near key words must be to each other. Same means that both words must be in the same field of the same record (e.g., title or abstract); with means both words are in the same section of the same field of the same record; adj requires that the words must be next to one another (adjacent) in the order specified; and near finds references in which words are next to one another in any order (e.g., sex abuse or abuse sex).
There are other ways to limit the search, such as by year of publication or limiting the field that is searched (e.g., title only). Certain stop words are not allowed to be used as key words (e.g., about, all, its), but all of these things can be learned by reading the online instructions. As you get into using a database, it is always a good idea to read the online instructions to see what can be accomplished and how.
Obviously, the search process can be broadened by inclusion of additional databases or indexes. For example, when I searched PsycARTICLES using the same descriptors (i.e., sex? and abuse and deaf?), I identified 57 additional references that did not overlap with those found in ERIC.
Box 3.8Journal Citation Entry From ERIC
Black, Patricia A. Glickman, Neil S. Demographics, Psychiatric Diagnoses, and Other Characteristics of North American Deaf and Hard-of-Hearing Inpatients [Journal Articles. Reports—Research] Journal of Deaf Studies and Deaf Education. v11 n3 p303–321 2006 AN: EJ738331. By permission of Oxford University Press.
Abstract: This study examined demographic and clinical data from a specialty deaf inpatient unit so as to better understand characteristics of severely and chronically mentally ill deaf people. The study compares deaf and hearing psychiatric inpatients on demographic variables, psychiatric discharge diagnoses, a language assessment measure, a cognitive ability measure, and a measure of psychosocial functioning and risk of harm to self and others. Overall, findings indicate a broader range of diagnoses than in past studies with posttraumatic stress disorder being the most common diagnosis. Compared with hearing patients in the same hospital, deaf patients were less likely to be diagnosed with a psychotic or substance abuse disorder and more likely to be diagnosed with a mood, anxiety, personality, or developmental disorder. Psychosocial functioning of the deaf patients was generally similar to hearing psychiatric patients. Deaf patients presented significantly higher risks than hearing patients in areas of self-harm and risk of sexual offending. Cognitive scores show that both the deaf and hearing inpatient population is skewed toward persons who are lower functioning. An additional surprising finding was that 75% of deaf individuals fell into the nonfluent range of communication in American Sign Language. (Author) A final word of advice: Cultivate a good relationship with your librarian. I invite the research librarian to my research classes to make a presentation on databases, search strategies, and documentation of findings from the search. Students report that visiting the research librarian is extremely helpful.
Interpret What You See
You can locate at least two types of documents in ERIC: journal articles and other references that are available through ERIC. Journals may be obvious because they have the name of the journal, but if you are not sure, look for an EJ code in the ERIC citation. Other references are noted with the abbreviation ED (education document) and are typically presentations made at professional meetings, curriculum guides, research reports, or other similar materials. An example of an ERIC full-text abstract for a journal article was presented in Box 3.8 and one for an ERIC document is in Box 3.9.
Box 3.9ED Document Citation From ERIC
Accession Number
ED477969
Author
Karcher, Michael J.
Title
The Hemingway: Measure of Adolescent Connectedness—Validation Studies.
Page Count
59
Peer Reviewed
No
Date of Publication
2001
ERIC Subject Headings
*Adolescent Development
Adolescents
Age Differences
Behavior Problems
Delinquency
*Interpersonal Relationship
*Psychometrics
*Social Environment
Substance Abuse
Test Validity
identifiers
*Social Connectedness.
Abstract
This investigation reports the development of a measure of adolescent connectedness and estimates of its psychometric properties. A measure was developed to assess the ecological and developmental dimensions of adolescent connectedness, defined as adolescents’ caring for and involvement in specific relationships and contexts within their social ecology. Exploratory and confirmatory factor analyses in studies one and two yielded theoretically consistent factor solutions. These models were cross-validated in studies three and four with three geographically and ethnically diverse adolescent samples totaling 1454 adolescents. The measure of adolescent connectedness demonstrated satisfactory inter-item and test-retest reliability and convergent validity across samples. Consistent with social control and problem-behavior therapy, two higher order factors emerged across all of these samples: conventional vs. unconventional connectedness. These two dimensions of connectedness were found to differentially explain substance use for delinquent and non-delinquent adolescents. Using this ecological assessment, adolescent connectedness appears to differ as a function of age, sex, and problem-behavior status; varies across relationships and contexts; reflects either conventional or unconventional behaviors and attitudes; and can explain engagement in risk-taking behaviors. (Contains 62 references and 6 tables.) (Author)
Notes: Paper presented at the Annual Conference of the American Psychological Association (109th, San Francisco, CA, August 24–28, 2001).
Level of Availability
1
Publication Type
Information Analyses. Reports—Research. Speeches/Meeting Papers.
Language
English
Entry Month
200402
Select Titles
Most databases provide an abstract of the articles listed. By scanning these abstracts, you can make a decision as to the worth of obtaining the complete article. Advances in technology now also make it possible to view many full-text articles while you are engaged in the search process. Hence, researchers are faced with a bit of a paradox concerning the amount of time it takes to do a literature review. If you only have the abstract, you read it quickly and make a determination if it is what you want. If you think it is and full text is not available, then you need to go to the library or order the article through interlibrary loan or some other mechanism. If the full text is available, you may find yourself (like me) reading many articles because they are interesting and then you wonder where the day has gone and you have not made the progress that you expected. Intellectual curiosity is good; focus is also good.
Step 5: Obtain Full-Text Resources
As mentioned previously, many journal articles and books are now available online in full-text versions. If you cannot obtain the article in this manner, then it would be good to check the list of holdings at your library. If the journal you seek is held by your library, you are in luck: Go to the shelves (or the librarian at the help desk) and read the article. However, if your library does not have the item, you may avail yourself of an interlibrary loan service. If you provide complete bibliographic information, the librarian can determine which other library has the article and make a request to have it sent to you. There is often a small charge for this service. In some libraries, obtaining a copy of the article is available by an online request as you are doing your search. The computer may ask you if you want to order the document, and then it will tell you how much it costs to obtain. You have the option of agreeing to pay the cost, and if you agree, the library that holds the reference is electronically contacted and asked to transmit the article to your library. Amazing! (The researcher’s equivalent of the Home Shopping Network.) obtain Full-Text Resources
Extending Your Thinking Primary and Secondary Sources
What is the difference between a primary source and a secondary source? When and why would you choose to use one or the other? Have you been able to locate a secondary source on your topic of interest?
What search strategies have you found to be particularly effective in locating research on your topic of interest? Have you used networking? Professional associations? Why would these be important resources? What computerized databases have you used? Do you feel comfortable using the computer to search for articles of interest for your research topic? Have you sought out journals or other sources of information (e.g., direct dialogue with individuals who are experiencing oppression) that represent the “transformative perspective” in research?
Some researchers recommend that you limit your literature review efforts to “mainstream” journals on the grounds that these represent the “cream of the crop” of research efforts. Transformative researchers might contend that this would result in a bias because the viewpoints of oppressed people might not be represented in those journals. Where do you stand on this issue?
If you have chosen to review an ED document from an ERIC search, that document may also be available in full text online. However, if it is not, then the document should be available for your review on microfiche in the library. The need to access documents on microfiche is becoming less and less common. The microfiche are organized in ascending order according to their ED numbers, so they are usually easy to find.
Step 6
Once you have the article in hand, read the document to determine if it is really what you want. You will want to record bibliographic information and notes on each article that you decide is relevant to your topic. This can be done electronically or manually, using old-fashioned note cards.
Bibliographic Information
If you are searching such databases as ERIC or PsycARTICLES, you can use a new digital resource called RefWorks to electronically save the bibliographic information about all the references that you select. When you are ready, RefWorks will print out a reference list in APA format (or the format that you select). That is not all: If the article is available in full text, you can save it in RefWorks with the bibliographic information. If you do not have access to this electronic resource, then you can save the bibliographic information on note cards, a word processing document, or a database file such as Excel. The important thing is to make sure you get ALL the information you need when you are working with the document so you do not have to try to find it later when you are writing up your literature review. Words of wisdom—painfully learned.
The most important thing to remember in recording bibliographic information is to be complete and accurate.:
Read and Prepare Bibliographic Information and Notes
The most important thing to remember in recording bibliographic information is to be complete and accurate. Some of the problems associated with recording bibliographic information have been reduced because of the ability to print such information directly from the computer screen. However, if you have hundreds of printouts, you may want to record the information on index cards or in some other easily retrievable electronic format. (I do not always have a computer with me when I want to record bibliographic information, so index cards are handy, although the memo function on a mobile phone can also be useful. My secret at the moment: I cut and paste from the full-text documents and then type up the references later; next time I write a book, I plan to use RefWorks.)
Although several options are available for the format of recording bibliographic information, the most common style for education and psychology is based on the American Psychological Association’s (2009) Publication Manual (6th ed.). This is the basic format for a journal citation:
Author’s Last Name, Initials. (date of publication). Title of journal article. Title of Journal, volume number(issue number), page numbers.
For example:
Black, P. A. & Glickman, N. S. (2006). Demographics, psychiatric diagnoses, and other characteristics of North American Deaf and hard-of-hearing inpatients. Journal of Deaf Studies and Deaf Education, 11(3), 303–321.
Book
Author’s Last Name, Initials. (date of publication). Title of book. Place of publication: Publisher.
For example:
Mertens, D. M. (2009). Transformative research and evaluation. New York, NY: Guilford.
Book Chapter
Author’s Last Name, Initials. (date of publication). Title of chapter. In Name of Editor (Ed.), Title of book (page numbers of chapter). Place of publication: Publisher.
For example:
LaFrance, J., & Crazy Bull, C. (2009). Researching ourselves back to life: Taking control of the research agenda in Indian Country. In D. M. Mertens & P. Ginsberg (Eds.), Handbook of social research ethics (pp. 135–149). Thousand Oaks, CA: Sage.
There are differences in citation style associated with different types of documents (e.g., books, chapters in books, government reports, etc.), so you are advised to obtain a copy of the APA Publication Manual (2009) to guide the compilation of bibliographic information. In addition, APA has added a great deal of information about how to handle Web-based information. Some of that is reflected in the Publication Manual, but you can also go to the APA’s website (www.apa.org) for an update on changes that they recommend or go to the APA style blog at http://blog.apastyle.org/.
The Purdue Online Writing Lab (https://owl.english.purdue.edu/owl/section/2/10/) also gives excellent examples of the many types of references you might encounter, including the digital object identifier (DOI) number when it is available for documents published online.
Notes on Each Study
Exactly what notes to write for each study varies greatly and depends on the nature of the study, the purpose of the review, and the intended use of the data. If the researcher intends to conduct a comprehensive literature review of studies that report their results in statistical form, the use of coding forms and computerized databases is recommended.
For empirical research studies (quantitative, qualitative, or mixed methods), the following outline can be helpful:
Area of interest; literature cited; need addressed; theoretical framework; research questions/hypothesis
Paradigm of researcher(s)
Design, including
Specific research approach
Sampling strategy
Characteristics of participants
Data collection instruments and procedures
Data analysis strategy
Results
Conclusions
Your own evaluation (including strengths and weaknesses and ideas for your own research, such as promising methodological or conceptual suggestions)
The evaluation of research reports is Step 7 (discussed next). Once you have evaluated the research report, you should return to your note cards or files and enter your own assessment of the strengths and weaknesses of the research.
Step 7: Evaluate the Research Reports
The criteria that are used to evaluate the research studies will depend on the design and data collection methods used in the study. You will be learning how to evaluate research as you progress through this text. A listing of critical analysis questions for evaluating primary research is provided at the end of each chapter. The questions are organized according to the sections of the research report (e.g., introduction, method, etc.), with additional specific questions relevant to each approach to research (e.g., experimental, quasi-experimental, etc.).
The APA Publications and Communications Board Task Force issued two reports (available at this textbook’s website) that are useful in this regard: one addressed criteria for reporting qualitative and mixed methods research (Levitt, Bamberg, Creswell, Frost, Josselson, & Suárez-Orozco, 2018) and another for quantitative research (Appelbaum, Cooper, Kline, Mayo-Wilson, Nezu, & Rao, 2018). The criteria for quantitative research distinguished between research that involved experimental manipulation, research that did not involve experimental manipulation, research with single subjects, and research in which data were collected on more than one occasion. Given that you have not yet studied designs, sampling, and so forth, you are not yet ready to apply the detailed guidance they provide when reading research studies. However, I provide you with a brief review checklist for quantitative, qualitative, and mixed methods studies based on the APA task force’s recommendations in Box 3.10. You will deepen your understanding of how to critically evaluate each part of a research study as you continue through this text and answer the questions that are relevant at the end of each chapter.
Step 8: Synthesize the Studies
The Harvard University Graduate School of Education (2018) defines synthesis in a literature review this way: To synthesize is to combine two or more elements to form a new whole. In the literature review, the “elements” are the findings of the literature you gather and read; the “new whole” is the conclusion you draw from those findings.
Before you actually begin the synthesis of the research, there are a few things to keep in mind. Organization is a plus. If you can develop a flexible framework for organizing the studies as you find them, it will be easier for you to approach the synthesis stage. I say flexible because the framework might add, delete, or redefine categories as you move through the review process. For example, for the revisions to the fifth edition of this book, I had categories for relevant studies I found in the early stages that related to each chapter title. As I found studies, I saved them into those files. As I began writing, I moved some of the files to more differentiated categories; for example, what started as the Chapter 1 introduction became paradigms, and paradigms became postpositivist, constructivist, transformative, and pragmatic. So as I approached writing, I had studies that were relevant to each part of the chapter. Of course, you sometimes run into problems in that one study may have applicability for more than one topic, but no one said this would be simple. I found using an Excel file with the author, date, approach, topic of the study, and the chapter number in which the article is cited helped me keep straight the many references in this book as I moved through the revisions. Some of my students like to keep tables of the studies they find and then organize the studies in tables under headings that make sense for their writing; some like to keep note cards on their studies that they can rearrange manually. Other students like to use a qualitative software package to save the articles as “data files” that can be coded and searched when it is time to write. This approach has merit but demands technical skills with the software and diligence to do the coding. However, once you get to the search-and-retrieve part, it may all seem worth it.
Box 3. Checklist for Evaluating Research Studies
Quantitative Studies
Introduction
Problem: Importance is clear
Literature review: Reports previous work and differences between that and current work
Hypothesis/aims/objectives: Specifically stated; based on literature/theory; connects to design
Method
Participants: Inclusion/exclusion criteria and demographic characteristics provided
Sampling: Sampling method clearly explained along with settings and locations; IRB approval; clearly explain decision basis for sample size and power of sample size
Data collection: Describe all instruments used; reliability; validity; administration procedures
Design: Specify design: experimental or not; randomization or not; longitudinal design; n-of-1
Data analysis, interpretation, results, discussion: Methods used to insure integrity of the data; analytic strategies used are appropriate; provide support for interpretations based on data; discuss relation to previous research, generalizability, and implications for future research10
Qualitative Studies
Introduction
Problem/question: Literature is used to frame the problem or question and its context; applicable literature is reviewed and critiqued to clarify knowledge gaps or practical needs.
Objectives/aims/goals: Purpose/aims/goals are reported along with target audience; provides rationale for design and approach to inquiry.
Method
Research design: Summarize research design (data collection, analysis, approaches, etc.)
Researcher/participants/data sources: Describe researcher’s relevant background for conducting this study; provide numbers and characteristics of participants/data sources (e.g., newspapers, Internet); describe nature of relationships and interactions between researcher and participants; describe recruitment and selection processes.
Data collection: Identify form of data collected; alterations made over the course of the study; amount of time spent in data collection; provide questions asked during data collection; identify recording methods for data.
Analysis/interpretation/integrity/results: Describe methods used to analyze data, codes, coding process, software used.
Describe strategies used to insure the integrity of the methods and claims made based on the analysis. Support importance of findings, similarities and differences from prior research; limits of transferability.
Mixed Methods Studies
Introduction
Problem/question: Same as for quantitative (quant) and qualitative (qual) research
Method
Design: See quant/qual standards and explain why mixed methods (MM) is appropriate and the MM design used.
Participants: Same as quant/qual; clearly identify sources of each type of data and where they fit in the MM design; describe samples for quant and qual in separate sections.
Data collection: Same as quant/qual
Data analysis/interpretation/results: Same as quant/qual but have separate sections for quant and qual data reporting; be specific about integration of findings from each part of the study. Show integration of findings via discussion, joint table displays, or graphs.