Critically Evaluating Research

Some research reports or assessments will require you critically evaluate a journal article or piece of research. Below is a guide with examples of how to critically evaluate research and how to communicate your ideas in writing.

To develop the skill of being able to critically evaluate, when reading research articles in psychology read with an open mind and be active when reading. Ask questions as you go and see if the answers are provided. Initially skim through the article to gain an overview of the problem, the design, methods, and conclusions. Then read for details and consider the questions provided below for each section of a journal article.

Title

  • Did the title describe the study?

  • Did the key words of the title serve as key elements of the article?

  • Was the title concise, i.e., free of distracting or extraneous phrases?

Abstract

  • Was the abstract concise and to the point?

  • Did the abstract summarise the study’s purpose/research problem, the independent and dependent variables under study, methods, main findings, and conclusions?

  • Did the abstract provide you with sufficient information to determine what the study is about and whether you would be interested in reading the entire article?

Introduction

  • Was the research problem clearly identified?

  • Is the problem significant enough to warrant the study that was conducted?

  • Did the authors present an appropriate theoretical rationale for the study?

  • Is the literature review informative and comprehensive or are there gaps?

  • Are the variables adequately explained and operationalised?

  • Are hypotheses and research questions clearly stated? Are they directional? Do the author’s hypotheses and/or research questions seem logical in light of the conceptual framework and research problem?

  • Overall, does the literature review lead logically into the Method section?

Method

  • Is the sample clearly described, in terms of size, relevant characteristics (gender, age, SES, etc), selection and assignment procedures, and whether any inducements were used to solicit subjects (payment, subject credit, free therapy, etc)?

  • What population do the subjects represent (external validity)?

  • Are there sufficient subjects to produce adequate power (statistical validity)?

  • Have the variables and measurement techniques been clearly operationalised?

  • Do the measures/instruments seem appropriate as measures of the variables under study (construct validity)?

  • Have the authors included sufficient information about the psychometric properties (eg. reliability and validity) of the instruments?

  • Are the materials used in conducting the study or in collecting data clearly described?

  • Are the study’s scientific procedures thoroughly described in chronological order?

  • Is the design of the study identified (or made evident)?

  • Do the design and procedures seem appropriate in light of the research problem, conceptual framework, and research questions/hypotheses?

  • Are there other factors that might explain the differences between groups (internal validity)?

  • Were subjects randomly assigned to groups so there was no systematic bias in favour of one group? Was there a differential drop-out rate from groups so that bias was introduced (internal validity and attrition)?

  • Were all the necessary control groups used? Were participants in each group treated identically except for the administration of the independent variable?

  • Were steps taken to prevent subject bias and/or experimenter bias, eg, blind or double blind procedures?

  • Were steps taken to control for other possible confounds such as regression to the mean, history effects, order effects, etc (internal validity)?

  • Were ethical considerations adhered to, eg, debriefing, anonymity, informed consent, voluntary participation?

  • Overall, does the method section provide sufficient information to replicate the study?

Results

  • Are the findings complete, clearly presented, comprehensible, and well organised?

  • Are data coding and analysis appropriate in light of the study’s design and hypotheses? Are the statistics reported correctly and fully, eg. are degrees of freedom and p values given?

  • Have the assumptions of the statistical analyses been met, eg. does one group have very different variance to the others?

  • Are salient results connected directly to hypotheses? Are there superfluous results presented that are not relevant to the hypotheses or research question?

  • Are tables and figures clearly labelled? Well-organised? Necessary (non-duplicative of text)?

  • If a significant result is obtained, consider effect size. Is the finding meaningful? If a non-significant result is found, could low power be an issue? Were there sufficient levels of the IV?

  • If necessary have appropriate post-hoc analyses been performed? Were any transformations performed; if so, were there valid reasons? Were data collapsed over any IVs; if so, were there valid reasons? If any data was eliminated, were valid reasons given?

Discussion and Conclusion

  • Are findings adequately interpreted and discussed in terms of the stated research problem, conceptual framework, and hypotheses?

  • Is the interpretation adequate? i.e., does it go too far given what was actually done or not far enough? Are non-significant findings interpreted inappropriately?

  • Is the discussion biased? Are the limitations of the study delineated?

  • Are implications for future research and/or practical application identified?

  • Are the overall conclusions warranted by the data and any limitations in the study? Are the conclusions restricted to the population under study or are they generalised too widely?

References

  • Is the reference list sufficiently specific to the topic under investigation and current?

  • Are citations used appropriately in the text?

General Evaluation

  • Is the article objective, well written and organised?

  • Does the information provided allow you to replicate the study in all its details?

  • Was the study worth doing? Does the study provide an answer to a practical or important problem? Does it have theoretical importance? Does it represent a methodological or technical advance? Does it demonstrate a previously undocumented phenomenon? Does it explore the conditions under which a phenomenon occurs?

How to turn your critical evaluation into writing
Identify Your Critical Points

Firstly, write down the points you want to make. Do this by using the above mentioned questions to go through the research to scan for potential problems. Then list these down and note if there any associated validity or reliability weaknesses related to your points (e.g. participants dropping out of the study [attrition] or, the variables were not clearly operationalised [construct validity difficult to determine].)

Decide on the Structure of your Critical Evaluation

Next, consider how you will structure your writing. If you are asked to critically evaluate the method and results of a study, then you could have a paragraph for each section. Or perhaps you want to structure your evaluation by considering the types of validity and reliability. Structuring your ideas is key to writing a clear and concise critical evaluation.

Explain Your Idea

Once you start writing your ideas down ensure you explain why a point is problematic to the study. First, identify the point in your writing and in the following sentence/s, explain why point was a weakness of the research. For example, "The instrument used to measure performance by Garcia (2014) has previously been used in sporting contexts only. However, the study by Garcia used this measure in a artistic context questioning whether the measure was appropriate to use. Consequently, the construct validity of this measure is questionable potentially reducing the reliability of the research conclusions."
Example from a journal article
An example of how to step-by-step critically evaluate a section of a method from a journal article is provided below:

Step 1. Identify our critical point: The sections outlined in red below identify two sections where psychometric properties (e.g. Cronbach's alpha or convergent validity) have not been provided for the measures.

Step 2. We have two examples from the method so lets present the idea in a critical evaluation paragraph about the method.

Step 3. Write our critical point first (blue font) and then explain why it's a weakness of the research (red font): "Although the research by Cseh, Phillips, and Pearson (2015) provide innovative findings, the method of their study has a few limitations. Firstly, the authors did not report psychometric properties for the Flow State Scale 2 and the Post-task creativity questionnaire. Therefore, the reliability and validity of these measures is not certain and the trustworthiness of the research findings is questionable."

Step 4. Continue find more critical points within the method to finish your paragraph.


Sources

Content adapted from Chapter 2 of The SPSS Manual provided in RMS1 and RMS2.
Graziano, A.M. & Raulin, M.L. (2007). Research methods: A process of inquiry (6th Edition). Boston, MA: Pearson Educational Group.
Cseh, G. M., Phillips, L. H., & Pearson, D. G. (2015). Flow, affect and visual creativity. Cognition and Emotion, 29(2), 281-291. doi: 10.1080/02699931.2014.913553