Critical appraisal

What is critical appraisal, why is it important and some useful resources.

Critical appraisal is an essential part of evidence-based healthcare.

The evidence cycle makes it clear that evidence should not just be found and then used, it also needs to be critically appraised. Numerous Randomised Controlled Trials (RCTs) are published every day, let alone other study types, and not all are of consistent quality.

Critical appraisal is important so that the good studies can be found amongst the many. These can then be used to confirm or change treatments and advice given to patients.

Critical appraisal generally seeks to answer two broad questions.

  • Is this study likely to be accurate (validity)?
  • Can we use its findings (relevance)?

To answer these questions, we pose further questions. We start by looking at what the purpose of the article was, and then whether the chosen study methodology was appropriate for answering this question.

Following that the described methods need to be considered for anything that is unclear, anything that seems confused, potential for bias, or confounders that have not been mitigated against.

There will very rarely be a perfect paper where everything that could be explained has been clearly done so, but just because a paper is imperfect, it does not mean it should be ignored.

The point is to look for potential flaws in the research that mean either its validity cannot be trusted, or it is no longer relevant in our setting or situation.

Finally, the results and conclusions should be looked at to see what the results were, how precise they were and whether they were statistically significant.

Critical appraisal does not mean simply summarising a paper descriptively. It is about forming a judgement upon it. The discipline of critical appraisal is critical in sense of questioning the research, its results, and conclusions, not in the sense of having to be critical of a paper.

It is alright to agree with all the conclusions of the authors, and it is fine to disagree with them all, or to agree with some and disagree with others, however it is not OK to be led to their conclusions by them. The reader should form their own conclusions.

No particular expertise is required to critically appraise an article, though more subject specific knowledge is an advantage for more technical papers.

The more experience and confidence an appraiser has the quicker they should be able to form their own opinions on the study.

Example from practice by Steve Glover

University Hospitals of Leicester NHS Libraries provide their critical appraisal training, called ‘Critical Reading Made Easy’, in two parts. In the morning session they cover the introduction to, and principles of, critical appraisal.

This covers, what it is, why it is important, an overview of the main study methodologies and some of the main statistical terms that papers use.

Throughout, the session is interspersed with games and activities which help to make a dry subject a little more fun and re-enforce the main learning points. These include activities to design your own clinical study, a quiz on which methodology best addresses different issues, and which treatment would you recommend based upon various statistics. Examples used are never too technical and tend to be quite light-hearted.

The other part of the training is the afternoon practical session in which papers are given out to be read by participants and then discussed as a group. This session only take place if there is a group of at least 3 as the group discussion element is vital.

Again, papers chosen reflect the fact that the groups are mixed abilities and include people from different specialties and job roles. They also tend to be more light-hearted papers

Every time the session is run different papers are chosen, which keeps the sessions interesting for the tutors and it means that everyone can get more practice as participants can come back multiple times.

Aims of the session are to raise the confidence of participants and the course is therefore evaluated against that aim. Feedback collected since 2016 from over 200 attendees show that on average this course increases confidence by just over 95%. The course has also been described as ‘the most fun that critical appraisal can be’.

Resources

Books

Bootland, D. (2014) Critical Appraisal for FCEM, CRC Press

Clearly written jargon free book aimed at the FCEM but very useful of learning and improving critical appraisal skills. [Last checked: 01/09/21]

Gosall, N. (2015) The Doctor’s Guide to Critical Appraisal (4th ed), PasTest

A comprehensive guide to understanding strengths and weaknesses of research design and interpreting results. [Last checked: 01/09/21]

Greenhalgh, T. (2019) How to Read a Paper (6th ed), Wiley-Blackwell

How to Read a Paper demystifies evidence-based medicine and explains how to critically appraise published research and also put the findings into practice [Last checked: 01/09/21]

Holloway, I., Kalvin, K. (2016) Qualitative Research in Nursing and Healthcare (4th ed), Wiley-Blackwell

This book provides a good explanation of qualitative research enabling the reader to build critical appraisal skills for this area of research. [Last checked: 01/09/21]

Perea, R. (2008) Statistics Toolkit. BMJ Books

Guides the reader through statistical concepts using flowcharts, diagrams and real life examples to reflect concepts in a simple and practical manner [Last checked: 01/09/21]

Petrie, A. and Sabin, C. (2019) Medical Statistics at a Glance (4th ed), Wiley-Blackwell

A concise and accessible introduction to this complex subject. It provides clear instruction on how to apply commonly used statistical procedures in an easy-to-read, comprehensive and relevant volume. [Last checked: 01/09/21]

Salkind, N. (2019) Statistics for People Who Think They Hate Statistics (7th ed), Sage

Textbook with self test questions and answers [Last checked: 01/09/21]

Courses

Critical Appraisal Skills Programme (CASP)

CASP is part of Better Value Healthcare, a training organisation led by Professor Muir Gray, based in Oxford. Provides workshops, checklists and e-modules. [Last checked: 01/09/21]

Open University: Mathematics and statistics

Free online course on mathematics and statistics for beginners. [Last checked: 01/09/21]

Websites

AGREE (Appraisal of Guidelines for Research and Evaluation)

Provides a tool for evaluating the quality and reporting of practice guidelines. [Last checked: 01/09/21]

BMJ Endgames: Statistics

Dr Philip Sedgewick is the author of the BMJ Endgames statistcs series. These are short clinically relevant explainations of statistical concepts. Note: no new content since 2015. [Last checked: 01/09/21]

Centre for Evidence-Based Medicine (CEBM)

From the University of Oxford. Checklists, workshops, educational resources. [Last checked: 01/09/21]

Cochrane Handbook (2021).

Guidance to authors preparing Cochrane reviews. Provides comprehensive explanations of bias. [Last checked: 01/09/21]

CONSORT (Consolidated Standards of Reporting Trials)

Evidence-based, minimum set of recommendations for reporting randomized trials. Provides a checklist for how RCTs are reported. This also helps to develop understanding of research design. [Last checked: 01/09/21]

Mathematics and Statistics for Dummies by Wiley

Short explanations of statistical concepts for beginners. [Last checked: 01/09/21]

PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analysis)

Evidence-based minimum set of items for reporting in systematic reviews and meta-analyses. Provides a checklist and flow diagram. This also helps to develop understanding of research design. [Last checked: 01/09/21]

STROBE (Strengthening the Reporting of Observational studies in Epidemiology)

STROBE is an international, collaborative initiative of epidemiologists, methodologists, statisticians, researchers and journal editors involved in the conduct and dissemination of observational studies, with the common aim of STrengthening the Reporting of OBservational studies in Epidemiology. Provides checklists and also helps to develop understanding of research design. [Last checked: 01/09/21]

 

Page last reviewed: 7 September 2021