Systematic Review Methods Programma del Corso November 21-25, 2016 Universitaria di Bologna Dr. Jenny Yost, RN, PhD [email protected] School of Nursing McMaster University Hamilton, Ontario, Canada 1

(1)

Systematic Review Methods Programma del Corso

November 21-25, 2016 Universitaria di Bologna

Dr. Jenny Yost, RN, PhD [email protected]

School of Nursing McMaster University Hamilton, Ontario, Canada

(2)

Course Objectives

At the completion of this introductory course to systematic review methods, the learner should be able to:

1. Understand the importance of systematic reviews in health care policy and practice decisions

2. Develop a clearly focused question

3. Identify appropriate inclusion and exclusion criteria 4. Develop a comprehensive search strategy

5. Identify appropriate criteria to assess the methodological quality of included studies

6. Describe methods for data synthesis

7. Demonstrate ability to interpret the results of meta-analyses 8. Develop a protocol for a systematic review

Evaluation

The final grade will be calculated based on two criteria:

1. Class Participation (20%)

2. Assignment: Systematic Review Protocol Presentation (80%) Class Participation

Each student will be independently evaluated on his/her level of engagement in the class. This will include: evidence of adequate preparation and critical thinking

demonstrated through questions asked in class, active participation in and substantive contributions to the assigned learning activities, responding thoughtfully to other

contributing new knowledge or resources to peers, and regular attendance.

Assignment: Systematic Review Protocol Presentation

This is a small-group assignment to be completed in groups of 2-3 individuals.

The overall purpose of the assignment is to develop a protocol for conducting a systematic review, applying the principles reviewed during the week. Each group will present their study proposal orally (15 minutes maximum) on Friday November 25, 2016. Dr. Yost and the other students in the course will critically evaluate the presentation (p. 11). The presentations can be delivered in either Italian or English.

Each group will be required to respond to questions (5 minutes) from their peers and Dr.

Yost. Each group will be asked to submit a copy of their presentation to Dr. Yost following the presentation.

2

(3)

Assignment Guidelines:

1. Identify an important topic concerning the effectiveness of a health care intervention.

2. Provide brief rationale for why the topic is important and why a systematic review needs to be conducted (e.g. based on gaps in the literature; clinical observations).

3. Identify team members/collaborators, their affiliation, and their role in the review.

4. Develop a clearly focused quantitative review question using PICO (population, intervention, comparison/control, and outcome). NOTE: if applicable, distinguish between primary and secondary outcome(s).

5. Identify appropriate inclusion and exclusion criteria to be applied in the review, including study design(s).

6. Describe a comprehensive strategy for searching for relevant published and unpublished studies.

7. Identify appropriate criteria to assess the methodological quality of the included studies.

8. Describe the approach for synthesizing the studies.

9. Provide details for communicating key messages from the review to appropriate audiences.

Course Schedule

This is the proposed schedule for the course. However, the schedule will be adapted as required to meet the learning needs and objectives of the students.

Monday November 21, 2016 Unit 1. Introduction to Systematic Reviews and Framing the Question

Tuesday November 22, 2016 Unit 2. Inclusion and Exclusion Criteria and Developing a Search Strategy

Wednesday November 23, 2016 Unit 3. Study Selection, Data Extraction, and Critical Appraisal of Intervention Studies

Thursday November 24, 2016 Unit 4. Summarizing the Evidence

Friday November 25, 2016 Unit 5. Presentation of Proposals, Evaluation &

Feedback

(4)

Unit 1: Introduction to Systematic Reviews and Framing the Question Unit Objectives:

1. Differentiate between different types of reviews.

2. Identify the importance of systematic reviews in health care policy and practice decisions.

3. Identify the components of high quality systematic reviews.

4. Develop a clearly focused quantitative systematic review question about the effectiveness of an intervention.

Required Readings:

Center for Reviews and Dissemination.(2008). 1.1 Getting Started. In Systematic

reviews: CRD’s guidance for undertaking reviews in healthcare (pp. 3-5). Retrieved from https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf

Center for Reviews and Dissemination.(2008). 1.2 The Review Protocol. In Systematic reviews: CRD’s guidance for undertaking reviews in healthcare (pp. 6-9). Retrieved from https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf

Cipriani, A., & Barbui, C. (2003). Dalle revisioni narrative a quelle sistematiche

e alle metanalisi, per un approccio modern alla valutazione dei trattamenti farmacologici in psichiatria. La Valutazione Dell’esito Dei Trattamenti in Psichiatria, 3, 221-234.

Retrieved from http://www.e-noos.it/rivista/3_03/pdf/4.pdf

Moher, D., Liberati, A., Tetzlaff, J., Altman, D., & The Prisma Group. (2015). Linee guida per il reporting di revisioni sistematiche e meta-analisi: Il PRISMA statement. Evidence, 7(6), 1-8. Retrieved from http://prisma-statement.org/documents/PRISMA%20Italian

%20Statement.pdf

Mosci, D., & Robb, M.C. (1999). Identificazione dei disegni di ricerca che meglio si adattano alla domanda. Parte 1: Disegni quantitativi. [Translated from Roberts, J. &

DiCenso, A. (1999). Identifying the best research design to fit the question. Part 1:

quantitative designs. Evidence Based Nursing, 2, 4-6.] Online http://www.evidencebasednursing.it/traduzioni_ebn/a1999_2_1.pdf

Other Helpful Resources:

6S hierarchy of pre-processed research evidence

 http://www.evidencebasednursing.it/6S.html

4

(5)

Prospero (Systematic Review Registration):

 http://www.crd.york.ac.uk/PROSPERO/joinup.php

 http://www.crd.york.ac.uk/PROSPERO/printPDF.php?

RecordID=39363&UserID=2105

Cochrane Handbook for Systematic Review of Interventions

 www.handbook.cochrane.org

 Chapter 2: Preparing a Cochrane review Cochrane Training

 http://training.cochrane.org/search/site/?f[0]=bundle

%3Aresource&f[1]=ss_language%3Aen

In-Class Activities:

1. Introductions.

2. Establish course ground rules.

3. Review course evaluation methods.

4. Each student will give a 2 minute presentation providing a brief overview of their systematic review question about the effectiveness of an intervention..

5. Identify small groups of 2-3 students for Assignment: Systematic Review Protocol Presentation.

6. Justify the group’s systematic review question.

7. Identify the “team” for the group’s systematic review.

(6)

Frame the Systematic Review Question:

Type of question (effectiveness, causation/harm, prognosis, diagnosis):

Free form question (sentence form):

Structured Question (PICO)

P:

I:

C:

O:

Study design(s):



6

(7)

Justify the Systematic Review Question: Part 1 Resource

Searched

# of Related Results Comments

Justify the Systematic Review Question: Part 2 Importance/Background of the Problem



Rationale for Conducting the Review/Gap in the Synthesized Literature



(8)

Identify the Systematic Review Team

Who will use the findings from the review?

Expertise in the content area (e.g.

clinicians, consumers):

Expertise in

systematic review methods:

8

(9)

Unit 2: Inclusion and Exclusion Criteria and Developing a Search Strategy Unit Objectives:

1. Identify the components to address in developing inclusion and exclusion criteria for a systematic review question about the effectiveness of an intervention.

2. Identify the components of a comprehensive systematic review search strategy.

Center for Reviews and Dissemination.(2008). 1.2 The Review Protocol. In Systematic reviews: CRD’s guidance for undertaking reviews in healthcare (pp. 10-13). Retrieved from https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf

Center for Reviews and Dissemination.(2008). 1.2 Undertaking the Review. In

Systematic reviews: CRD’s guidance for undertaking reviews in healthcare (pp. 16-27).

Retrieved from https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf

Robb, M.C. (1998). Cercando la migliore evidenza. Parte 2: Cercando su CINAHl e Medline. [Translated from McKibbon A. K., & Marks, S. (1998). Searching for the best evidence. Part 2: searching CINAHL and Medline. Evidence Based Nursing, 1, 105- 107.] Retrieved from http://www.evidencebasednursing.it/traduzioni_ebn/a1998_1_4.pdf

 Chapter 2: Preparing a Cochrane review

 Chapter 5: Defining the review question and developing criteria for including studies

 Chapter 6: Sources to Search Cochrane Training

1. Identify inclusion and exclusion criteria for the group’s systematic review question.

2. Identify search terms for the group’s systematic review question.

3. Identify resources to search relevant to the group’s systematic review question.

(10)

Identify Inclusion/Exclusion Criteria:

Question Component Inclusion Criteria Exclusion Criteria Population

Intervention

Comparison

Outcome

Study Design

10

(11)

Identify Search Terms:

Question

Component Term Synonyms Spelling

Variants Abbreviation Population

Intervention

Comparison

Outcome

(12)

Where to Search:

Resource √ to

search

Rationale for searching Electronic Databases:

MEDLINE: available freely via Pubmed;

bibliographic records (with and without abstracts) or biomedical literature from 1996 onwards;

www.ncbi.nlm.nih.gov/pubmed/

EMBASE: records of biomedical literature from 1974 onwards

PsychInfo: records of literature on psychology and related behavioural and social sciences from 1967 Cumulative Index to Nursing and Allied Health Literature (CINAHL): records of literature on all aspects of nursing and allied health disciplines Allied and Complementary Medicine Database (AMED): alternative medicine database with abstracts included for many records, dating back to 1995.

NHS Economic Evaluation Database: structured abstracts of economic evaluations of healthcare interventions identified by regular searching of bibliographic databases, hand searching of key major medical journals;

www.crd.york.ac.uk/CRDWeb/

MIDIRS: broad reference available to obstetricians, midwives, and consumers;https://www.midirs.org/

OTseeker: systematic reviews and appraised randomized trials in occupational therapy freely available; www.otseeker.com

Physiotherapy Evidence Database (PEDro):

systematic reviews and appraised randomized trials in physiotherapy freely available;

www.pedro.fhs.usyd.edu.au/

Scopus: 15,000 journals (of which over 1,200 are open access journals) and 500 conference

proceedings

Other Electronic Databases:

For additional examples of electronic databases see the Cochrane Handbook, Box 6.2.b Examples of subject-specific electronic databases

12

(13)

Trial Registers:

The Cochrane Central Register of Controlled Trials (CENTRAL): records of clinical trials in healthcare identified through the Cochrane Collaboration,

http://www.cochranelibrary.com/about/central- landing-page.html

International Standard Randomized Controlled Trial Number (ISRCTN): www.isrctn.com

UK Clinical Research Network:

https://www.ukctg.nihr.ac.uk/

Other Trial Registers: For additional examples of trial registers see the Cochrane Handbook, Box 6.2.h: Examples of national and international trials registers

Citation Search

Science Citation Index/Web of Science: relevant studies found through electronic and/or manual search can be used to identify further relevant citations by electronically locating other citation on the same topic

Dissertations

ProQuest Dissertation & Theses Database:

indexes more than 2 million doctoral dissertations and masters’ theses

Index to Theses: over 500,000 theses in Great Britain and Ireland; www.theses.com

DissOnline: indexes 50,000 German dissertations;

www.dissonline.de

Conference Proceedings Scopus

BIOSIS

Index of Scientific and Technical Proceedings Conference Papers Index

(14)

Other Specific Conferences:

Grey Literature

Open Grey: bibliographical references of grey literature produced in Europe;

http://www.opengrey.eu/

British National Bibliography of Report Literature: www.bl.uk

National Technical Information Service:

www.ntis.gov

Hand Searching Key Journals:

Hand Searching of Reference Lists:

Contacting Experts:

14

(15)

Unit 3: Study Selection, Data Extraction, and Critical Appraisal of Intervention Studies

Unit Objectives:

1. Identify the components to address in developing study selection criteria for a systematic review question.

2. Identify the components to address in developing a data extraction tool for a systematic review question.

3. Identify relevant appraisal tools for conducting a systematic review about the effectiveness of an intervention.

Center for Reviews and Dissemination.(2008). 1.3.2 Study Selection. In Systematic reviews: CRD’s guidance for undertaking reviews in healthcare (pp. 23-27). Retrieved from https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf

Center for Reviews and Dissemination.(2008). 1.3.3 Data Extraction. In Systematic reviews: CRD’s guidance for undertaking reviews in healthcare (pp. 28-32). Retrieved from https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf

Center for Reviews and Dissemination.(2008). 1.3.4 Quality Assessment. In Systematic reviews: CRD’s guidance for undertaking reviews in healthcare (pp. 33-44). Retrieved from (https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf

Ciliska, D., Thomas, H., & Buffet, C. (2012). A Compendium of Critical Appraisal Tools for Public Health Practice (Revised). [tool]. Hamilton, ON: National Collaborating Centre for Methods and Tools. Retrieved from

http://www.nccmt.ca/pubs/CompendiumToolENG.pdf

Mosci, D. (2000). Statistics per evidence based nursing. [Translated from Sheldon, T.

(2000). Statistics for evidence-based nursing. Evidence Based Nursing, 3, 4-6.] Online http://www.evidencebasednursing.it/traduzioni_ebn/a2000_3_1.pdf

Pellecchia, C., & Robb, M.C. (2000). Valutazione degli studi sul trattamento o sugli interventi di prevenzione. [Translated from Cullum, N. (200). Evaluation of studies of treatment or prevention interventions. Evidence Based Nursing, 3, 100-102.] Online http://www.evidencebasednursing.it/traduzioni_ebn/ug_2000_3_4.pdf

(16)

 Chapter 7: Selecting studies and collecting data

 Chapter 8: Assessing risk of bias in included studies Cochrane Training

Effective Practice and Organisation of Care (EPOC) (2013). Data collection form:

EPOC Resources for review authors. Oslo: Norwegian Knowledge Centre for the Health Services. Retrieved from: http://epoc.cochrane.org/epoc-specific-resources-review- authors

Systematic Review Software:

 Rayyan: http://rayyan.qcri.org/

 Distiller: https://distillercer.com

 Covidence: https://www.covidence.org/

 EPPI Reviewer: http://eppi.ioe.ac.uk/cms/Default.aspx?

alias=eppi.ioe.ac.uk/cms/er4

1. Identify study selection criteria for Title and Abstract screening for the group’s systematic review question.

2. Identify study selection criteria for Full Text screening for the group’s systematic review question.

3. Identify appropriate critical appraisal tool(s) for the group’s systematic review question.

16

(17)

Identify Study Selection/Screening Criteria:

Question Component

Title & Abstract Full Text Population

Intervention/

Exposure

Comparison

Outcome

Study Design

Other

(18)

Select a Risk of Bias/Quality Assessment Tool for Intervention Studies:

Tool √ to Use Rationale for Using

Cochrane Risk of Bias:

http://methods.cochrane.org/bi as/assessing-risk-bias-

included-studies#Sources

%20of%20Bias%20in

%20Clinical%20Trials Critical Appraisal Skill Programme (CASP):

Randomized Controlled Trial Checklist

http://www.casp-uk.net/casp- tools-checklists

ROBINS-I: a tool for assessing risk of bias in non-randomised

studies of interventions http://www.bmj.com/content/b mj/355/bmj.i4919.full.pdf Effective Public Health Practice Project: Quality Assessment tool for Quantitative Studies

http://www.ephpp.ca/tools.html Scottish Intercollegiate Guideline Network:

Methodology Checklist 2:

Randomised Controlled Trials

http://www.sign.ac.uk/methodo logy/checklists.html

Other Tool:

18

(19)

Unit 4: Summarizing the Evidence Unit Objectives:

1. Identify and determine appropriate methods for data synthesis 2. Differentiate between clinical and statistical heterogeneity

3. Identify the components assessed in applying GRADE to systematic reviews 4. Identify strategies for communicating the results of a systematic review

Center for Reviews and Dissemination. (2008). 1.3.5 Data Synthesis. In Systematic reviews: CRD’s guidance for undertaking reviews in healthcare (pp. 45-76). Retrieved from https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf

Center for Reviews and Dissemination.(2008). 1.3.6 Report Writing. In Systematic reviews: CRD’s guidance for undertaking reviews in healthcare (pp. 77-83). Retrieved from https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf

Effective Practice and Organisation of Care. (2013) Synthesising results when it does not make sense to do a meta-analysis: EPOC Resources for review authors. Oslo:

Norwegian Knowledge Centre for the Health Services. Retrieved from http://epoc.cochrane.org/epoc-specific-resources-review-authors

Barwick, M. (2013). Knowledge Translation Planning Template. Ontario: The Hospital for Sick Children. Retrieved from http://melaniebarwick.com/training.php

 Chapter 9: Analysing data and undertaking meta-analyses Cochrane Training

%3Aresource&f[1]=ss_language%3Aen GRADE Working Group

 GRADE Website: http://www.gradeworkinggroup.org/

 GRADE Publications: https://gradepro.org/guidelines-development#develop-

(20)

 GRADE Handbook:

http://gdt.guidelinedevelopment.org/app/handbook/handbook.html

GradePro

 https://gradepro.org/

Reardon, R., Lavis, J., & Gibson, J. (2006). Institute for Work and Health: From research to practice: A knowledge transfer planning guide

(http://www.iwh.on.ca/system/files/documents/kte_planning_guide_2006.pdf Review Manager (RevMan)

 http://tech.cochrane.org/revman

1. Assess studies for clinical and statistical heterogeneity.

2. Apply GRADE to an example outcome from a systematic review.

3. Interpret Summary of Findings (SoF) table for example outcome from a systematic review.

4. Identify strategies for communicating the results of a group’s systematic review.

20

(21)

GRADE: Criteria for Assessing Overall Risk of Bias

Risk of bias Across studies Interpretation Considerations GRADE Assessment of Study Limitation

Low risk of bias. Most information is from studies at low risk of bias.

Plausible bias unlikely to seriously alter the results.

No apparent

limitations. No serious limitations, do not downgrade Unclear risk of

bias. Most information is from studies at low or unclear risk of bias.

Plausible bias that raises some doubt about the results.

Potential limitations are unlikely to lower confidence in the estimate of effect.

No serious limitations, do not downgrade.

Potential limitations are likely to lower confidence in the estimate of effect

Serious limitations, downgrade one level.

High risk of bias. The proportion of information from studies at high risk of bias is sufficient to affect the interpretation of results.

Plausible bias that seriously weakens confidence in the results.

Crucial limitation for one criterion, or some limitations for multiple criteria, sufficient to lower confidence in the estimate of effect.

Serious limitations, downgrade one level

Crucial limitation for one or more criteria sufficient to

substantially lower confidence in the estimate of effect

Very serious limitations, downgrade two levels

From: Cochrane Handbook for Systematic Review of Interventions: Table 12.2.d: Further guidelines for factor 1 (of 5) in a GRADE assessment: Going from assessments of risk of bias to judgements about study limitations for main outcomes

(22)

GRADE: Criteria for Assessing Overall Quality

Study Design

Quality of Evidence

Lower if Higher if Confidence in an estimate

of effect across those considerations

Randomized Trial

High (4) Risk of Bias - 1 Serious - 2 Very serious Inconsistency - 1 Serious - 2 Very serious Imprecision - 1 Serious - 2 Very serious Publication Bias - 1 Likely - 2 Very likely

Large effect +1 Large + 2 Very large Dose response + 1 Evidence of a gradient

All plausible confounding + 1 Would reduce a demonstrated effect

+ 1 Would suggest a spurious effect if no effect was observed

High



Moderate Moderate



Non

Randomized Trial

Low (2) Low



Very Low Very low



Applying GRADE

Participants (No. of Studies)

Study Limitations

Inconsistency Indirectness Imprecision Publication Bias

Overall Quality of

Evidence (GRADE)

22

(23)

Communicating Results/Knowledge Translation

Audience Messenger Goal* Strategy for

Translating the Message

*Example Goals:

Generate Impart Inform

 Awareness

 Interest

 Practice Change

 Behaviour Change

 Policy Change

 Knowledge

 Tools

 Research

 Product

 Patent

(24)

Class Participation: Evaluation Criteria

Student: __________________________________________________________

Evaluation Criteria Rating

Poor Fair Good Very Good Excellent 1. Demonstrates adequate

preparation for class discussion.

2. Demonstrates critical thinking in defining and exploring concepts.

3. Actively participates in and provides substantive contributions during assigned learning activities.

4. Responds thoughtfully to others, contributing new knowledge or resources.

5. Regularly attends class sessions.

24

(25)

Systematic Review Protocol Presentation: Evaluation Criteria

Proposal Title: __________________________________________________________

Evaluator: _____________________________________________________________

Rating: Poor Fair Good Very Good Excellent

1 2 3 4 5

Criteria Rating Comments

1. Provides brief rationale for why the topic is important and why a

systematic review needs to be conducted.

1 2 3 4 5

2. Identify team

members/collaborators, their affiliation, and their role in the review.

1 2 3 4 5

3. Develops a clearly focused

quantitative review questions applying PICO/PECO and, if applicable,

distinguishes between primary and secondary outcome(s).

1 2 3 4 5

4. Identifies appropriate inclusion and exclusion criteria to be applied in the review, including study design(s).

1 2 3 4 5

6. Describes a comprehensive strategy for searching for relevant

published and unpublished studies. 1 2 3 4 5 7. Identify appropriate criteria to

assess the methodological quality of the included studies.

1 2 3 4 5

8. Describes an approach for

synthesizing the studies. 1 2 3 4 5

(26)

9. Provides details for communicating key messages from the review to appropriate audiences.

1 2 3 4 5

Introduction to Systematic Review Methods Course Evaluation: Dr. Jenny Yost

Please complete and return this form to Jenny Yost. Thank you for taking the time to provide feedback!

Please rate the course:

Poor Excellent

OVERALL rating of course. . . . 1 2 3 4 5 6 7

Organization . . . 1 2 3 4 5 6 7 Relevance and usefulness of the

information . . . 1 2 3 4 5 6 7

How your needs and expectations were met 1 2 3 4 5 6 7

What were the BEST FEATURES of this course?

What features of this course do you recommend CHANGING?

Please rate this Presenter (Jenny Yost):

Poor Excellent

OVERALL rating of presenter . . . 1 2 3 4 5 6 7

Organization and preparedness. . . 1 2 3 4 5 6 7

Expertise in the topic area . . . 1 2 3 4 5 6 7

Enthusiasm for presenting the subject. . . 1 2 3 4 5 6 7

Clarity of communication . . . 1 2 3 4 5 6 7

Facilitation of discussion. . . ……….. 1 2 3 4 5 6 7

Additional comments:

26