SYSTEMATIC REVIEW METHODS

(1)

Unit 3

SYSTEMATIC REVIEW

METHODS

(2)

1. Identify the components to address in developing study selection criteria for a systematic review question.

2. Identify the components to address in developing a data extraction tool for a systematic review question.

3. Identify relevant appraisal tools for

conducting a systematic review about the effectiveness of an intervention.

OBJECTIVES: UNIT 3

(3)

 Explicit

Objective

Minimize the risk

Mistakes

Bias

STUDY SELECTION/SCREENING

(4)

STUDY SCREENING PROCESS

1. Screen titles and abstracts

1. Screen full-text text

(5)

INCLUSION/EXCLUSION: EXAMPLE

Intervention Review Question

 Are knowledge translation (KT) interventions

directed to nurses in tertiary care are effective for improving EBP knowledge, skills, behaviours, and, as a result, patient outcomes

(6)

Identify

Study Selection/

Screening Criteria

IN-CLASS ACTIVIT Y

(7)

 Yes

No

Unclear

Not used at Full-Text

SCREENING JUDGEMENTS

(8)

 Who will do the screening?

Multiple publications

Disagreements

Lack of information

PRACTICAL CONSIDERATIONS

(9)

EXAMPLE

(10)

DOCUMENTING SCREENING

 What to document

PRISMA (Flow Diagram)

(11)

(12)

 “Data”

 any information about (or deriving from) a study

 Important data

 Relevant data

DATA EXTRACTION

(13)

 Importance

Data to extract

General information

Study characteristics

Participant characteristics

Intervention/Comparison

Outcome data/results

DATA EXTRACTION FORM

(14)

Reviewer performing data extraction

Date of data extraction

Features of the study

Record number (to uniquely identify study)

Author

Article title

Citation

Type of publication (e.g. journal article, conference abstract)

Source of funding

GENERAL INFORMATION

(15)

Aim/objectives of the study

Duration/dates

Setting/Country/Context

Study design

Study inclusion and exclusion criteria

Recruitment procedures used (e.g. details of how randomisation was performed)

Unit of allocation (e.g. participant, cluster) STUDY CHARACTERISTICS

(16)

 Characteristics of participants

Age

Gender

Ethnicity

Socio-economic status

Disease characteristics

Co-morbidities

PARTICIPANT CHARACTERISTICS

(17)

 Groups (e.g. Intervention/Comparison)

 Number

 Number of participants enrolled in each group

 Description of the intervention(s) and comparison(s)

 dose

 route of administration

 number of cycles

 duration

 provider

 how the intervention was developed

 theoretical basis

 co-interventions

INTERVENTION/COMPARISON

(18)

 TiDIER Checklist

Hoffman et al. (2014). Better reporting of interventions: template for

intervention description and replication (TIDieR) checklist and guide. BMJ.

http://www.bmj.com/content/348/bmj.

g1687.long

INTERVENTION/COMPARISON

(19)

(20)

DATA EXTRACTION: EXAMPLE

(21)

Unit of assessment/analysis

Statistical techniques used

For each pre-specified outcome:

Whether reported

Definition used in study

Measurement tool or method used

Unit of measurement (if appropriate)

Length of follow-up, number and/or times of follow-up measurements

OUTCOME DATA/RESULTS

(22)

 For all groups

 Number of participants included in analysis

Number of withdrawals, exclusions,

lost to follow-up

(23)

Summary outcome data

 Dichotomous: number of events, number of participants

 Continuous: mean and standard deviation

Results of study analysis

 Treatment Effect

 Dichotomous: odds ratio, risk ratio, absolute risk difference

 Continuous: mean difference, effect size

 Statistical significance

 P value

 Confidence interval

(24)

DICHOTOMOUS OUTCOMES

Cochrane Handbook

(25)

What are the effects of mobile phone messaging reminders for attendance at healthcare

appointments?

Intervention Study: Chen et al., 2008

DATA EXTRACTION: PRACTICE

(26)

mean value of the outcome measurements in each group

standard deviation of the outcome measurements in each group

number of participants on whom the outcome was measured in each group

CONTINUOUS OUTCOMES

(27)

DATA EXTRACTION: EXAMPLE

(28)

 Who will extract data?

Managing the data

Multiple publications

 What to document?

PRACTICAL CONSIDERATIONS

(29)

RISK OF BIAS ASSESSMENT

Purpose

Assess the validity of the included studies, the extent to which results are true

Importance

Flaws in the design/conduct of a study can

result in bias, and in some cases influence the effects of the intervention

Assists in determining the confidence in the results of the systematic review

(30)

 Numerous tools exist

Different tools may produce different assessment

Choose based on aspects of study’s design

TOOLS

(31)

RCT Non-Randomized Trial

CASP √

CEBM √

Joanna Briggs Institute √ √

SIGN √

Cochrane ROB √

JADAD √

ROBINS-I √

EPHPP √ √

(32)

REPORTING

Provide judgement

for each of the criteria

for each study

Within text

describe the numbers of study that meet each criteria

describe the number(s) of studies that do not meet each criteria

(33)

Select a

Risk of Bias Tool

IN-CLASS ACTIVIT Y

(34)

Practice

Risk of Bias Assessment

What are the effects of mobile phone messaging reminders for attendance at healthcare

appointments?

Intervention Study: Chen et al., 2008

IN-CLASS ACTIVIT Y

(35)

OTHER BIAS - EXAMPLES

 L ow r isk

 no industry funding or statement that industry funders had no involvement other than funding

 baseline characteristics are similar across groups

 study is sufficiently powered (sample size calculation met) or each arm has ≥30 participants

 if applicable, the analysis has accounted for clustering

 H ig h r isk ( se le ct if a ny o f t h e fo llowing c onditions a r e p r e se nt):

 industry funding and involvement in some aspect of design, collection, interpretation

 extreme baseline imbalance (significant differences in outcome between study groups) without attempt to adjust for imbalance in analysis

 unit of analysis issues are not addressed or there is a failure to account for clustering in the analysis

 the study is not sufficiently powered or there is no mention of sample size calculations or power analyses; one or more study arms has < 30 participants

 possibility for co-intervention

 possibility for contamination

 U n cle ar r isk:

 industry funding with no explicit details about other involvement

 insufficient information to assess whether important risk exists or insufficient rationale/evidence that identified problem will introduce bias

(36)

RISK OF BIAS: EXAMPLE

(37)

 Quantitative Studies

Cochrane Risk of Bias

Qualitative Studies

JBI Qualitative Assessment and Review Instrument