SYSTEMATIC REVIEW METHODS

(1)

Unit 4

SYSTEMATIC REVIEW

METHODS

(2)

1. Identify and determine appropriate methods for data synthesis

2. Differentiate between clinical and statistical heterogeneity

3. Identify the components assessed in applying GRADE to systematic reviews 4. Identify strategies for communicating

the results of a systematic review

OBJECTIVES: UNIT 4

(3)

 Descriptive summary

Synthesis

 Narrative synthesis

Meta-analysis

SYNTHESIS

(4)

 Clinical heterogeneity

 Population

Intervention

Outcome

Statistical heterogeneity

 Other

SYNTHESIS DECISION

(5)

 Degree to which there is variation between studies beyond chance

Informs

synthesis decision

statistical model for meta-analysis

Tests

χ ² (chi-squared)

I ²

STATISTICAL HETEROGENEIT Y

(6)

Practice

Heterogeneity Assessment

IN-CLASS ACTIVIT Y

(7)

 Intervention Review Question

 What are the effects of mobile phone messaging reminders for attendance at healthcare

appointments?

 Clinical Heterogeneity

 p. 10-11

 p. 25-35

 Statistical Heterogeneity

 p. 13-15

IN-CLASS ACTIVIT Y: HETEROGENEIT Y

(8)

 Avoid reporting study by study

Direction, size, and consistency of the effect

Overall assessment of the strength of the evidence

NARRATIVE SYNTHESIS

(9)

1. Summary statistic to describe intervention effect is calculated for each study

 Description of effect

 Dichotomous

 Continuous

2. Summary (pooled) intervention effect

estimate across studies is calculated as a weighted average of the intervention effects estimated in the individual studies

 Statistical method

 Fixed effects

 Random effects

META - ANALYSIS

(10)

 Dichotomous

1. Consistency

2. Ease of understanding 3. Other

 Continuous

Measurement of outcome

DESCRIPTION OF EFFECT

(11)

Fixed-Effects Random-Effects

Conceptual Considerations

• Assumes effects are the same in all studies

• e.g. Among

these X studies and no others, what is the

impact of the intervention(s) on the outcome?

• Assumes effects differ across studies and the pooled estimate is the mean effect

• e.g. Among all studies, of which these X studies are a random sample, what is the impact of the

intervention(s) on the outcome?

Statistical

Considerations

• within-study variance

• within-study and between- study variances

Practical

Considerations

• Narrow CI

• Large studies

have much more weight than

small studies

• Wider CI

• Large studies have more weight than small studies, but the gradient is smaller than in fixed-effects model

Guyatt, G., Rennie, D., Mead, M. O., Cook, D. J. Users' Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice, 3rd ed.

(12)

 Different approaches, no one knows the best

approach and (fortunately, rarely) the method can yield noticeable differences in results [ M u r a d et a l . ( 2 01 5 ) Us er s G ui d e to t h e M ed i c a l L i ter at u r e 3

^{r d}

e d . , p . 5 0 9 ]

 Rationale for Fixed

 One study is much larger or more trustworthy than one or more smaller studies and yield quite different results

 Number of studies is very small (< 5)

 Rationale for Random

 Uncertainty about the accuracy and applicability of a particular point estimate increasing with increasing variability in results across studies

 Interested in not just available studies, but applying them to a wider population

RANDOM VS FIXED MODELS

(13)

(14)

Dichotomous

 Fixed

 Inverse variance

 If studies are small or low event rates, consider Mantel Hanzel/Peto

 Random

 DerSimonian and Laird is the most common

 Mantel Hanzel

Continuous

 Inverse-variance (either fixed or random)

METHODS

(15)

Purpose

 explore the certainty of pooled results by repeating the analyses having made some changes to the data or methods

 if pooled results are not affected by these changes -> higher degree of certainty about the pooled results

Examples

 study design

 statistical methods

SENSITIVIT Y ANALYSIS

(16)

 Differ from sensitivity analysis

Produce “new” estimates of effect

Identify in advance, with rationale

 Require an adequate number of studies

Magnitude of the difference

SUBGROUP ANALYSIS

(17)

 Studies with positive effects have a better chance of being published

Always a risk in systematic reviews

Funnel Plot

plot of the intervention effect estimates from individual studies against some

measure of each study’s size or precision

PUBLICATION BIAS

(18)

Center for Reviews and Dissemination.(2008). Systematic reviews: CRD’s guidance for

undertaking reviews in healthcare . https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf

(19)

System for rating the quality of a body of evidence in systematic reviews and

guidelines

Different “end” points

 Systematic reviews “end” after rating the

quality of evidence for outcomes and clearly presenting the results in evidence tables

 Guidelines continue to formulating recommendations

GRADE

(20)

 Risk of bias/Study limitations

Inconsistency

Indirectness

Imprecision

 Publication bias

FACTORS FOR DETERMINING

THE QUALIT Y OF EVIDENCE

(21)

Focus is on the outcome, not the individual study

Study limitations for RCTs

 Lack of allocation concealment

 Lack of blinding

 Incomplete accounting for patients and outcome events

 Selective outcome reporting bias

 Other bias

RISK OF BIAS/STUDY LIMITATIONS

(22)

Risk of bias

Across studies

Interpretation Considerations GRADE

Assessment of Study

Limitations

Low risk of bias.

Most

information is from studies at low risk of

bias.

Plausible bias unlikely to

seriously alter the results.

No apparent limitations. No serious

limitations, do not downgrade

Unclear risk of bias.

Most

information is from studies at low or unclear risk of bias.

Plausible bias that raises some doubt about the results.

Potential limitations are

unlikely to lower confidence in the estimate of effect.

No serious

limitations, do not downgrade.

Potential limitations are likely to lower confidence in the estimate of effect

Serious limitations, downgrade one level.

High risk of bias.

The proportion of information from studies at high risk of bias is

sufficient to affect the interpretation of results.

Plausible bias that seriously weakens confidence in the results.

Crucial limitation for one criterion, or some limitations for multiple criteria, sufficient to lower confidence in the estimate of effect.

Serious limitations, downgrade one level

Crucial limitation for one or more criteria sufficient to

substantially lower confidence in the estimate of effect

Very serious limitations, downgrade two levels

Cochrane Handbook

(23)

 Unexplained heterogeneity of results

 Never rate “up”

 Judge related to relative measures of effect

 Consider rating down if

 Point estimates vary widely across studies

 Confidence intervals (CIs) show minimal or no overlap;

 Statistical test for heterogeneity

 low P-value

 Large I

²

INCONSISTENCY

(24)

Guyatt et al., (2011).

Journal of Clinical Epidemiology, 64, 1294-1302.

(25)

 When the population, intervention, or

outcomes differ from those in which we are interested

 Consider rating down if there is indirectness related to

 population

 intervention

 outcome

 comparison

INDIRECTNESS

(26)

Table 5.7: Common surrogate measures and corresponding patient-important outcomes

[GRADE Handbook: http://gdt.guidelinedevelopment.org/app/handbook/handbook.html#h.m9385o5z3li7 ] Condition Patient-important outcome(s) Surrogate outcome(s)

Diabetes mellitus Diabetic symptoms, hospital admission, complications (cardiovascular, eye, renal, neuropathic)

Blood glucose, A1C

Hypertension Cardiovascular death,

myocardial infarction, stroke

Blood pressure Dementia Patient function, behavior,

caregiver burden

Cognitive function

Osteoporosis Fractures Bone density

Adult Respiratory Distress Syndrome

Mortality Oxygenation

End-stage renal disease Quality of life, morbidity (such as shunt thrombosis or heart failure), mortality

Hemoglobin

Venous thrombosis Symptomatic venous thrombosis Asymptomatic venous thrombosis Chronic respiratory

disease

Quality of life, exacerbations, mortality

Pulmonary function, exercise capacity Cardiovascular disease Myocardial infarction, vascular

events, mortality

Serum lipids, coronary calcification,

calcium/phosphate metabolism

(27)

 Different recommendations for guidelines and systematic reviews

 Optimal information size (OIS) criterion

 the total number of patients included in a systematic review is greater than the number of patients generated by

a conventional sample size calculation for a single adequately powered trial

 Do not rate down if

 if OIS is met and the 95% CI excludes no effect

 if the total sample size is very large, at least 2000, and perhaps 4000 patients

 Consider rating down if

 OIS is not met

 OIS is met and the 95% CI overlaps no effect

IMPRECISION

(28)

 Extent to there is uncertainty about the magnitude of the effect due to selective publication of studies

 Rate down by maximum of one level

 Consider rating down

 studies are observational in design

 discrepancies in the results of small studies and large studies

 unpublished studies obtained by the authors show different results than published studies

 early publication of positive results (lag bias)

 asymmetrical funnel plot

PUBLICATION BIAS

(29)

 Evidence profile

Summary of findings table

(30)

Applying GRADE

IN-CLASS ACTIVIT Y

(31)

 PRISMA checklist

 http://prisma-statement.org/

 Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A, Clarke, M.,

Devereaux, P. J., Kleijnen, J., & Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies to evaluate health care interventions: Explanation and elaboration. PLoS

Medicine, 6(7). Retrieved from

http://journals.plos.org/plosmedicine/article?id=10.

1371/journal.pmed.1000100

SYSTEMATIC REVIEW REPORTING

(32)

 Knowledge User

 Anyone who is likely to use research findings to make

informed decisions about practices, programs, and/or policies

 Integrated Knowledge Translation (iKT)

 Engaging the knowledge user audience(s) throughout the research process

 End-of-Grant Knowledge Translation

 How to translate findings to knowledge-user audience(s) when the research is completed

 Goals

 related to raising awareness/interest and promoting action.

 should be appropriate to the nature of the research findings and the target audience

KNOWLEDGE TRANSLATION

(33)

 What is the message

 e.g. research findings

 To whom

 e.g. the audience

 By whom

 e.g. the messenger

 How

 e.g. the strategy/method

 With what expected impact

 e.g. the knowledge goal(s)

R e a r d o n , R . , L a v i s , J . , & G i b s o n , J . ( 2 0 0 6 ) . I n s t i t u t e f o r Wo r k a n d H e a l t h : Fr o m r e s e a r c h t o p r a c t i c e : A k n o w l e d g e t r a n s f e r p l a n n i n g g u i d e R e t r i e v e d f r o m

h t t p : / / w w w. i w h . o n . c a / s y s te m / f i l e s / d o c u m e n t s / k te _ p l a n n i n g _ g u i d e _ 2 0 0 6 . p d f

SYSTEMATIC REVIEW METHODS

Unit 4