Building the Evidence Base for Medical Devices:Strategies and Pitfalls 4

(1)

4 Building the Evidence Base for Medical Devices:

Strategies and Pitfalls

R. Taylor

world. However, many European countries (e. g. UK, Sweden, Germany and the Netherlands) for a medical device to be listed or reimbursed by a healthcare payer have, in recent times, introduced the additional evidence hurdles of clinical effectiveness (i. e. what are the benefits of a medical device over and above current therapy?) and cost effectiveness (i. e. are these health benefits worth the additional cost of the medical device [2]?).

Superimposed on these evidential requirements, is the recognition for the hierarchy of evidence. Ac- cording to the evidence hierarchy, the randomized controlled trial (RCT) is regarded as the highest level of evidence for judging the effectiveness of therapeu- tic interventions [3]. For example, evidence collected within two or more well conducted randomized-controlled trials would be regarded as “level I++ evidence” and as a result would receive a “grade A” policy recommendation. In contrast, evidence from an un- controlled case series represents “level-IV evidence”

and would receive a “grade D” policy recommendation. One frequently used example of such an evidence hierarchy [4] is shown in ⊡ Table 4.1.

The Pitfalls of Building Evidence for Medical Devices

A number of characteristics of medical devices challenge the conduct of the classic double-blind random- Why the Increasing Need for Evidence?

There is an increasing global trend for healthcare policy makers to use evidence to assist their popula- tion level decisions, so-called evidence-based policy.

Such policy decisions can be at the level of hospital, primary care trust, a region or a whole country [1]. In the current climate of rising healthcare costs, many healthcare providers and payers also wish to include not only consideration of evidence of clinical out- comes but also the costs of medical technologies, such as new medical devices or drugs. Evidence is therefore increasingly seen as a tool to assist health care policy makers in their efforts to contain costs.

This paper discusses a number of issues related to the evidence for medical devices and clinical related procedures: the changing face of evidence requirements of healthcare policy makers; the challenges of generating evidence for medical devices and procedures: and finally, some suggestions for how medical device evidence should be collected in the future.

What Do We Mean by “Evidence”?

For many years there has been a requirement for ef- ficacy (i. e. what are the benefits of a medical device compared to no therapy or placebo) and safety data for a medical device for licensing and obtaining a CE mark in Europe or equivalent in other parts of the

(2)

20 Part I · Development in Neuroscience: Pain and Dystonia

4

ized controlled trial, often undertaken for pharma- ceuticals. Some of these key differences between medical devices and drugs that underpin these challenges are summarized in ⊡ Table 4.2 below. The dif- ficulty in undertaking medical device and clinical procedure trials has been comprehensively reviewed elsewhere [5, 6].

Nevertheless, as will be argued below, many of these difficulties of medical device clinical trial design can be overcome (at least in part) by innovative trial methodology and design.

Strategies for Building an Evidence Base

A case example is used to illustrate the strategies of building an evidence base: spinal cord stimulation (SCS) for patients with failed backed surgery syndrome/chronic leg and back pain (FBSS/CLBP).

Knowing Your Evidence Base

The first component of the strategy for evidence building is to thoroughly know the evidence base for the medical device. Systematic review and meta- analysis are recognized methods to comprehensively and explicitly assess the evidence base for a given

⊡ Table4.1.Levelsofevidenceandgradesofpolicy-recommendations(adaptedfrom[4]) e

c n e d i v e f o s l e v e L +

+

I Highqualitymeta-analyses,systematicreviewsofRCTs,orRCTswithaverylowriskofbias +

I Well-conductedmeta-analyses,systematicreviewsofRCTs,orRCTswithalowriskofbias I Meta-analyses,systematicreviewsorRCTs,orRCTswithahighriskofbias

+ + I

I Highqualitysystematicreviewsofcase-controlorcohortstudiesorhighqualitycase-controlor e h t t a h t y t i l i b a b o r p h g i h a d n a e c n a h c r o , s a i b , g n i d n u o f n o c f o k s i r w o l y r e v a h t i w s e i d u t s t r o h o c

l a s u a c s i p i h s n o i t a l e r +

I

I Wellconductedcase-controlorcohortstudieswithalowriskofconfounding,bias,orchanceanda l

a s u a c s i p i h s n o i t a l e r e h t t a h t y t i l i b a b o r p e t a r e d o m I

I Case-controlorcohortstudieswithahighriskofconfounding,bias,orchanceandasignificantrisk l

a s u a c t o n s i p i h s n o i t a l e r e h t t a h t I

I

I Non-analyticstudies,e.g.casereports,caseseries V

I Expertopinion

s n o i t a d n e m m o c e r f o s e d a r G

A Atleastonemeta-analysis,systematicreview,orRCTratedasI++anddirectlyapplicabletothe f o y l l a p i c n i r p g n i t s i s n o c e c n e d i v e f o y d o b a r o s T C R f o w e i v e r c i t a m e t s y s a r o n o i t a l u p o p t e g r a t

l l a r e v o g n i t a r t s n o m e d d n a n o i t a l u p o p t e g r a t e h t o t e l b a c i l p p a y l t c e r i d + I s a d e t a r s e i d u t s

s t l u s e r f o y c n e t s i s n o c

B AbodyofevidenceincludingstudiesratedasII++directlyapplicabletothetargetpopulation s a d e t a r s e i d u t s m o r f e c n e d i v e d e t a l o p a r t x e r o s t l u s e r f o y c n e t s i s n o c l l a r e v o g n i t a r t s n o m e d d n a

+ I r o + + I

C AbodyofevidenceincludingstudiesratedasII+directlyapplicabletothetargetpopulationand + + I I s a d e t a r s e i d u t s m o r f e c n e d i v e d e t a l o p a r t x e r o s t l u s e r f o y c n e t s i s n o c l l a r e v o g n i t a r t s n o m e d

D EvidencelevelIIIorIVorextrapolatedevidencefromstudiesratedasII+

⊡ Table4.2.Differencesbetweenmedicaldevices s

g u r d d n a

s g u r

D Devices

d n u o p m o c g n i g n a h c n

U Constantlyevolving e

s a e r c n i s n o i t a c il p m o C

e s u h t i w

e s a e r c e d s n o i t a c il p m o C

e s u h t i w o t d e t a l e r n u s t l u s e R

ll i k s n a i c i s y h p

h t i w y r a v s t l u s e R

r o t a r e p o e l b a li a v a y ll a u s u o b e c a l

P Lessplacebo

e r a r r e v o s s o r

C Crossovercommon

(3)

Building the Evidence Base for Medical Devices ²¹

4

⊡ Table4.3.Challengesofmedicaldevicetrials c

i t s i r e t c a r a h

C Challenge Trialdesignsolutions PROCESStrialfeatures s

e u s s i c if i c e p s - e c i v e D

n o e g r u s / n a i c i n i l C

e c i v e d r o f e c n e r e f e r p

n o i t a z i m o d n a

R 1.Offerrandomizationbefore .

1referraltodevicecenter e t a i r p o r p p a y l l a c i n i l c t c e l e S . 2 .

1comparator

- r o p p o s t n e i t a p l l a r e f f O . 3 .

1tunitytoreceivedevice

o t d e z i m o d n a r s t n e i t a p S C S

) l a c i g r u s - n o n ( r o S C S

l a c i d e m l a n o i t n e v n o c

r o f n o i t p o h t i w t n e m e g a n a m

6 t a r e v o s s o r c o t s t n e i t a p

s h t n o m n

e e w t e b n o i t c a r e t n I

/ n a i c i n i l c d n a e c i v e d

n o e g r u s

e v r u c g n i n r a e

L 1.Recruitcenterswith .

1experiencedclinicians d n a e m i t r e v o a t a d t c e l l o C . 2 .

1adjustforlearningcurve n o i t a z i m o d n a r r e t s u l C . 3 .

1(randomizecenters)

h t i w s r e t n e c d e t i u r c e R

s r e t n a l p m i S C S d e c n e i r e p x e

n i e g n a h c l a t n e m e r c n I

e m i t r e v o e c i v e d

l a i r t f o g n i m i

T 1.Headtoheadcomparisons .

1ofdeviceversions - l e v e d e c i v e d t a s s e s s A . 2 .

1opmentmilestones

S C S Y G R E N Y S f o n o i t a u l a v E

m e t s y s

e v i s a v n i - i m e s / e v i s a v n I

e r u d e c o r p

, s t n e i t a p f o g n i d n i l B

d n a s n a i c i n i l c

s r e h c r a e s e r

e r u d e c o r p m a h S . 1

n g i s e d F F O / N O . 2

e m o c t u o t n e d n e p e d n I . 3 .

1assessment

f o t c e f f e a i s e h t s e r a p o t e u D

t o n s a w g n i d n i l b , S C S

e l b i s s o p

s e u s si ci fi c e p s - e ci v e d n o N

n o i t a l u p o p y r o t c a r f e

R Finitesamplesizeand r e w o p l a c i t s i t a t s f o k c a l

t n e m t i u r c e r r e t n e c i t l u M

e m o c t u o y r a m i r p n o l a i r t r e w o P

s s o r c a s t n e i t a p S C S d e t i u r c e R

s r e t n e c l a n o i t a n r e t n i x r

o t a r a p m o c d e t p e c c a o

N Non-standardization Pragmatictrialdesignwhere t s e b

"

l a c o l y l p p a s r e t n e c

"

e r a c

l a c i d e m l a n o i t n e v n o C

d e i l p p a t n e m e g a n a m

f o e c i t c a r p e h t o t g n i d r o c c a

s r e t n e c c

i f i c e p s - y p a r e h T

s e m o c t u o

r o f s e m o c t u o t n a v e l e r r I

g n i k a m y c i l o p

d e s u c o f - t n e i t a p f o n o i t c e l l o C

s e m o c t u o

l a n o i t c n u f d e d u l c n i s e m o c t u O

d e t a l e r - h t l a e h d n a ) y r t s e w s O (

- F S ( s e m o c t u o e f i l f o y t i l a u q

) D 5 - Q E d n a 6 3 therapy [7] A systematic review of SCS for FBSS and

CLBP identified 72 case series, one RCT and one co- hort study [8]. The RCT showed that patients receiv- ing SCS experienced both a significantly higher level of pain relief and lower requirement for opiate anal- gesia compared to re-operation [9]. Using the Har- bour and Miller evidence assessment scale, there is

“level I+” evidence for the use of SCS in FBSS. How- ever, despite level-I evidence, in Europe current management practice for FBSS patients would be more likely to be optimal non-surgical medical care rather than re-operation. Thus the policy recommendation was “grade B”.

A systematic review provides the means by which evidence grading can be undertaken and also evidence gaps can be identified.

Moving up the Evidence Hierarchy

The second component of the strategy of building is to move as high as possible up the evidence hierarchy.

In the case of SCS for FBSS, this required a new RCT to be designed and undertaken that compared SCS with conventional medical management. With the support of one of the device manufacturers (Med- tronic Sarl), in 2000 a group of clinicians experienced in SCS and clinical trial specialists met to consider how best to design such a trial. As a result, the PROCESS (A Prospective, Randomized, Controlled, Multicenter Study to Evaluate the Effectiveness and Cost-Effectiveness of Spinal Cord Stimulation) trial was born [10].

(4)

22 Part I · Development in Neuroscience: Pain and Dystonia

4

PROCESS was designed to both overcome a number of the potential challenges of medical device trials and also explicitly cater for the changing evidence needs of policy makers. The challenges of device trials and some of the solutions are summarized in

⊡ Table 4.3.

Conclusions

There has traditionally been a requirement for effi- cacy and safety evidence for the licensing of medical devices. However, health-policy makers are increasingly expecting data of the “real world” clinical effectiveness and cost effectiveness of medical devices.

Furthermore, such data needs to be collected using a randomized controlled trial design. The collection of such data requires innovative design of medical device clinical trials.

References

1. Muir Gray JA (ed) Evidence-based policy. Churchill Living- stone, Edinburgh, 1997

2. Taylor RS, Drummond MF, Salkeld G, Sullivan SD. Inclusion of cost effectiveness in licensing requirements of new drugs: the fourth hurdle. BMJ 2004; 329: 972–975 3. Sackett DL, Hayes RB, Guyatt GH, Tugwell P (eds) Clinical

epidemiology: A basic science for clinical medicine, 2nd edn. Little, Brown and Company, Boston, 1991

4. Harbour R, Miller J. A new system for grading recommendations in evidence based guidelines. BMJ 2001; 323:

334–336

5. Seibert M et al. On behalf of EUCOMED. Health Technology Assessment for Devices in Europe. Int J Technol Healthcare 2002; 18: 734–740

6. Hartling L et al. Challenges of systematic reviews of clinical devices and procedures. Ann Intern Med 2005; 204:

1100–1111

7. Oxman AD, Cook DJ, Guyatt GH. Users’ guides to the medical literature. VI. How to use an overview. Evidence-Based Medicine Working Group. JAMA 1994; 272: 1367–1371 8. Taylor RS, Taylor RS, Buscher E, Van Buyten J. Systematic

review and meta-analysis of the effectiveness of spinal cord stimulation in the management of failed back surgery syndrome. Spine 2005; 30: 152–160

9. North RB, Kidd DH, Farrokhi F, Piantadosi S. Spinal cord stimulation versus repeated lumbosacral spine surgery for chronic pain: A randomized, controlled trial. Neuro- surgery 2005; 56: 98–107

10. Kumar K, North R, Taylor R, Van den Abeele C, Gehring M et al. Spinal cord stimulation versus conventional medical management: a prospective, randomised, controlled, multi- centre study of patients with Failed Back Surgery Syndrome (PROCESS study). Neuromodulation 2005; 8: 213–218