thesis Master‘s Final

Final

Master‘s

thesis

March 17

2018

Clinical Decision Support Systems: a

systematic literature review of drug-drug

interaction alerts

2

Table of Contents

Summary ... 3

Acknowledgements ... 4

Conflict of interest ... 4

Abbreviations... 4

Terms ... 5

Introduction ... 5

Aim ... 8

Objectives ... 8

Methods ... 8

Results ... 9

Discussion ... 17

Limitations ... 18

Clinical implications, practical and research recommendations ... 19

Conclusions ... 19

3

Summary

Dr. Artin Hosseinion

Final Master‘s Thesis: Clinical Decision Support Systems: a systematic literature review of

drug-drug interaction alerts

Clinical Decision Support Systems (CDSS) integrate a medical knowledge and patient data to

produce a computerized case specific advice. A type of such advice are drug-drug interaction (DDI)

alerts, expected to reduce inappropriate co-prescription rates and medical errors. However reported

evidence on efficacy of such systems is limited.

The aim: to perform a systematic evaluation of CDSSs on effectiveness to report DDI cases and improve

prescriber/patient outcomes.

Objectives: (1) to analyze principles of DDI alerts and reporting; (2) review quality of CDSS alerts of

DDI; (3) to present the findings; and (4) to provide relevant recommendations.

Methods: an online literature search was conducted on Medline using key words „Clinical Decision

Support Systems“ and „Drug Interaction“. The outcomes of interest were changes in prescription, patient

outcomes and data on specificity/sensitivity/positive predictive value (PPV) of DDI alerts. Exclusion

criteria were no full text available, other (non-CDSS) alert systems, non-prescription interventions, no

comparator used for outcome assessment, qualitative studies and opinion surveys.

Results: the search returned a total of 490 publications: n=357 in PubMed, n=44 in Cochrane Library,

and n=89 on manual search. Following the review 476 articles were excluded, leaving 14 studies of

interest. The eligible studies were divided into 2 groups: the 1

_{group of studies (n=6) addressed the}

benefits of DDI alerts on prescriber/patient outcomes, and the 2

_{(n=8) group reported PPV or}

sensitivity/specificity values. There was a variation among prescriber outcome measures among the

1st group author teams, and comparable quantitative outcome data was not possible to produce. Half of

the 1

_{group studies reported a statistically significant beneficial effect from DDI alert on prescriber‘s}

behavior (n=3.50%), while the other half of the studies reported none (n=3.50%). One study (16.7%)

evaluated patient outcomes, and a significant benefit was observed. One study (16.7%) was stopped

prematurely due to safety concerns. The reported PPV of DDI alerts was very low reaching up to 10%

in most studies. Advanced CDSS have demonstrated higher PPV compared to the basic vendor software

(14.8% vs 9.9%, p<0.05). Reported DDI alert sensitivity ranged between 9%-96%. Reported specificity

also depended on the software vendor (29-94%), and was higher in advanced CDSS.

4

Conclusions: Current CDSSs do not meet the prescribers’ needs. The evidence on patient outcome is

lacking, and further studies are needed to establish their role in clinical care.

Acknowledgements

I sincerely thank my supervisor Dr. Vita Špečkauskienė for advice and review of the Final

Master‘s Thesis.

Also, I would like to express my gratitude to Professor Viktoras Šaferis for an interesting and educational

topic of my graduate work.

I would like to thank my friends, Julien-Wassim Achard, Simon Radmand, Zeid Al-idani for support.

And last, but not least, I must express profound gratefulness to my dear parents who made my entire

journey through medical studies financially possible.

Conflict of interest

I declare no conflict of interest related to this work.

Abbreviations

BICS -

Brigham Integrated Computer System;

CDSS - Clinical Decision Support Systems;

CPRS - Computerized Patient Record System;

DDI - Drug-drug interaction;

NSAID - Non-steroid anti-inflammatory drugs;

PPV - Positive predictive value;

R - Response;

5

VISTA - Veterans Information System Technology Architecture;

USA - United States of America.

Terms

The Final Master’s Thesis ‘Clinical Decision Support Systems: a systematic literature review

of drug-drug interaction alerts’ was prepared during the period of October, 2017 – April, 2018.

Introduction

Contemporary developments of clinical medicine rely on modern advances in tools and

techniques that support the healthcare system workers in daily practice and decision making. Among

such tools are Clinical Decision Support Systems (CDSS), which are known as computerized knowledge

systems which use integrated patient data to generate a patient-specific advice (1). These systems

integrate a medical knowledge base, patient data and an inference engine to produce a case specific

advice at the point in time the decisions are made. Introduction of such systems has been shown to

improve patient outcomes and reduce the cost of care (2) (3). They are comparable to implication of

artificial intelligence into medical field with potential to revolutionize the future healthcare, medical

teaching and practice. However to this date their use and applications are still limited (4) (5). These

systems are capable of performing the following functions (6):

• Administrative: support of clinical coding and documentation, procedure/referral authorization.

• Management: storing clinical details (e.g. research/therapeutic protocols, tracking

referrals/follow-ups, preventive care).

• Control: monitoring medication orders, avoiding duplicate or unnecessary tests (cost saving

measures).

• Decision support: establishment of clinical diagnosis/ treatment plan, promoting best clinical

practice or condition-specific guidelines, population-based management.

Medication consumption is constantly increasing with numerous multi-drug treatment prescriptions

dispensed on a daily basis, therefore possibility of drug-drug interaction (DDI) is one of the most

clinically relevant issues encountered in practice. A DDI is defined as a clinically significant

physiological alteration in the exposure and/or response to a drug that has occurred as a result of

co-administration of another drug (7). Exposure to potential DDI can cause preventable drug-related harm

and is a serious medical error (8). To aid prescription process CDSS may include alerts, reminders, order

6

sets and drug dose calculations. DDI alerts are of particular interest as their nature is very suitable for

highly efficient automated interventions performed by computerized CDSS (9). Many electronic

prescription and medication information systems include some form of CDSS alerting regarding

drug

dosing, duplicate therapy, and

potential DDI. DDI alerts most commonly appear during the prescription

medication order entry or at dispensing/verification process at the pharmacy. CDSS-assisted DDI

checking is estimated to yield a return on time investment of almost 10 years (2).

Principles of DDI alerts and reporting in CDSS. The DDI database integrated into the CDSS is usually

linked to a prescription system of a healthcare institution or pharmacy (6). It provides automatic alerts

during drug prescription process in cases of potentially dangerous or contraindicated drug combinations.

For optimal and advanced use CDSS should be implemented into a local/national database, which allows

access to patient specific data such as age, sex, height, weight, blood test results, and the medication a

patient is already using (10). Through certain software algorithms an alert could then be triggered or not,

providing a relevant patient specific warning rather than alerting on each potential DDI by default. The

types of DDI alerts range from non-intrusive reminders to mandatory “hard-stop” prescription warnings

(the prescriber is required to abandon the prescription or prescribe an alternative drug) (11). Some

providers use color coding for the alert messages to reflect the urgency of the warning by the CDSS: e.g.

red – very important information (avoid combination), yellow – important information to consider (dose

adjustment recommended), white – information only with no action required (12). Higher alert

acceptance and user compliance rates are observed when only high severity DDI alerts are interruptive

in nature, otherwise they only increase tendency to ignore, work around or override them (11) (13).

Certain providers offer an on-demand rather than computer triggered DDI alerting mode as a solution to

reduce existing computerized prescribing problems, however in effectiveness such systems have not

shown any advantages (14) (15). Prescribers response to DDI alerts may be of passive nature including

situations where DDI alerts can be completely missed or ignored (message pop-up on prescription

screen), or active – acknowledgement (a button press to accept a potential DDI), overriding (sometimes

with a request to type in a reason/explanation for combination prescription) or „hard stop“ prescription

orders (abandon prescription/prescribe alternative) (11). Although „hard stop“ DDI alerts significantly

reduce inappropriate co-prescription rates, they have been shown to be problematic at times in clinical

practice and are rarely used, except for clinical trials (16).

Quality of CDSS DDIs alerts. There is no agreed standard to define qualitative performance of CDSS.

However, as other computerized software it can be described by objective performance characteristics

and a degree of user satisfaction. There is a substantial variability in DDI alerting performance across

electronic prescribing and pharmacy software systems published in the literature (17) (18) (19) (20) (21).

Based on current publications and reviews, degree of excellence in CDSS performance is very low, and

7

improving the state-of-the art of CDSS faces a few considerable challenges. Alerts of ambiguous or low

clinical relevance are a major limitation of these systems and have caused considerable clinical

frustration and user dissatisfaction with reported alert override rates consistently exceeding 90% (5) (22)

(23) (24) (25). One of the major contributing factor to the latter phenomenon is a lack of guidelines and

approved standards for determining clinical relevance of DDIs, and without such guidance DDIs are

often inappropriately extrapolated to other drugs within the same therapeutic/pharmacologic class (26)

(27). Also, many „possible“ DDIs lack high quality evidence to support their existence, controlled

clinical studies in relevant populations are scarce, as well as random cases are underreported and often

lack information (26) (28) (29). Commercially available DDI databases in an effort to reduce legal

liability, include almost all possible DDIs, including those that confer extremely low risk to exposed

patients leading to excess of irrelevant and non-specific alerts and consequent alert fatigue (30). Multiple

alerts of low importance create disturbances in already burdened healthcare professionals work flow and

may even be source of clinical errors rather than a preventive mean from them (31). Another issue with

DDI alert generation is database inconsistencies among different software providers: certain DDIs are

ranked as high risk by one provider while at the same time are considered low risk by the other CDSS

(32). Research and investigation of DDI in CDSS is usually focused on how CDSS intervention

influences prescribing behavior and patient outcomes. These studies include quasi-experimental designs

(uncontrolled or controlled before-and-after studies, interrupted time series) and randomized controlled

trials (3) (11) (16) (31) (33) (34) (35). Published reviews report that the use of CDSS reduces healthcare

costs and time consumption, improves patient safety, and prescriber practice (2) (36) (37) (38) (39). Yet,

evidence of improved patient outcomes has been limited (36) (39) (40). Quantitative parameters

describing effectiveness and specificity of CDSS DDI alerts are reviewed in more detail along with the

findings of this study in the discussion section below the results. Alternative CDSS research strategies

include qualitative research methods to provide a deeper understanding of subjective aspects of the

interaction among healthcare professionals, patients and the electronic CDSS interface. Generally CDSS

users (physicians) expressed pronounced irritation due to abundance of irrelevant alerts, which often led

to ignore them (4) (25). Despite being aware of the fact that the decision support system contributes to

safer and more effective treatment of patients, prescribers were dissatisfied with the usability of this tool,

as well as pointed out some technical barriers and limitations (36) (41). Based on the current overview,

current CDSS DDI alerting appears not to meet the prescribers’ needs, and its role in improving

healthcare practice and patient outcomes is currently unclear.

8

Aim

The aim of this study is to perform a systematic evaluation of CDSSs on effectiveness to report DDI

cases and improve prescriber/patient outcomes.

Objectives

1. To analyze principles of DDI alerts and reporting.

2. To conduct a literature review regarding quality of CDSS alerts of DDI.

3. To present the evidence-based findings on effectiveness of DDI alerts to improve prescriber and

patient outcomes.

4. To provide relevant practical recommendations regarding CDSS and DDI alerts based on current

scientific data and research.

Methods

The PRISMA guidelines on the conduct and reporting of systematic reviews was applied (42).

Protocol and information sources: A search of studies published between January 1976 and January

2018 addressing CDSS and DDIs was conducted in Medline accessed through PubMed, and The

Cochrane Library. Key words included „Clinical Decision Support Systems“ and „Drug Interaction“

(no quotes used). Expanders applied included „Apply related words“ and „Search within the full text

of the articles“. Besides electronic database literature search, additional potentially relevant studies

were identified via manual searching (references of published articles, related articles suggested by

electronic search engines).

Eligibility criteria and study selection: Only studies published in English were further included in the

analysis. Studies must have evaluated the impact of automatic computer-derived DDI alerts and

notifications at the time of medication prescription. Studies were included if they analyzed the

outcomes from the intervention (DDI alert) by comparing to either a pre-test period and/or a control

group to determine alert effectiveness. The outcomes of interest were changes in prescription

(cancelling order or prescribing another appropriate drug), and/or patient outcomes (clinical evidence

of potential/actual harm or benefits to direct welfare of the patient), and/or data on

specificity/sensitivity or positive predictive value (PPV) of DDI alerts. A two-staged approach was

used for review process. In the first stage all titles and abstracts were screened for relevance. Items not

meeting inclusion criteria and duplicates were removed. In the second stage of review process articles

potentially meeting the inclusion criteria were retrieved and reviewed regarding final inclusion

9

eligibility. Following full-text review final number of studies meeting the inclusion criteria was

identified.

Exclusion criteria: Articles were excluded if studies were only in abstract form with no full text

available, involved other clinical information systems (hospital documentation and information

portals not linked to CDSS), addressed non-prescription interventions (imaging, blood test orders, etc),

if no comparator was used for outcome assessment, outcomes measured were different from

investigated. Qualitative studies and opinion surveys were also excluded.

Data collection: Data about study setting, design, participant characteristics, interventions and

outcomes was obtained from each study by one investigator.

Analysis: A number of eligible studies of interest was eventually identified. Reported outcomes and

endpoints varied among studies. Comparable quantitative data was insufficient for meta-analysis,

therefore the findings were grouped and reported in a narrative approach. Results were categorized as

either statistically significant (beneficial impact on prescription/patient outcomes) or of no statistically

significant impact (include no statistical analysis performed). Effectiveness of CDSS was illustrated

by reported PPV by study authors or calculated manually if unreported but the published data allowed

these calculations.

Results

Literature review on principles of DDI alerts/reporting (Objective 1) and quality assessment

(Objective 2) are presented in the introduction, and further addressed in the discussion sections.

Comprehensive literature search conducted on February 14, 2018 identified a total of 490 published

items: 401 on electronic database search (n=357 in PubMed, and n=44 in Cochrane Library), and 89 on

manual search. Following the first stage screening 409 articles were removed: 58 duplicates, 348

irrelevant titles/abstracts, 2 missing or inaccessible full text and 1 published not in English language. A

total of 81 articles were eligible for the second stage of review. After second stage full text revision 14

studies were selected that are relevant and correspondent to study objectives. The flow chart of the study

selection process is presented below in Figure 1. Detailed results of the literature search are presented

as a supplementary data (Annex 1).

10

Figure 1. A flow chart of the study selection process.

Consequently the eligible studies were divided into 2 groups (Figure 2). The first group of studies (n=6)

aimed to address the benefits of DDI alerts on prescriber/patient outcomes, and the second group (n=8)

reported objective evidence of DDI alert relevance (PPV or sensitivity/specificity values).

11

Figure 2. Overview of the selected studies. DDI – drug-drug interaction.

The studies investigating efficacy and benefits of DDI alerts (Table 1) in clinical setting were published

between the years 2005 and 2012, and predominately originate from the United States of America (n=5,

83.3%) (3) (11) (16) (33) (34). One study (16.7%) was conducted in Europe (Italy) (31). All

investigations were carried out at the hospital in-patient environment. No eligible studies addressed

CDSS derived DDI concerns or effectiveness in outpatient setting. The patient population contained

adult individuals in all reported studies. The targeted CDSS users (prescribers) were either solely

physicians/clinicians (n=2, 33.3%), physicians/nurses (n=1, 16.7%), or physicians/nurses/pharmacists

(n=2, 33.3%). In one study (16.7%) prescribers were not specified. CDSS manufacturer details and other

study characteristics are summarized in Table 1. Half of the studies (n=3, 50%) were of retrospective

design. All authors investigated the effect of the intervention alert on at least one outcome measure of

prescriber‘s behavior. There was a variation among prescriber outcome measures applied by different

author teams, therefore comparable quantitative outcome data was not possible to produce. Three studies

(50%) assessed order cancellation rate in response to alerts, 1 (16.7%) – inappropriate prescribing rate,

1 (16.7%) – alert override rate, and 1 (16.7%) – alert adherence and incidence of potential DDIs (with no

significant statistical calculations provided). Half of the studies reported a statistically significant

beneficial effect from DDI alert on prescriber‘s behavior (including one study demonstrating adverse

consequences with consequent early termination due to prescription delays (16)), while the other half of

the studies reported none. One study did not report neither general DDI alert adherence nor inappropriate

prescription rate, and no statistical analysis was performed (31). However, they have observed an

increase in prescribing errors (number of potential DDIs) in certain clinical areas after DDI alert

implementation, while a positive trend toward reduction of such errors was observed in the others. A

definitive conclusion whether DDI has a proven beneficial impact on prescriber outcomes cannot be

12

drawn due to inconsistent prescriber outcome measures among investigators, and very limited number

of eligible studies identified which already demonstrate diverging results (50:50 benefit/no-benefit

ratio).

14

Only one (16.7%) of 6 studies has additionally evaluated the effect of DDI alerts on patient outcomes,

demonstrating that there is some beneficial effect of alert implementation: a significant reduction of

in-hospital falls related to co-administration of 2 or more psychotropic drugs to geriatric patients was

observed after DDI alert introduction, however other patient outcomes (length of hospitalization or

number of days of altered mental status) remained unchanged (3). One study with a potential to

address bleeding complication rates related to co-prescription of trimethoprim-sulfamethoxazole and

warfarin was stopped prematurely due to safety concerns by “hard-stop” DDI alert delaying or

preventing the patients from required treatment in cases where such co-prescription was necessary

(34). There were no other studies identified to report DDI related patient outcome measures, however

such outcomes were reported with other drug-related CDSS alerts (drug-condition or drug-allergy

interaction), which are irrelevant to this review (36) (38) (39).

There were 8 studies on CDSS reporting data on objective relevance of DDI alerts (PPV or

sensitivity/specificity) (5) (9) (10) (15) (43) (44) (45) (46). Detailed information on these studies are

presented in Table 2. Two (25%) studies originated from the USA, and 6 (75%) from Europe (The

Netherlands and Switzerland). Software vendors were unreported in USA studies. Only 1 study

(12.5%) was of prospective design (10), the remainder (n=7, 87.5%) were based on retrospective data

analysis. The number of alerts analyzed by authors was either not provided or varied between 365

and 16643. Two studies did not provide the PPV, however calculations were possible based on

supplied data. The PPV of DDI alerts was extremely low reaching up to 10% in most studies. Higher

reported PPV values by Silverman et al are applicable for separate drugs only (PPV 71% for

Theophylline and 60% for Quinidine DDI interactions), and did not represent the general trends of

PPV 0-41% for remainder of drugs (45). Advanced CDSS that integrate laboratory and clinical patient

data into DDI alert generation have demonstrated significantly higher PPV compared to the basic

vendor software (14.8% vs 9.9%, p<0.05) (10). Reported DDI alert sensitivity was in a broad range

of 9%-96%. Despite high sensitivity reported by some authors, current CDSS cannot be relied on due

to particularly low PPV (9) (46). The data on DDI alert specificity was unavailable and not provided

by majority of the authors teams (not reported in 6 (75%) out of 8 publications). Reported specificity

was again dependent on the software vendor (29-94%), and was higher in advanced CDSS (43) (46).

Beeler et al made an interesting insight in his investigation: computer triggered DDI check generated

approximately 45 times more alerts than would do a specialist trained pharmacist for the same clinical

scenarios (15). Compared to other available types of CDSS alerts, DDI alerts appear to be the least

relevant ones, while the best PPV (34.1%-73.3%) was reported in drug-lab interactions (10).

15

First author, year

Country

Patient

population

Prescribers/CDSS users

CDSS

Study design

Number of

alerts

PPV

(%)

Sensitivity

(%)

Specificit

y (%)

Silverman, 2004

USA

Pediatric/adult Physicians/ pharmacists

n/a

1

_{Retrospective}

₁₆₆₄₃

_0-71

_n/a

_n/a

Cash, 2009

USA

Pediatric

n/a

n/a

1

_{Retrospective}

_n/a

_1.4

_n/a

_n/a

Van der Sijs, 2010

The Netherlands

n/a

Pharmacists

TPM (2.8.2), Zamicom (2006-1),

Chipsoft (4.8 FP), Centrasys (1.20

SP1), Medicatie/EVS (2.41), Theriak

(3.4.3)

Retrospective

n/a

n/a

38-79

_29-94

Doormaal, 2010

The Netherlands

Adult

Pharmacists

Medicator (iSOFT)

Retrospective

365

12

96

91

Fritz, 2012

Switzerland

Adult

Physicians/Pharmacists

Pharmavista, Micromedex

DrugReax, Artheso TheraOpt

Prospective

743

5.7-8

4

_9.1-87.9

_n/a

Zorina, 2012

Switzerland

n/a

Physicians

CDSS MediQ, ID PHARMA

CHECK

Retrospective

1759/1041

n/a

70.6-72.4

_n/a

Eppenga, 2012

The Netherlands

Adult

Pharmacists

Centrasys (iSOFT), Pharmaps

Medicatiebewaking PLUS

Prospective 2607/2256

3

9.9-14.8

n/a

n/a

Beeler, 2013

Switzerland

n/a

Physicians/Pharmacists

Galdat/hospINDEX

Retrospective

7902

1.6

16

Table 1. Studies reporting data on effectiveness of CDSS DDI alerts. CDSS – clinical decision support system, PPV – positive predictive value,

USA – United States of America, n/a – data not available (not provided by authors).

1

_{Computerized prescriber order entry integrated in institutional computer system with CDSS, software vendor not provided by authors;}

_{Depends on specific drug;}

3

_{Depends on software vendor;}

4,5

_{Positive predictive value calculated for the review and defined as the quotient of the number of advice/interventions to prevent a possible adverse}

17

Discussion

DDI alerts are reported to be both – the most prevalent and the most frequently overridden

or ignored drug safety alerts by CDSS users (47).

While CDSS DDI alerts have a potential and are

designed to decrease prescribing errors, we found no significant evidence to support beneficial effect

of this type of alerts on patient outcome measures, which is one of the most important findings in this

study. Similar conclusions were drawn by other reviewers’ teams (40) (48). Such absence of patient

outcome evidence related to DDI alerts over decades of CDSS research may be explained by difficulty

to investigate and attribute the outcome of interest to the effect of potential/definite DDI: e.g.

incidence of geriatric patient falls may be related to psychotropic drug prescription but also may be a

result of multiple environmental and other factors. Such studies of DDI reporting although not

impossible to conduct, appear to be very time consuming, lack reliable objective measures, as well

as include a number of potential confounding factors, and are very likely to yield negative results

which would be difficult to publish. Drug-condition or drug-allergy interactions are therefore more

favorable for patient outcome research (e.g. QT interval prolongation on electrocardiogram as an

effect of QT interval prolonging drug prescription, anaphylactic reaction due to medication, etc.).

The current evidence base does not show a clear indication that DDI alerts are indeed effective on

modifying prescribers’ behavior and avoiding prescription errors. While some studies have shown to

improve these outcomes, there were a couple which reported none, and one study has even presented

contradictory findings with potential for increase of medical errors in some clinical areas (31).

PPV is known to be a gold standard for all diagnostic tools in medicine, and is a desirable, if not

mandatory, scientific finding. However, few authors report PPV of CDSS DDI alerts or enough data

for its computation, which is an unexpected finding. Reported PPVs are below or near 40%, which is

of low statistical value. These findings are in keeping with other authors’ data presented in their CDSS

review publications (41) (49). While computer-based CDSS appear as a potentially highly effective

method to prevent incidence of deleterious effects of DDI, low PPV of DDI alerts retains CDSS from

being a reliable tool in daily practice. The broad range of DDI alert sensitivity is probably attributable

to the differing level of DDI significance employed by different vendors and applied in different

institutions: increasing the types and numbers of alerts likely increases the false positive rates of the

overall and individual alerts (5). In real life it is difficult to balance DDI alerts: in many in-patient

hospital settings many medications known to cause significant DDIs are often used together on a

routine basis (e.g. sedatives and anesthetics in critical care units and operation theatres). The most

frequent reasons for DDI alert override given in the literature are as follows: (a) no clinical importance

of alert, (b) no alternative prescriptions available and benefits outweigh the risks, (c) potential DDI

18

can be managed by appropriate monitoring; (d) dose adjustment available, (e) the patient had

previously tolerated the medications (9) (13) (50). Abundance of irrelevant alerts often lead to

subsequent “alert fatigue” when important alerts are being ignored along with unimportant ones. Most

CDSSs allow to turn off specific DDI warnings depending on interaction severity (high/average/low

risk) thus reducing the number of false positive alerts and firing an alert only when the severity of an

interaction is high (13). However, lowering sensitivity of DDI alerts may compromise patient safety.

Thus balancing user needs and patient demands is always a conflicting situation: increasing sensitivity

results in increase of inappropriate alerts and decreasing specificity (43) (51). Having studies

comparing CDSSs with respect to drug safety alerting could identify ‚best choice CDSS software

vendors with highest sensitivity and specificity rates. However such studies are scarce and limited to

single-country and usually single-center surveys (9) (10) (43) (44). A factor that contributes to a better

“advanced” CDSS performance appears integration of laboratory and clinical patient data. CDSS

installed and functioning as part of internal hospital systems appears to be superior compared to

isolated CDSS firing DDI alerts by default (9) (10) (43). A “smart” CDSS‘s algorithm should

generally rely on literature-based evidence and practice-based experience, which means it should

always take the clinical scenario into account before reporting DDI alerts: e.g. most frequent DDI

alert of risk of hyperkalemia related to treatment of hypokalemia by potassium sparing diuretics and

potassium supplement would be suppressed in such system (52). Implementation of such concepts

appear a promising strategy in improving specificity of computer-triggered DDI alerts. The

knowledge database of a CDSS should be continuously updated by evaluating what effects medication

alerts have in daily practice (10).

No significant improvement or benefits to clinical patient outcomes have been demonstrated by this

study. On the contrary, current literature review has shown that vast majority of CDSSs used in

clinical practice lack these properties altogether.

Limitations

Technical: possibility of exclusion of relevant papers due to the search terms (terminology used to

describe CDSS and prescription alerts differs among systems, countries and journals, as later turned

out in the search process). This limitation was partially corrected by manual searching for relevant

publications;

Publication bias and selective reporting of study findings cannot be excluded;

The studies of interest were heterogeneous in interventions, populations, and outcome measures;

Small sample size of eligible studies;

19

Many studies were not randomized;

DDI alerts appear to lack correlation among different vendors: e.g. a combination of sulpiride and

haloperidol is rated as “high danger” by one CDSS provider and only “low risk” by the other (44),

alert design was not uniform among studies.

Clinical implications, practical and research recommendations

1. Improvement in CDSS functionality to balance DDI alert appropriateness is highly desirable.

2. Sensitivity and/or PPV of DDI alerts should be systematically reported in CDSS research.

3. Studies assessing effects of CDSS on patient outcomes are needed.

Conclusions

Current CDSSs neither meet the prescribers’ needs, nor are completely acceptable for globalized

implementation. The evidence on patient outcome is lacking, and further studies are needed to

establish their role in clinical care.

20

References

1. Wyatt J, Spiegelhalter D. Field trials of medical decision-aids: potential problems and solutions.

Proceedings of the Annual Symposium on Computer Application in Medical Care. 1991:3-7.

2. Helmons PJ, Suijkerbuijk BO, Nannan Panday PV, Kosterink JG. Drug-drug interaction checking

assisted by clinical decision support: a return on investment analysis. J Am Med Inform Assoc.

2015 Jul and 22(4):764-72.

3. Peterson JF, Kuperman GJ, Shek C, Patel M, Avorn J, Bates DW. Guided prescription of

psychotropic medications for geriatric inpatients. Arch Intern Med. 2005 Apr 11 and 165(7):802-7.

4. Horn JR, Gumpper KF, Hardy JC, McDonnell PJ, Phansalkar S, Reilly C. Clinical decision

support for drug-drug interactions: improvement needed. Am J Health Syst Pharm. 2013 May 15.

5. Cash JJ. Alert fatigue. Am J Health Syst Pharm., 2009, 66(23):2098-101.

6. Perreault L, Metzger J. A pragmatic framework for understanding clinical decision support.

Journal of Healthcare Information Management. 1999 and 13(2):5-21.

7. Oates JA. Chapter 5. The science of drug therapy. In: Brunton LL, editor. Goodman & Gilman’s

The Pharmacological Basis of Therapeutics. 11. McGraw-Hill, Medical Publishing Division and

2006.

8. Aspden P, Institute of Medicine (U.S.). Committee on Identifying and Preventing Medication

Errors. Preventing medication errors. Washington, DC: National Academies Press and 2007.

9. Fritz D, Ceschi A, Curkovic I, Huber M, Egbring M, Kullak-Ublick GA, Russmann S.

Comparative evaluation of three clinical decision support systems: prospective screening for

medication errors in 100 medical inpatients. Eur J Clin Pharmacol. 2012 Aug;68(8):1209-19.

10. Eppenga WL, Derijks HJ, Conemans JM, Hermens WA, Wensing M, De Smet PA. Comparison

of a basic and an advanced pharmacotherapy-related clinical decision support system in a hospital

care setting in the Netherlands. J Am Med Inform Assoc. 2012 Jan-Feb;19(1):66-71.

11. Paterno MD, Maviglia SM, Gorman PN, et al. Tiering drug-drug interaction alerts by severity

increases compliance rates. J Am Med Inform Assoc. 2009 Jan-Feb and 16(1):40-6.

12. Böttiger Y., Laine K., Andersson M. L., et al. 2009 SFINX- a drug-drug interaction database

designed for clinical decision support systems. Eur J Clin Pharmacol., 65., 6., 627-633 .

21

13. Shah NR, Seger AC, Seger DL et al. Improving acceptance of computerized prescribing alerts

in ambulatory care. Am Med Inform Assoc. 2006 and 13:5-11.

14. R. Tamblyn, A. Huang, L. Taylor, et al. 2008 A randomized trial of the effectiveness of

on-demand versus computer-triggered drug decision support in primary care. J Am Med Inform Assoc.,

15., 430-438.

15. Beeler PE, Eschmann E, Rosen C, Blaser J. Use of an on-demand drug-drug interaction checker

by prescribers and consultants: A retrospective analysis in a Swiss teaching hospital. Drug Saf.

2013 Jun;36(6):427-34.

16. Strom BL, Schinnar R, Aberra F, et al. Unintended effects of a computerized physician order

entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial. Arch.

Intern. Med., 170 (September (17)) (2010), pp. 1578-1583.

17. Jankel CA, Martin BC. Evaluation of six computerized drug interaction screening programs.

Am J Hosp Pharm. 1992 and 49(6):1430–1435.

18. Hazlet TK, Lee TA, Hansten PD, Horn JR. Performance of community pharmacy drug

interaction software. J Am Pharm Assoc (Wash) 2001 and 41(2):200–204.

19. Smith WD, Hatton RC, Fann AL, Baz MA, Kaplan B. Evaluation of drug interaction software

to identify alerts for transplant medications. Ann Pharmacother. 2005 and 39(1):45–50.

20. Saverno KR, Hines LE, Warholak TL, et al. Ability of pharmacy clinical decision-support

software to alert users about clinically important drug-drug interactions. J Am Med Inform Assoc.

2011 and 18(1):32–37.

21. Metzger J, Welebob E, Bates DW, Lipsitz S, Classen DC. Mixed results in the safety

performance of computerized physician order entry. Health Aff (Millwood) 2010 and 29(4):655–

663.

22. Murphy JE, Forrey RA, Desiraju U. Community pharmacists' responses to drug-drug interaction

alerts. Am J Health Syst Pharm. 2004 and 61(14):1484–1487.

23. Weingart SN, Simchowitz B, Shiman L, et al. Clinicians' assessments of electronic medication

safety alerts in ambulatory care. Arch Intern Med. 2009 and 169(17):1627–1632.

24. Isaac T, Weissman JS, Davis RB, et al. Overrides of medication alerts in ambulatory care. Arch

Intern Med. 2009 and 169(3):305–311.

22

25. Bryant AD, Fletcher GS, Payne TH. Drug interaction alert override rates in the Meaningful Use

era. No evidence of progress. Applied Clinical Informatics. 2014 and 5(3):802–813.

26. Hines LE, Malone DC, Murphy JE. Recommendations for generating, evaluating, and

implementing drug-drug interaction evidence. Pharmacotherapy. 2012 Apr and 32(4):304–13.

27. Hansten PD, Horn JR, Hazlet TK. ORCA: OpeRational ClassificAtion of drug interactions. J

Am Pharm Assoc (Wash) 2001 Mar-Apr and 41(2):161–5.

28. Caccia S, Garattini S, Pasina L, Nobili A. Predicting the clinical relevance of drug interactions

from pre-approval studies. Drug Saf. 2009 and 32(11):1017–39.

29. Holbrook A, Grootendorst P, Willison D, Goldsmith C, Sebaldt R, Keshavjee K. Can current

electronic systems meet drug safety and effectiveness requirements? AMIA Annu Symp Proc.

2005:335–9.

30. Ridgeley MS, Greenberg MD. Too many alerts, too much liability: Sorting through the

malpractice implications of drug-drug interaction clinical decision support. Saint Louis University

Journal of Health Law and Policy. 2012 and 5(257):257–96.

31. Polidori P, Di Giorgio C, Provenzani A. Incidence of potential drug interactions in a transplant

centre setting and relevance of electronic alerts for clinical practice support. Inform. Prim. Care, 20

(4) (2012), pp. 257-262.

32. Smithburger PL, Buckley MS, Bejian S, Burenheide K, Kane-Gill SL. A critical evaluation of

clinical decision support for the detection of drug–drug interactions, Expert Opinion on Drug

Safety, 2011 10:6, 871-882.

33. Lin CP, Payne TH, Nichol WP, Hoey PJ, Anderson CL, Gennari JH. Evaluating clinical

decision support systems: monitoring CPOE order check override rates in the Department of

Veterans Affairs' Computerized Patient Record System. J. Am. Med. Inform. Assoc., 15

(September–October (5)) (2008), pp. 620-626.

34. Strom BL, Schinnar R, Bilker W, Hennessy S, Leonard CE, Pifer E. Randomized clinical trial

of a customized electronic alert requiring an affirmative response compared to a control group

receiving a commercial passive CPOE alert: NSAID–warfarin co-prescribing as a test case. J. Am.

Med. Inform. Assoc., 17 (July–August (4)) (2010), pp. 411-415.

35. J. Grimshaw, M. Campbell, M. Eccles, N. Steen, 2000 Experimental and quasi-experimental

designs for evaluating guideline implementation strategies. Family Practice and 17:S11-S18.

23

36. Garg AX, Adhikari NK, McDonald H et al. Effects of computerized clinical decision support

systems on practitioner performance and patient outcomes: a systematic review. J Am Med Assoc,

2005 and 293:1223–1238.

37. Damiani G, Pinnarelli L, Colosimo SC et al. The effectiveness of computerized clinical

guidelines in the process of care: a systematic review. BMC Health Serv Res, 2010 and 10:2.

38. Robertson J, Walkom E, Pearson SA, Hains I, Williamsone M, Newby D. The impact of

pharmacy computerised clinical decision support on prescribing, clinical and patient outcomes: a

systematic review of the literature. Int J Pharm Pract, 2010 and 18:69–87.

39. Jaspers MW, Smeulers M, Vermeulen H, Peute LW. Effects of clinical decision-support

systems on practitioner performance and patient outcomes: a synthesis of high-quality systematic

review findings. J Am Med Inform Assoc, 2011 and 18:327–334.

40. Page N, Baysari MT, Westbrook JI. A systematic review of the effectiveness of interruptive

medication prescribing alerts in hospital CPOE systems to change prescriber behavior and improve

patient safety. Int J Med Inform. 2017 Sep and 105:22-30.

41. McCoy AB, Thomas EJ, Krousel-Wood M, Sittig DF. Clinical decision support alert

appropriateness: a review and proposal for improvement. Ochsner J. 2014 Summer and

14(2):195-202.

42. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, Clarke M, Devereaux

PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and

meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009

Jul 21;339:b2700..

43. Van der Sijs H, Bouamar R, van Gelder T, Aarts J, Berg M, Vulto A. Functionality test for drug

safety alerting in computerized physician order entry systems. Int J Med Inform. 2010 Apr and

79(4):243-51.

44. Zorina OI, Haueis P, Semmler A, Marti I, Gonzenbach RR, Guzek M, Kullak-Ublick GA,

Weller M, Russmann S. Comparative evaluation of the drug interaction screening programs MediQ

and ID PHARMA CHECK in neurological inpatients. Pharmacoepidemiol Drug Saf. 2012

Aug;21(8):872-80.

45. Silverman JB, Stapinski CD, Huber C, Ghandi TK, Churchill WW. Computer-based system for

preventing adverse drug events. Am J Health Syst Pharm. 2004 Aug 1 and 61(15):1599-603.

24

46. Doormaal van JE, Rommers MK, Kosterink JG, Teepe-Twiss IM, Haaijer-Ruskamp FM, Mol

PG. Comparison of methods for identifying patients at risk of medication-related harm. Qual Saf

Health Care. 2010 Dec and 19(6):e26.

47. Van der Sijs H, Mulder A, van Gelder T, Aarts J, Berg M, Vulto A. Drug safety alert generation

and overriding in a large Dutch university medical centre. Pharmacoepidemiol Drug Saf. 2009 Oct

and 18(10):941-7.

48. Schedlbauer A, Prasad V, Mulvaney C, et al. What evidence supports the use of computerized

alerts and prompts to improve clinicians' prescribing behavior? J. Am. Med. Inform. Assoc., 16

(July–August (4)) (2009), pp. 531-538.

49. Rommers MK, Zwaveling J, Guchelaar HJ, Teepe-Twiss IM. Evaluation of rule effectiveness

and positive predictive value of clinical rules in a Dutch clinical decision support system in daily

hospital pharmacy practice. Artif Intell Med.2013 Sep and 59(1):15-21.

50. Tamblyn R, Huang A, Taylor L, Kawasumi Y, Bartlett G, Grad R, Jacques A, Dawes M,

Abrahamowicz M, Perreault R, Winslade N, Poissant L, Pinsonneault A (2008). A randomized trial

of the effectiveness of on-demand versus computer-triggered drug decision support in primary care.

J Am Med Inform Assoc 15(4):430–438.

51. Van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized

physician order entry, J. Am. Med. Inform. Assoc. 13 (2006) 138–147.

52. Oertle M. Frequency and nature of drug-drug interactions in a Swiss primary and secondary

acute care hospital. Swiss Med Wkly 142:0., 2012.

FULL RESULTS OF THE LITERATURE SEARCH

Results of PubMed database search

1: Tolley CL, Slight SP, Husband AK, Watson N, Bates DW. Improving medication-related clinical decision support. Am J Health Syst Pharm. 2018 Feb 15;75(4):239-246. doi: 10.2146/ajhp160830. PubMed PMID: 29436470. 2: Shen Y, Yuan K, Chen D, Colloc J, Yang M, Li Y, Lei K. An ontology-driven clinical decision support system (IDDAP) for infectious disease diagnosis and antibiotic prescription. Artif Intell Med. 2018 Feb 9. pii:

S0933-3657(17)30234-8. doi: 10.1016/j.artmed.2018.01.003. [Epub ahead of print] PubMed PMID: 29433958.

3: Cornu P, Phansalkar S, Seger DL, Cho I, Pontefract S, Robertson A, Bates DW, Slight SP. High-priority and low-priority drug-drug interactions in different international electronic health record systems: A comparative study. Int J Med Inform. 2018 Mar;111:165-171. doi: 10.1016/j.ijmedinf.2017.12.027. Epub 2018 Jan 9. PubMed PMID: 29425628.

4: Haefeli WE, Seidling HM. [Electronic decision support to promote medication safety]. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. 2018 Jan 16. doi: 10.1007/s00103-017-2685-8. [Epub ahead of print] Review. German. PubMed PMID: 29340732.

5: Teixeira V, Rubin L, Rebrij R, Tamborindeguy A, Martinez R, Bacigalupo JC, Luna D. Implementation of an Outsourced Transnational Service of Clinical

Decision Support System. Stud Health Technol Inform. 2017;245:1384. PubMed PMID: 29295463.

6: Luna DR, Rizzato Lede DA, Rubin L, Otero CM, Ortiz JM, García MG, Rapisarda RP, Risk MR, González Bernaldo de Quirós F. User-Centered Design Improves the Usability of Drug-Drug Interaction Alerts: A Validation Study in the Real

Scenario. Stud Health Technol Inform. 2017;245:1085-1089. PubMed PMID: 29295269. 7: Heringa M, van der Heide A, Floor-Schreudering A, De Smet PAGM, Bouvy ML. Better specification of triggers to reduce the number of drug interaction alerts in primary care. Int J Med Inform. 2018 Jan;109:96-102. doi:

10.1016/j.ijmedinf.2017.11.005. Epub 2017 Nov 12. PubMed PMID: 29195711. 8: Nanji KC, Seger DL, Slight SP, Amato MG, Beeler PE, Her QL, Dalleur O, Eguale T, Wong A, Silvers ER, Swerdloff M, Hussain ST, Maniam N, Fiskio JM, Dykes PC, Bates DW. Medication-related clinical decision support alert overrides in

inpatients. J Am Med Inform Assoc. 2017 Oct 27. doi: 10.1093/jamia/ocx115. [Epub ahead of print] PubMed PMID: 29092059.

9: Koutkias V, Bouaud J. Contributions from the 2016 Literature on Clinical Decision Support. Yearb Med Inform. 2017 Aug;26(1):133-138. doi: 10.15265/IY-2017-031. Epub 2017 Sep 11. PubMed PMID: 29063553. 10: Zenziper Straichman Y, Kurnik D, Matok I, Halkin H, Markovits N, Ziv A, Shamiss A, Loebstein R. Prescriber response to computerized drug alerts for electronic prescriptions among hospitalized patients. Int J Med Inform. 2017 Nov;107:70-75. doi: 10.1016/j.ijmedinf.2017.08.008. Epub 2017 Aug 31. PubMed PMID: 29029694.

11: Maulucci G, Cordelli E, Rizzi A, De Leva F, Papi M, Ciasca G, Samengo D, Pani G, Pitocco D, Soda P, Ghirlanda G, Iannello G, De Spirito M. Phase separation of the plasma membrane in human red blood cells as a potential tool for diagnosis and progression monitoring of type 1 diabetes mellitus. PLoS One. 2017 Sep 7;12(9):e0184109. doi: 10.1371/journal.pone.0184109. eCollection 2017. PubMed PMID: 28880900; PubMed Central PMCID: PMC5589169.

12: Wong A, Wright A, Seger DL, Amato MG, Fiskio JM, Bates D. Comparison of Overridden Medication-related Clinical Decision Support in the Intensive Care

Unit between a Commercial System and a Legacy System. Appl Clin Inform. 2017 Aug 23;8(3):866-879. doi: 10.4338/ACI-2017-04-RA-0059. PubMed PMID: 28832067.

13: Amkreutz J, Koch A, Buendgens L, Muehlfeld A, Trautwein C, Eisert A. Prevalence and nature of potential drug-drug interactions among kidney transplant patients in a German intensive care unit. Int J Clin Pharm. 2017

Oct;39(5):1128-1139. doi: 10.1007/s11096-017-0525-4. Epub 2017 Aug 19. PubMed PMID: 28823018.

14: Tso GJ, Tu SW, Musen MA, Goldstein MK. High-Risk Drug-Drug Interactions Between Clinical Practice Guidelines for Management of Chronic Conditions. AMIA Jt Summits Transl Sci Proc. 2017 Jul 26;2017:531-539. eCollection 2017. PubMed PMID: 28815153; PubMed Central PMCID: PMC5543385.

15: Page N, Baysari MT, Westbrook JI. A systematic review of the effectiveness of interruptive medication prescribing alerts in hospital CPOE systems to change prescriber behavior and improve patient safety. Int J Med Inform. 2017

Sep;105:22-30. doi: 10.1016/j.ijmedinf.2017.05.011. Epub 2017 May 27. Review. PubMed PMID: 28750908.

16: Kane-Gill SL, O'Connor MF, Rothschild JM, Selby NM, McLean B, Bonafide CP, Cvach MM, Hu X, Konkani A, Pelter MM, Winters BD. Technologic Distractions (Part 1): Summary of Approaches to Manage Alert Quantity With Intent to Reduce Alert Fatigue and Suggestions for Alert Fatigue Metrics. Crit Care Med. 2017

Sep;45(9):1481-1488. doi: 10.1097/CCM.0000000000002580. Review. PubMed PMID: 28682835.

17: Blagec K, Kuch W, Samwald M. The Importance of Gene-Drug-Drug-Interactions in Pharmacogenomics Decision Support: An Analysis Based on Austrian Claims Data. Stud Health Technol Inform. 2017;236:121-127. PubMed PMID: 28508787. 18: Kuo YC, Cheng SH. Adoption of medication alert systems in hospital outpatient departments in Taiwan. Int J Med Inform. 2017 Jun;102:111-117. doi:

10.1016/j.ijmedinf.2017.03.012. Epub 2017 Mar 23. PubMed PMID: 28495338. 19: Plank-Kiegele B, Bürkle T, Müller F, Patapovas A, Sonst A, Pfistermeister B, Dormann H, Maas R. Data Requirements for the Correct Identification of Medication Errors and Adverse Drug Events in Patients Presenting at an Emergency Department. Methods Inf Med. 2017 Aug 11;56(4):276-282. doi: 10.3414/ME16-01-0126. Epub 2017 Apr 28. PubMed PMID: 28451686.

20: Lund M, Petersen TS, Dalhoff KP. Clinical Implications of P-Glycoprotein Modulation in Drug-Drug Interactions. Drugs. 2017 May;77(8):859-883. doi: 10.1007/s40265-017-0729-x. Review. PubMed PMID: 28382570.

21: Bahar MA, Hak E, Bos JHJ, Borgsteede SD, Wilffert B. The burden and

management of cytochrome P450 2D6 (CYP2D6)-mediated drug-drug interaction (DDI): co-medication of metoprolol and paroxetine or fluoxetine in the elderly.

Pharmacoepidemiol Drug Saf. 2017 Jul;26(7):752-765. doi: 10.1002/pds.4200. Epub 2017 Mar 26. PubMed PMID: 28345306.

22: Ruhland DJ, Bellone JM, Wilkes E. Implementation and Assessment of an Ambulatory Prescribing Guidance Tool to Improve Patient Safety in the Geriatric Population. Consult Pharm. 2017 Mar 1;32(3):169-174. doi: 10.4140/TCP.n.2017.169. PubMed PMID: 28270272.

23: Wong A, Amato MG, Seger DL, Slight SP, Beeler PE, Dykes PC, Fiskio JM, Silvers ER, Orav EJ, Eguale T, Bates DW. Evaluation of medication-related clinical decision support alert overrides in the intensive care unit. J Crit Care. 2017 Jun;39:156-161. doi: 10.1016/j.jcrc.2017.02.027. Epub 2017 Feb 20. PubMed PMID: 28259059.

24: Romagnoli KM, Nelson SD, Hines L, Empey P, Boyce RD, Hochheiser H. Information needs for making clinical recommendations about potential drug-drug interactions: a synthesis of literature review and interviews. BMC Med Inform Decis Mak. 2017 Feb 22;17(1):21. doi: 10.1186/s12911-017-0419-3. Review. PubMed PMID: 28228132; PubMed Central PMCID: PMC5322613.

25: Amkreutz J, Koch A, Buendgens L, Trautwein C, Eisert A. Clinical decision support systems differ in their ability to identify clinically relevant drug

interactions of immunosuppressants in kidney transplant patients. J Clin Pharm Ther. 2017 Jun;42(3):276-285. doi: 10.1111/jcpt.12508. Epub 2017 Feb 21. PubMed PMID: 28224645.

26: Gaziano TA, Abrahams-Gessel S, Gomez-Olive FX, Wade A, Crowther NJ, Alam S, Manne-Goehler J, Kabudula CW, Wagner R, Rohr J, Montana L, Kahn K, Bärnighausen TW, Berkman LF, Tollman S. Cardiometabolic risk in a population of older adults with multiple co-morbidities in rural south africa: the HAALSI (Health and Aging in Africa: longitudinal studies of INDEPTH communities) study. BMC Public Health. 2017 Feb 17;17(1):206. doi: 10.1186/s12889-017-4117-y. PubMed PMID: 28212629; PubMed Central PMCID: PMC5314614.

27: Elliott LS, Henderson JC, Neradilek MB, Moyer NA, Ashcraft KC, Thirumaran RK. Clinical impact of pharmacogenetic profiling with a clinical decision support

tool in polypharmacy home health patients: A prospective pilot randomized controlled trial. PLoS One. 2017 Feb 2;12(2):e0170905. doi:

10.1371/journal.pone.0170905. eCollection 2017. PubMed PMID: 28151991; PubMed Central PMCID: PMC5289536.

28: Luna DR, Rizzato Lede DA, Otero CM, Risk MR, González Bernaldo de Quirós F. User-centered design improves the usability of drug-drug interaction alerts:

Experimental comparison of interfaces. J Biomed Inform. 2017 Feb;66:204-213. doi: 10.1016/j.jbi.2017.01.009. Epub 2017 Jan 17. PubMed PMID: 28108211.

29: Ferrández O, Urbina O, Grau S, Mateu-de-Antonio J, Marin-Casino M, Portabella J, Mojal S, Riu M, Salas E. Computerized pharmacy surveillance and alert system for drug-related problems. J Clin Pharm Ther. 2017 Apr;42(2):201-208. doi: 10.1111/jcpt.12495. Epub 2017 Jan 12. PubMed PMID: 28078665. 30: Keasberry J, Scott IA, Sullivan C, Staib A, Ashby R. Going digital: a narrative overview of the clinical and organisational impacts of eHealth

technologies in hospital practice. Aust Health Rev. 2017 Dec;41(6):646-664. doi: 10.1071/AH16233. PubMed PMID: 28063462.

31: Reynolds JL, Rupp MT. Improving Clinical Decision Support in Pharmacy: Toward the Perfect DUR Alert. J Manag Care Spec Pharm. 2017 Jan;23(1):38-43. doi:

10.18553/jmcp.2017.23.1.38. PubMed PMID: 28025921.

32: Schreiber R, Gregoire JA, Shaha JE, Shaha SH. Think time: A novel approach to analysis of clinicians' behavior after reduction of drug-drug interaction alerts. Int J Med Inform. 2017 Jan;97:59-67. doi: 10.1016/j.ijmedinf.2016.09.011. Epub 2016 Sep 24. PubMed PMID: 27919396.

33: Horsky J, Aarts J, Verheul L, Seger DL, van der Sijs H, Bates DW. Clinical reasoning in the context of active decision support during medication

prescribing. Int J Med Inform. 2017 Jan;97:1-11. doi:

10.1016/j.ijmedinf.2016.09.004. Epub 2016 Sep 23. PubMed PMID: 27919368. 34: Russ AL, Melton BL, Daggy JK, Saleem JJ. Pilot evaluation of a method to assess prescribers' information processing of medication alerts. J Biomed Inform. 2017 Feb;66:11-18. doi: 10.1016/j.jbi.2016.11.011. Epub 2016 Nov 28. PubMed PMID: 27908833.

35: Nabovati E, Vakili-Arki H, Taherzadeh Z, Saberi MR, Medlock S, Abu-Hanna A, Eslami S. Information Technology-Based Interventions to Improve Drug-Drug Interaction Outcomes: A Systematic Review on Features and Effects. J Med Syst. 2017 Jan;41(1):12. Epub 2016 Nov 26. Review. PubMed PMID: 27889873. 36: Capobussi M, Banzi R, Moja L, Bonovas S, González-Lorenzo M, Liberati EG, Polo Friz H, Nanni O, Mangia M, Ruggiero F. [Computerized decision support systems: EBM at the bedside]. Recenti Prog Med. 2016 Nov;107(11):589-591. doi: 10.1701/2484.25970. Italian. PubMed PMID: 27869877.

37: Payne TH, Desai BR. Examination of medication clinical decision support using Bayes' theorem. Am J Health Syst Pharm. 2016 Nov 15;73(22):1876-1878. PubMed PMID: 27821399.

Kornhuber J, Fromm MF, Bürkle T, Prokosch HU, Maas R. Development of a Standardized Rating Tool for Drug Alerts to Reduce Information Overload. Methods Inf Med. 2016 Dec 7;55(6):507-515. Epub 2016 Oct 26. PubMed PMID: 27782288. 39: Rush JL, Ibrahim J, Saul K, Brodell RT. Improving Patient Safety by Combating Alert Fatigue. J Grad Med Educ. 2016 Oct;8(4):620-621. PubMed PMID: 27777683; PubMed Central PMCID: PMC5058605.

40: Weersink RA, Bouma M, Burger DM, Drenth JP, Hunfeld NG, Kranenborg M, Monster-Simons MH, van Putten SA, Metselaar HJ, Taxis K, Borgsteede SD. Evaluating the safety and dosing of drugs in patients with liver cirrhosis by literature review and expert opinion. BMJ Open. 2016 Oct 12;6(10):e012991. doi:

10.1136/bmjopen-2016-012991. Review. PubMed PMID: 27733414; PubMed Central PMCID: PMC5073492.

41: Dawes M, Aloise MN, Ang JS, Cullis P, Dawes D, Fraser R, Liknaitzky G, Paterson A, Stanley P, Suarez-Gonzalez A, Katzov-Eckert H. Introducing pharmacogenetic testing with clinical decision support into primary care: a feasibility study. CMAJ Open. 2016 Sep 21;4(3):E528-E534. eCollection 2016 Jul-Sep. PubMed PMID: 27730116; PubMed Central PMCID: PMC5047800. 42: Caraballo PJ, Hodge LS, Bielinski SJ, Stewart AK, Farrugia G, Schultz CG, Rohrer-Vitek CR, Olson JE, St Sauver JL, Roger VL, Parkulo MA, Kullo IJ, Nicholson WT, Elliott MA, Black JL, Weinshilboum RM. Multidisciplinary model to implement pharmacogenomics at the point of care. Genet Med. 2017

Apr;19(4):421-429. doi: 10.1038/gim.2016.120. Epub 2016 Sep 22. PubMed PMID: 27657685; PubMed Central PMCID: PMC5362352.

43: Chaparro JD, Classen DC, Danforth M, Stockwell DC, Longhurst CA. National trends in safety performance of electronic health record systems in children's hospitals. J Am Med Inform Assoc. 2017 Mar 1;24(2):268-274. doi: 10.1093/jamia/ocw134. PubMed PMID: 27638908.

44: Luna D, Otero C, Risk M, Stanziola E, González Bernaldo de Quirós F. Impact of Participatory Design for Drug-Drug Interaction Alerts. A Comparison Study

Between Two Interfaces. Stud Health Technol Inform. 2016;228:68-72. PubMed PMID: 27577343.

45: McEvoy DS, Sittig DF, Hickman TT, Aaron S, Ai A, Amato M, Bauer DW, Fraser GM, Harper J, Kennemer A, Krall MA, Lehmann CU, Malhotra S, Murphy DR, O'Kelley B, Samal L, Schreiber R, Singh H, Thomas EJ, Vartian CV, Westmorland J, McCoy AB, Wright A. Variation in high-priority drug-drug interaction alerts across

institutions and electronic health records. J Am Med Inform Assoc. 2017 Mar

1;24(2):331-338. doi: 10.1093/jamia/ocw114. PubMed PMID: 27570216; PubMed Central PMCID: PMC5391726.

46: Tan Y, Hu Y, Liu X, Yin Z, Chen XW, Liu M. Improving drug safety: From adverse drug reaction knowledge discovery to clinical implementation. Methods. 2016 Nov 1;110:14-25. doi: 10.1016/j.ymeth.2016.07.023. Epub 2016 Jul 30. Review. PubMed PMID: 27485605.

47: de Wit HA, Hurkens KP, Mestres Gonzalvo C, Smid M, Sipers W, Winkens B, Mulder WJ, Janknegt R, Verhey FR, van der Kuy PH, Schols JM. The support of medication reviews in hospitalised patients using a clinical decision support system. Springerplus. 2016 Jun 24;5(1):871. doi: 10.1186/s40064-016-2376-1. eCollection 2016. PubMed PMID: 27386320; PubMed Central PMCID: PMC4920784. 48: Khalifa M, Zabani I. Improving Utilization of Clinical Decision Support

Systems by Reducing Alert Fatigue: Strategies and Recommendations. Stud Health Technol Inform. 2016;226:51-4. PubMed PMID: 27350464.

49: Hicks JK, Stowe D, Willner MA, Wai M, Daly T, Gordon SM, Lashner BA, Parikh S, White R, Teng K, Moss T, Erwin A, Chalmers J, Eng C, Knoer S. Implementation of Clinical Pharmacogenomics within a Large Health System: From Electronic Health Record Decision Support to Consultation Services. Pharmacotherapy. 2016

Aug;36(8):940-8. doi: 10.1002/phar.1786. Epub 2016 Jul 20. PubMed PMID: 27312955. 50: Bitter K, Schlender JF, Woltersdorf R. [Drug-drug interactions in the elderly

: Which ones really matter?]. Internist (Berl). 2016 Jul;57(7):728-34. doi: 10.1007/s00108-016-0078-y. Review. German. PubMed PMID: 27294383. 51: Young A, Tordoff J, Dovey S, Reith D, Lloyd H, Tilyard M, Smith A. Using an Electronic Decision Support Tool to Reduce Inappropriate Polypharmacy and Optimize Medicines: Rationale and Methods. JMIR Res Protoc. 2016 Jun

10;5(2):e105. doi: 10.2196/resprot.5543. PubMed PMID: 27288200; PubMed Central PMCID: PMC4920961.

52: Wipfli R, Ehrler F, Bediang G, Bétrancourt M, Lovis C. How Regrouping Alerts in Computerized Physician Order Entry Layout Influences Physicians' Prescription Behavior: Results of a Crossover Randomized Trial. JMIR Hum Factors. 2016 Jun 2;3(1):e15. doi: 10.2196/humanfactors.5320. PubMed PMID: 27255612; PubMed Central PMCID: PMC4911510.

53: Kannan V, Swartz F, Kiani NA, Silberberg G, Tsipras G, Gomez-Cabrero D, Alexanderson K, Tegnèr J. Conditional Disease Development extracted from Longitudinal Health Care Cohort Data using Layered Network Construction. Sci Rep. 2016 May 23;6:26170. doi: 10.1038/srep26170. PubMed PMID: 27211115; PubMed Central PMCID: PMC4876508.

54: Eschmann E, Beeler PE, Schneemann M, Blaser J. Developing strategies for predicting hyperkalemia in potassium-increasing drug-drug interactions. J Am Med Inform Assoc. 2017 Jan;24(1):60-66. doi: 10.1093/jamia/ocw050. Epub 2016 May 12. PubMed PMID: 27174894.

55: Heinz A, Schlagenhauf F, Beck A, Wackerhagen C. Dimensional psychiatry: mental disorders as dysfunctions of basic learning mechanisms. J Neural Transm (Vienna). 2016 Aug;123(8):809-21. doi: 10.1007/s00702-016-1561-2. Epub 2016 May 4. Review. PubMed PMID: 27145766.

56: Finkelstein J, Friedman C, Hripcsak G, Cabrera M. Potential utility of precision medicine for older adults with polypharmacy: a case series study. Pharmgenomics Pers Med. 2016 Apr 15;9:31-45. doi: 10.2147/PGPM.S101474. eCollection 2016. PubMed PMID: 27143951; PubMed Central PMCID: PMC4846041. 57: Dix A, Vlaic S, Guthke R, Linde J. Use of systems biology to decipher

host-pathogen interaction networks and predict biomarkers. Clin Microbiol Infect. 2016 Jul;22(7):600-6. doi: 10.1016/j.cmi.2016.04.014. Epub 2016 Apr 22. Review. PubMed PMID: 27113568.

58: Stingl JC, Kaumanns KL, Claus K, Lehmann ML, Kastenmüller K, Bleckwenn M, Hartmann G, Steffens M, Wirtz D, Leuchs AK, Benda N, Meier F, Schöffski O, Holdenrieder S, Coch C, Weckbecker K. Individualized versus standardized risk assessment in patients at high risk for adverse drug reactions (IDrug) - study protocol for a pragmatic randomized controlled trial. BMC Fam Pract. 2016 Apr 26;17:49. doi: 10.1186/s12875-016-0447-6. PubMed PMID: 27112273; PubMed Central PMCID: PMC4845354.

59: Heringa M, Siderius H, Floor-Schreudering A, de Smet PA, Bouvy ML. Lower alert rates by clustering of related drug interaction alerts. J Am Med Inform

Assoc. 2017 Jan;24(1):54-59. doi: 10.1093/jamia/ocw049. Epub 2016 Apr 23. PubMed PMID: 27107437.

60: Lebowitz MB, Olson KL, Burns M, Harper MB, Bourgeois F. Drug-Drug Interactions Among Hospitalized Children Receiving Chronic Antiepileptic Drug Therapy. Hosp Pediatr. 2016 May;6(5):282-9. doi: 10.1542/hpeds.2015-0249. Epub 2016 Jan 1. PubMed PMID: 27081190.

61: Tilson H, Hines LE, McEvoy G, Weinstein DM, Hansten PD, Matuszewski K, le Comte M, Higby-Baker S, Hanlon JT, Pezzullo L, Vieson K, Helwig AL, Huang SM, Perre A, Bates DW, Poikonen J, Wittie MA, Grizzle AJ, Brown M, Malone DC. Recommendations for selecting drug-drug interactions for clinical decision support. Am J Health Syst Pharm. 2016 Apr 15;73(8):576-85. doi:

10.2146/ajhp150565. PubMed PMID: 27045070; PubMed Central PMCID: PMC5064943. 62: Turgeon J, Michaud V. Clinical decision support systems: great promises for

better management of patients' drug therapy. Expert Opin Drug Metab Toxicol. 2016 Sep;12(9):993-5. doi: 10.1517/17425255.2016.1171317. Epub 2016 Apr 11. Erratum in: Expert Opin Drug Metab Toxicol. 2016 Sep;12 (9):iii. PubMed PMID: 27021287. 63: De Meulder M, Waldron MP, Li L, Peay MG, Tingler MJ, Hidy BJ, Verhaeghe T, Jenkins RG. Development and validation of HILIC-ESI/MS/MS methods for simultaneous quantitation of several antipsychotics in human plasma and blood. Bioanalysis. 2016 Apr;8(8):765-94. doi: 10.4155/bio.16.27. Epub 2016 Mar 23. PubMed PMID: 27005848.

64: Bedford M, Stevens P, Coulton S, Billings J, Farr M, Wheeler T, Kalli M, Mottishaw T, Farmer C. Development of risk models for the prediction of new or worsening acute kidney injury on or during hospital admission: a cohort and nested study. Southampton (UK): NIHR Journals Library; 2016 Feb. PubMed PMID: 26937542.

65: Jang C, Yoo KB, Kim W, Park MY, Ahn EK, Park RW. New Alert Override Codes for the Drug Utilization Review System Derived from Outpatient Prescription Data from a Tertiary Teaching Hospital in Korea. Healthc Inform Res. 2016 Jan;22(1):39-45. doi: 10.4258/hir.2016.22.1.39. Epub 2016 Jan 31. PubMed PMID: 26893949; PubMed Central PMCID: PMC4756057.

66: Wu X, Wu C, Zhang K, Wei D. Residents' numeric inputting error in computerized physician order entry prescription. Int J Med Inform. 2016

Apr;88:25-33. doi: 10.1016/j.ijmedinf.2016.01.002. Epub 2016 Jan 15. PubMed PMID: 26878759.

67: Alagiakrishnan K, Wilson P, Sadowski CA, Rolfson D, Ballermann M, Ausford A, Vermeer K, Mohindra K, Romney J, Hayward RS. Physicians' use of computerized clinical decision supports to improve medication management in the elderly - the Seniors Medication Alert and Review Technology intervention. Clin Interv Aging. 2016 Jan 25;11:73-81. doi: 10.2147/CIA.S94126. eCollection 2016. PubMed PMID: 26869776; PubMed Central PMCID: PMC4734726.

68: Rasoolimoghadam M, Safdari R, Ghazisaeidi M, Maharanitehrani M, Tahmasebiyan S. Designing Decision Support System to Detect Drug Interactions Type 2 Diabetes. Acta Inform Med. 2015 Dec;23(6):336-8. doi: 10.5455/aim.2015.23.336-338. PubMed PMID: 26862240; PubMed Central PMCID: PMC4720826.

69: Bauermeister J, Giguere R, Dolezal C, Leu CS, Febo I, Cranston RD, Mayer K, McGowan I, Carballo-Diéguez A. To Use a Rectal Microbicide, First Insert the Applicator: Gel and Applicator Satisfaction Among Young Men Who Have Sex With Men. AIDS Educ Prev. 2016 Feb;28(1):1-10. doi: 10.1521/aeap.2016.28.1.1. PubMed PMID: 26829253; PubMed Central PMCID: PMC4942182.

70: Sönnichsen A, Trampisch US, Rieckert A, Piccoliori G, Vögele A, Flamm M, Johansson T, Esmail A, Reeves D, Löffler C, Höck J, Klaassen-Mielke R, Trampisch HJ, Kunnamo I. Polypharmacy in chronic diseases-Reduction of Inappropriate Medication and Adverse drug events in older populations by electronic Decision Support (PRIMA-eDS): study protocol for a randomized controlled trial. Trials.

2016 Jan 29;17:57. doi: 10.1186/s13063-016-1177-8. PubMed PMID: 26822311; PubMed Central PMCID: PMC5526277.

71: Hart T, Xie L. Providing data science support for systems pharmacology and its implications to drug discovery. Expert Opin Drug Discov. 2016;11(3):241-56. doi: 10.1517/17460441.2016.1135126. Epub 2016 Jan 9. Review. PubMed PMID: 26689499; PubMed Central PMCID: PMC4988863.

72: Meldolesi E, van Soest J, Damiani A, Dekker A, Alitto AR, Campitelli M, Dinapoli N, Gatta R, Gambacorta MA, Lanzotti V, Lambin P, Valentini V. Standardized data collection to build prediction models in oncology: a prototype for rectal cancer. Future Oncol. 2016 Jan;12(1):119-36. doi: 10.2217/fon.15.295. Epub 2015 Dec 17. Review. PubMed PMID: 26674745.

73: Salili AR, Hammann F, Taegtmeyer AB. [ Preventing adverse drug events using clinical decision support systems]. Ther Umsch. 2015 Dec;72(11-12):693-700. doi: 10.1024/0040-5930/a000739. Review. German. PubMed PMID: 26654813.