Waiting time information in the Italian NHS: A citizen perspective

(1)

ContentslistsavailableatScienceDirect

Health

Policy

jo u rn al h om ep a ge :w w w . e l s e v i e r . c o m / l o c a t e / h e a l t h p o l

Waiting

time

information

in

the

Italian

NHS:

A

citizen

perspective

Sabina

De

Rosis

a

_,

_Elisa

_Guidotti

a,∗

_,

_Sara

_Zuccarino

a

_,

_Giulia

_Venturi

b

_,

_Francesca

_Ferré

a a_Management_and_Health_Laboratory,_Institute_of_Management_and_Department_EMbeDS,_Scuola_Superiore_Sant’Anna,_Pisa,_Italy

b_Italian_Natural_Language_Processing_Laboratory_(ItaliaNLP_Lab),_Institute_of_{Computational}_Linguistics_“A._Zampolli”_(ILC-CNR),_Pisa,_Italy

a

r

t

i

c

l

e

i

n

f

o

Articlehistory:

Received14January2020

Receivedinrevisedform4May2020 Accepted11May2020 Keywords: Waitingtimes Healthcare Onlineinformation Readability Italy

a

b

s

t

r

a

c

t

Publicinvolvementinthemanagementandcommunicationofwaitingtimesisknowntosupport initia-tivestoreducewaitingtimes,aswellasincreasefairnessandpromotetransparencyandaccountability. Inordertoimprovetransparencyandcommunicationtocitizens,ItalyrecentlyupdatedtheNational RegulatoryPlanforWaitingLists(2019–2021),whichcallsforthedisclosureofwaitingtimeinformation onhealthcareproviderwebpages.Thisstudyanalyseswaitingtimeinformationforoutpatientvisitsand digitalservicesavailableontheinstitutionalwebsitepagesof144publichealthcareorganisationsinnine regionsandtwoautonomousprovincesofItaly.Webpageswereanalysedbothintermsoftheavailable information/services,usingagrid,andintermsofthequalityofthetextusinganadvanced readabil-ityassessmenttool(READ-IT).Thisinformationwascomplementedandvalidatedbyregionalhealthcare keyinformantsduringresearch-speciﬁcworkshops.Waitingtimeinformationdisclosure,digitalservices andtextreadabilityvariedbothwithinandbetweentheregionalhealthcaresystemsandorganisations. Thetypesandcharacteristicsofwaitingtimeinformationandstatisticsvaryconsiderablywithanegative impactontheiruseforbenchmarkingandtheirreadabilityandusabilityforbookingpurposes.Overall, communicationweaknessesduetolowharmonizationandclarityofinformationcanundermineefforts ineffectivelyinformingandinvolvingthepublicthroughonlinewaitingtimedatadisclosure.

1. Introduction

InseveralOECDcountries,waitingtimes(WTs)areamajor

pol-icyissue[1]andoneofthemainconcernsforthegeneralpublic[2].

Severalcountrieshaveimplementednationalstrategiestoaddress

WTs,bothonthesupplyanddemandsides[3].

Publicinvolvement in WT management and communication

supportinitiativestoreduceWTs,increasefairnessandpromote

transparency and accountability [4–6]. The publicwants to be

engagedandinformedabouthowWTsaremanaged[6,7].Patients

that receive accurate information on WTs and the reasons for

waiting, appear tomaintaina senseof control duringthewait

(includingreducedanxiety)andthistendstoincreasetheir

will-ingnesstowait[8–10].Theprecisionandvalidityofsuchdataand

theircontextualizationiscrucialingeneratingvalueforpatients

[11].Manycountriesarethusinvestingheavilyincreatingsystems

tomakeinformationonWTsavailable,usingwebsitesforpublic

disclosure.In ordertoimproveaccountability,transparencyand

communicationtocitizens,theNationalHealthcareSystem(NHS)

∗ Correspondingauthorat:ScuolaSuperioreSant’Anna,PiazzaMartiridella Lib-ertà33,56127,PisaItaly.

E-mailaddress:elisa.guidotti@santannapisa.it(E.Guidotti).

inItalymandatedthedisclosureofWTinformationonhealthcare

providers’webpagesforspeciﬁcoutpatientvisits,diagnostic

ser-vicesandelectiveprocedures.Theserecommendationsarepartof

theNationalRegulatoryPlanforWaitingLists(NationalPlan),the

strategicdocumentdeﬁningtheframeworkforWTmanagement

inItaly.ThelatestNationalPlanwasissuedin2019andstressed

theneedtoguaranteeamaximumWTforselectedoutpatientand

electivesurgeries,whileimprovingsystemefﬁciencyand

enhanc-ingaccessand informationtocitizens(seeBox 1forthedetails

ofthethree-yearNationalPlan2019–2021).Thedocumentalso

reiteratestherecommendationsalreadyoutlinedintheprevious

NationalPlan.

TheaboveareumbrellameasurespromotedacrossallItalian

regionalhealthcare systems.Eachregionalhealthcare systemis

alsorequiredtoissuearegionalWTplandetailinglocalmeasures

andrequirementsforachievingthegoalsoftheNationalPlan.

InmostItalianregionalhealthcaresystems,patientaccessfor

selectedhealthcareservicesisenhancedbyusingprioritylevels

formanagingwaitinglists,re-engineeringbookingprocesses(e.g.

bycreatinguniﬁedbookingcentres)andimprovingavailable

infor-mationtocitizensaboutWT.However,currentpolicies-including

theimplementationofthenationallegislationonpublicly

report-ingofWTs-varyamongregions,withsomebeingveryactiveand

https://doi.org/10.1016/j.healthpol.2020.05.012

(2)

Table1

Numberofwebsitesanalysedbyregionandtypeoforganisation.

Region/Autonomous province(AP)

RegionalandLHAsa

websites

Publichospital Institutionsb_websites

FormerOrganisationsc_still

activewebsites

Total

Apulia 7 4 11

Basilicata 3 2 5

Bolzano(AP) 2 2

FriuliVeneziaGiulia 6 2 8

Liguria 6 4 10 Lombardy 9 33 42 Marche 5 3 8 16 Trento(AP) 2 2 Tuscany 4 6 7 17 Umbria 3 2 5 Veneto 10 3 13 26 Total 57 59 28 144

a_LHAs_and_two_LHAs_with_an_academic_proﬁle_operating_in_Friuli_Venezia_Giulia. b_Public_hospital_enterprises,_THs_and_IRCCSs.

c _Public_healthcare_{organisations}_with_still_active_website_after_mergers.

othersoftenlaggingbehind.Regionalautonomyinhealthcare

ser-viceshasledtotheadoptionofdifferentapproachesforreporting

WTperformanceinformation.Todate,thereisalackofsystematic

comparativeassessmentsoftheinformationprovidedbetweenand

withinregionsandhealthcareorganisations.

Anotherkey aspectthatis missing is theassessment ofWT

informationqualityinordertounderstandwhethertheWTdata

availableonlineareactuallyusefulandreadilyusableforcitizens.

Usefulnessrelatestothesatisfactionofinformationneedsbased

onusers’expectations,whileinformationisusableifusers

effec-tivelyincorporatespeciﬁcinformationintoadecisionprocessand

thusisperceivedassalient,legitimateandcredible[12].In

disclos-ingpublicperformanceinformation,itisimportanttoensurethat

informationthathealthcareprovidershopeisusefulwillactually

helpcitizensintheirdecision-making[13].

Currentevidencehighlightsthatpublicdisclosureof

informa-tionimpacts onboth providersand populationbehaviours, but

not in a uniformfashion [14]. In thehealthcare sector

perfor-manceinformationaffectsmanagers’andprofessionals’behaviour

[15–17],buttoalessextentthepublic’sdecision-making[18,19].

Onlinepublicdisclosureof WTsanswers theneedfor public

accountability.However,ifmeasurementsystemsarenotbased

onhowthemeasurementswillbeactuallyused,theperformance

informationthatcomesoutoftheprocesshasa highchanceof

beingnotusedorusedinappropriately.Performanceinformation

alsoaddressescitizensintheirrolesascustomersofpublic

ser-vices,particularlywhengovernmentscreateyardstickcompetition

betweenhealthcareprovidersandthususerscan“shoparound”

[13].InformationonWTsshouldthusbeeasytoﬁnd,readableand

usablebycitizenfordecisionssuchasschedulingselected

outpa-tientandelectiveservices.Onlinebookingshouldalsobeprovided

[20–22].

Finally,readabilityiskeytoperformancemeasurement

infor-mation.Providingunderstandableinformation(e.g.explanations

oftablesand graphs)iscrucial topreventmisinterpretationsof

WTdata,misleadingconclusionsandconsequentdistrustin

pub-licinstitutions[4,6,11,23]. Forinstance,websitesshouldalsobe

readablebythosewithlowliteracyskills.Researchonthe

read-ability of health-related information in Italy, suchas informed

consents,hasshownthatthequalityofwritteninformationisstill

low[24].

This paper aims to describe (i) variation both within and

betweenregionalhealthcaresystemsinItalywithregardtoonline

WTinformationforoutpatientvisitsandunderstandthelevelof

usefulnessinitscurrentformat,(ii)thereadabilityofsuch

infor-mationthroughthereadabilityanalysisofonlineinformation;and

(iii)theusabilityofWTpublicreportingforcitizens.The

analy-siswascarriedoutbyreviewingtheinstitutionalwebsitepagesof

publichealthcareorganisations.

Thepaperisstructuredasfollows:Section2presentsthe

mate-rialandmethods;theresultsoftheanalysisarethenpresentedin

Section3.Lastly,thediscussion,highlightinginsightsand

sugges-tionsforfurtherresearch,isreportedinSection4.

2. Materialsandmethods

2.1. Studysetting

The Italian healthcare system is a universal decentralised

Beveridge system that comprises nineteen regions and two

autonomous provinces (APs). Since the early 1990s, legislative

reformshavegraduallytransferredpolitical,administrative,ﬁscal

and ﬁnancial responsibilities regardingthe provisionof

health-carefromthenationalgovernmenttotheregionsand APs.The

devolutionpoliciesledtheregionsandAPstodevelop different

organisational and fundingmodels [25] and differences in the

qualityofcareprovided,thelevelofhealthcareexpenditureand

ﬁnancialperformance canbeobserved. Thesystemis currently

organisedandgovernedatthreelevels:national,regionalandlocal

[26]. The central government hasa stewardshiprole: it

deter-minesthecorehealthbeneﬁtstobeuniformlygrantedacrossItaly

andallocatestheﬁnancialresourcestotheregionalgovernments

through general taxation. These regional governments oversee,

organizeand deliverprimary,secondary andtertiaryhealthcare

services,aswellaspreventiveandhealthpromotionservices.They

deﬁnetheirownregionalhealthplans,coordinatethestrategiesof

theregions,allocatethebudgetwithintheirsystemsandmonitor

quality,appropriatenessandefﬁciencyoftheservicesprovided.The

locallevelensurestheprovisionofprimary,secondaryandtertiary

healthcareservices,aswellasthepreventiveandhealthpromotion

servicesthrough:

䊉local health authorities ‘LHAs’ (geographically based

organ-isations, responsible for delivering public health services,

communityhealthcareservicesandprimarycaredirectly,while

secondaryandspecialistcarethroughdirectlymanaged

facili-tiesorbyoutsourcingtopublichospitalinstitutionsorprivate

accreditedproviders);

䊉publichospitalinstitutions(whichoftencooperatewithMedical

SchoolsandworkasTeachingHospitals);

䊉privateaccreditedproviders.

Thesystemallowsthecitizenstochoosethehealthcareprovider

(3)

Table2

Numberandresponsibilitiesoftheexpertsengagedinthetwomeetingsandtheworkshoponcommunicationanddigitalservices.

Numberofparticipants(n)

Domainofexpertise Firstmeeting

November2018,Pisa

WorkshopMarch 2019,Pisa

SecondmeetingMay 2019,Florence

Clinicalexpertise 1 2 1

Management 6 10 5

Performanceandinformationﬂows 18 26 16

Total 25 38 22

Ourstudyfocuses ontheanalysisof thetype and qualityof

WTinformationgatheredfromtheinstitutionalwebsites ofthe

locallevelpublichealthcareorganisationsandtheregional

web-site.Speciﬁcally,thedataandanalysisrefertothenineregionsand

thetwoAPsthathaveadoptedtheItalianRegionalPerformance

EvaluationSystem(IRPES)(Table1).

The IRPES measures and evaluates the multiple healthcare

performanceofpublichealthcareorganisations,fromﬁnancial

via-bilitytoqualityandpatientsatisfaction,throughasystematicand

publicly-disclosedbenchmarkingsystem[27].Since2008,

mem-bership of theIRPES hasbeen ona voluntarybasis, offering a

long-termbenchmarkingopportunity.Throughregularmeetings

andworkshops,theIRPESprovidesopportunitiesforresearch

col-laborationonspeciﬁctopics[28]andexchangewithkeyinformants

fromtheregions.

Ouranalysisincludes144websitesofpublichealthcare

organi-sations:namelyregions,LHAs,publichospitalenterprises,teaching

hospitals(THs)andnationalpublichospitalsforscientiﬁcresearch

(IstitutidiRicoveroeCuraaCarattereScientiﬁco-IRCCS,inItalian).

Theanalysisalsoincludesthepublichealthcareorganisations

thatstillhaveawebsiteeventhoughtheyweremergedwithother

healthcareorganisationsduringtherecentreorganisationofthe

regionalhealthcaresystem(Table1).

Inaddition,weanalysedthehealthcare/WTregionalportalsof

thefollowingregions:Apulia,Basilicata,FriuliVeneziaGiulia,

Lig-uria,Lombardy,Umbria,andVeneto.

Data collectionand analysisfocused ontheWTinformation

forspecialistvisitscoveredbythecurrentNationalPlan,aswell

asthedigital bookingand payment servicesforoutpatient

vis-its.Welikewiseinvestigatedwhetherandhowregionsfollowed

theGuidelinesonDigitalServiceDesignforthePublic

Administra-tion,publishedbytheAgencyforDigitalItaly(Agenziaperl’Italia

Digitale-AgID,inItalian)[29].

Threeresearchersindependentlyexploredthewebsites

follow-ingagriddesignedonthebasisoftheNationalPlan

recommenda-tionsandtheAgIDguidelines.Exploitingauser/citizenperspective,

datawerecollectedbysearchingforinformationonthewebsites

usingthegridfromSeptembertoDecember2018.

Anydisagreementsinthegridapplicationfordatacollection

weredebatedamongallauthorsuntilaconsensuswasreached.If

theauthorshadanydoubtswhenexaminingthecollecteddata,a

secondroundanalysisofthewebsitewasrun.

Thegridwasstructuredintothreesections:website

adaptabil-ity,WT information, and digital services (seethe Appendix for

details).

Thesectiononwebsiteadaptabilityassessesboththequalityof

browsing,irrespectivelyofthedeviceusedandtheresponsiveness

ofthewebpages(RWD)[30].Thewebinspectorfunctionalityof

browsers,suchasGoogleChrome,wasusedfortheRWDanalysis.

ThesectiononWTinformationwasdesignedtoassess:

䊉thepresence/absenceofWTdataandrelatedinformation;

䊉thelocationofsuchinformationonthewebsite;

䊉whethertheaccesstoWTinformationwasopentothepublicor

restrictedtosomegroups;

䊉whetherWTsforspecialistvisitslistedintheNationalPlanwere

disclosed;

䊉howWTsweremeasuredanddisclosed:themeasureadopted

(e.g. mean WT), whether the WTs were retrospective or

prospective,theupdatefrequency,theaggregationofdata(e.g.

healthcaredistricts),thesettingofcare(publicornot);

䊉whetherWTconsultationallowedforsimultaneouse-booking

foroutpatientservices;

䊉whetherthehealthcare organisationsalsooffereda platform

and/ormobileapplications(APPs)tosearchforWTinformation.

Thesectionondigitalservicesassessedtheofferingofe-booking

ande-paymentsfor specialistvisits,intermsofavailability and

modeofoperation.

Duringdatacollection,theonlinetextsregardingWTswere

col-lectedtoanalysetheirqualityusingREAD-IT(herecalledGlobal

READ-ITindex[31])andGulpEase[32].

TheGlobalREAD-ITindexwastheﬁrstreadabilityassessment

toolavailablefortheItalianlanguagebasedonNaturalLanguage

Processing(NLP)techniquesandmachinelearningalgorithms.It

assessesthereadabilityofdocumentsbycombiningtraditionalraw

textfeatureswithlexical,morpho-syntacticandsyntactic

informa-tion.Theindexrangesbetween0and100,theeasierthereadability,

thelowerthescore.

GulpEasewastheﬁrstindexdevelopedfortheItalianlanguage

based on traditional raw textfeatures (i.e. sentence and word

length).Itrangesbetween0and100-theeasierthereadability,the

higherthescore-withthethresholdof80highlightingatext

read-ablebylesseducatedpeople(primaryschool),60bylow-medium

educatedpeople(secondaryschool),and40bymedium-high

edu-catedpeople(highschool).

WhiletheGulpEaseindexisaproxyforthelexicalandsyntactic

complexityofatext,theGlobalREAD-ITindexcapturesdifferent

aspectsoflinguisticcomplexity,i.e.lexical(suchasmeasuresof

lex-icalrichness),morpho–syntacticfeatures(suchaslexicaldensity

andverbalmood)andsyntacticaspects(suchasorderingpatterns

ofsyntacticelements,structureofverbalpredicatesand

subordi-natefeatures).

The lexical complexity of texts was also assessed using a

vocabulary-based index,which refers tothedistribution ofthe

wordscontainedintheBasicItalianVocabulary (BIV),including

wordshighlyfamiliartonativeItalianspeakers[33].Thisindexis

calculatedasapercentageofadoptedwordsfromtheBIV,with

percentageshigherthan80generallysignallingatextwithahigh

levelofreadability.

Theregionalscoreforreadabilitywasgivenbythemeanvalue

obtainedbytheorganisationsineachregion.

Afterdatacollectionandthepreliminaryanalysis,twomeetings

andaworkshopwereorganisedtodiscussandvalidatetheresults

withregionalrepresentativesandlocalprofessionals,asexpertsof

thedomainand/orinformedonspeciﬁcfeaturesofWTsanddigital

services(Table2).KeyinformantswereidentiﬁedfromtheIRPES

networkofexperts.Duringthemeetings,collecteddatawere

dis-cussedandinterpretedandnewindicatorsoncommunicationand

(4)

Box1

MainfeaturesoftheNationalRegulatoryPlanforWaitingLists2019–2021.

GOALS

• MaximumWTguaranteeforselectedoutpatientvisits(n=14), outpatientdiagnostics(n=52),andelectivesurgeries(n=18).Healthcare serviceswereselectedconsideringareaswheretimelinessiskey (oncology,cardiovascular),ordemandishigh(specialistvisits),orwith hightechnologicalcomplexity(diagnostics),orwherethereisstill variabilityinaccessbetweenregions.

• Implementationofsinglebookingcentresandimprovementofonline bookingsystems.

• Guaranteeofbookingserviceswithoutinterruptiontoworkschedule. • Guaranteeoffullproductioncapacityofmedicaltechnologiesavailable

(e.g.MRIscan).

• ImprovementofWTpublicdisclosureatregionalandlocallevels(e.g. throughwebsites).

OVERALLMEASURESFORACHIEVINGSTATEDGOALS

• UseofprioritizationcriteriatomanageWTsbasedonclinicalcriteria andprofessionaljudgement.Fourlevelsofprioritywereidentiﬁed: urgent,shorttime,deferrable,andelectiveservice.

• Clearidentiﬁcationofﬁrstaccessandfollow-uppatientsanduseof dedicatedwaitinglists.

• Improvementofrecallsystemstoprevent“noshow”andintroduction ofﬁnancialpenaltiesfor“noshow”.

• Useofperformanceindicatorsforwaitinglistandwaitingtime monitoringatanationallevel.

• CreationoftheNationalObservatoryonWaitingListsforsupporting regionsandautonomousprovincesinimplementingandmonitoringthe effectiveapplicationoftheNationalPlanprovisions.

TheaboveareumbrellameasurespromotedacrossallItalianregional healthcaresystems.Eachregionalhealthcaresystemisalsorequiredto issuearegionalWTplandetailinglocalmeasuresandrequirementsfor achievingthegoalsoftheNationalPlan.

3. Results

Variabilitywithinandbetweenregions/APswasfoundinWT

informationdisclosure,onlineserviceavailabilityandtext

read-ability. The websites analysed showed differences in terms of

breadthanddepthofWTinformation,usabilityofsuchinformation

forbookingpurposesandtheirreadability.

3.1. Websiteoverviewandadaptability

Eachregion/APhasaninstitutionalwebsitewithahealthcare

section.Sevenregionsprovidedcitizenswithdedicatedportalson

healthcare/WTs(e.g.Veneto[34]).

Atthe local level, allLHAs have developed their own

insti-tutionalwebsite.However,websitelayout(i.e.dataorganisation

and location on thewebsite) and navigational design differed.

Someregions(e.g.Tuscany)adopted acommonregionallayout

anddesign,thusfacilitatingnavigationacrossdifferent

organisa-tionalwebsites.Otherregions-suchasBasilicata-optedforaless

homogeneousapproach.

Websiteadaptabilityanalysisshowedthatallthewebsitesare

adaptiveoratleastpresentaresponsivewebdesign.Citizensare

thusabletoconsultallthewebsitesusingbothacomputeranda

mobiledevice.

3.2. Waitingtimes

Morethan96%ofthewebsitespresentedasectiondedicated

toWTsforappointmentswithspecialists,withdescriptive

infor-mationontheWTs.FourorganisationsdisplayedWTdataintables

withnoexplanation.

More thanhalf of theorganisations (56%) showed WTs for

appointments with specialists in an online section devoted to

bureaucraticissues,called“TransparentAdministration”in

com-pliancewithItalianregulationsontransparency.Anothergroup

of organisations(30 %) showedWTinformation ontheir home

page.Theothers showedWTsin thecitizen/usersectionof the

webpage.Theanalysisrevealedthatabout32%ofthehealthcare

organisationsplacedWTinformationforappointmentswith

spe-cialistsoutsidetheirwebsite,eitheronaninter-organisationalor

regionalpages.Thelatterapproachwasadoptedby12outof16

publichealthcareorganisationsintheMarche[35]andall

health-careorganisationsinFriuliVeneziaGiulia[36].

WTinformationwasaccessibleforallusers,withfew

excep-tions.

Around30%oftheorganisationsprovidedWTswithout

detail-ingdatainaccordancewiththeNationalPlanprioritycodes.Around

one-thirddisplayedWTsonlybythreeprioritycodes:‘Shorttime’,

‘Deferrable’and‘Elective’.Onlyabout22%disclosedWTsusing

the fourpriority codes. A smaller percentage (13 %) presented

WTsbyprioritycodesdifferentfromthoselistedintheNational

Plan.Theremainingorganisationsuploaded WTdata

differenti-ated bysome combinationsof theNationalPlan prioritycodes

only.

Atleasttwo distincttypesofWTsbasedontwoinformation

ﬂowswere identiﬁed:1)“completedwaits”, i.e.a retrospective

lookatpatientswhohadalreadyreceivedcare,2)“expected

wait-ingtime”,i.e.aprospectivelookattheavailabilityofcarefornew

patients. However,theanalysisshowedthat the64%of public

healthcareorganisationsdidnotincludethesourceofdataused

tomeasureWTs.

WTstatisticsweredisplayedindifferentwaysandoftenmore

thanonemeasurewasprovided.MeanWTwasthemostfrequently

displayedmeasure(almost50%oforganisations),followedbythe

percentageofvisitsfallingwithintherequiredstandardmaximum

WT(nearly39%oforganisations)andbytheminimumWT(about

19%ofcases).

Other ways of measuring WTs included the maximum WT

(around15%ofcases)andﬁrstdateavailableforanappointment

withaspeciﬁcspecialist(almost12%ofcases).Morethan30%of

organisationsshowedWTdataaccordingtoothermeasures.For

example,inFriuliVeneziaGiuliaWTsweredisclosedas“estimated

waitingtimes”,calculatedonthebasisofthethirdavailableslot

foundwithasimulationbasedontelephonecallsthroughthe

book-ingcentre.However,inmorethan15%ofcasesWTstatisticswere

providedwithoutanyexplanationaboutthemeasureadopted.

Thedistributionofmeasuresusedbypublichealthcare

organi-sationstomonitorWTsisshowninFig.1.

WT statistics were updated with different frequencies. The

majorityofwebsites updateddataeverymonth(24%) orevery

quarter(18%).Inafewcasesthestatisticswererevisedeveryyear

(7%)oreverysixmonths(8%).Lessthan20%oforganisations

updatedthestatisticseveryweekoreveryday,makingWTs

avail-ableinreal-time.Aquarteroftheorganisationsdidnotstatethe

frequencyofupdates.

ThestudyrevealedthatWTdatawereprovidedaccordingto

dif-ferentlevelsofaggregation.Theminimumlevelwastheindividual

provider(e.g.laboratoryservices)andabout50%oforganisations

showedWTinformationaggregatedatthislevel.Approximately32

%ofcasesprovidedWTdatafortheorganisationasawhole(e.g.TH,

LHA).Atotalof16%displayedWTdatabylocalhealthcaredistrict

orbyAreaVasta(i.e.theentityappointedtocoordinateLHAsand

THsactionsinageographicalarea).Asmallpercentageof

health-careorganisationsdisclosedWTsaggregatedattheregionallevel

orbyDirectlyManagedHospital.

WTswerealwaysavailableforoutpatientspecialistcare

pro-videdbypublicpractices,butrarely(4%)formedicalstaffworking

privatelywithinapublichospital,eventhoughtheNationalPlan

requireshealthcareinstitutionstoprovidetheinformationforboth

practices.Elevenpercentoftheinstitutionsreportedthattheyonly

disclosedpublicpracticeWTs.Nevertheless,thisinformationwas

(5)

Fig.1.WaitingtimestatisticsinnineregionsandtwoautonomousprovincesofItaly,percentageoforganisationsshowingspeciﬁcWTtypeineachregion.

Approximately66%oftheorganisationsdidnotclearlyspecify

whetherWTdatareferredtopublicorganisationsonlyorincluded

privateaccreditedinstitutions. WTdisclosure for appointments

withspecialistsofferedbyprivateaccreditedproviderswas

sel-domdisplayed(26%ofcases).Only7%ofthewebsitesdeclared

whetherWTinformationreferredexclusivelytopublicproviders.

3.3. Digitalservices

Around63%oftheorganisationsenableduserstoconsultWTs

andsimultaneouslye-bookingforappointmentswithspecialists.

Inallthesecases,asynchronous/real-timeservicefore-booking

wasavailable.Another5%ofthehealthcareorganisationsallowed

citizenstobookavisitinanasynchronousway(i.e.theuserssend

anemailorﬁll-inaformonthewebsiteandtheyarelatercontacted

byanoperatortobooktheappointment),withoutthepossibility

toconsulttheWTsallatonce.

About21 %of thee-booking serviceswere provided bythe

individualorganisation,while50%weremanagedbytheregions.

Theauthoritiesin Marchedidnot provideane-bookingservice

andonlyafeworganisationsinLiguriaandTuscanyprovidedthis

serviceatthetimeofourstudy.Apulia,Basilicata,Bolzano,Friuli

VeneziaGiulia,Lombardy,TrentoandUmbriaofferedreal-time

e-bookingforappointmentswithspecialistforalmostallthepublic

providers.

Wealsocheckedwhetherappointmentswithspecialistscould

bepaidforonline.Tenoutoftheelevenregions/APsimplemented

onlinepaymentforthisservice.Withinthesetenregions,mostof

theorganisationsofferedthisdigitalservice,exceptforafewcases.

Atotalof75%oftheorganisationswerefoundtohaveAPPs,

offeringawiderangeofdifferentservices.Around20%of

organi-sationsalloweduserstopayforoutpatientcareviaanAPP.Inmore

than40%ofcases,e-bookingforappointmentswithaspecialist

wasavailableviaanAPP,for exampleApulia (PugliaSalute)and

Lombardy(Salutile)APPs.Onaverage,only50%ofthe14

outpa-tientappointmentswithspecialistslistedintheNationalPlanwere

availablefore-bookingusinganAPP.

Speciﬁcindicatorsonthedigitalisationofbookingandpaying

forhealthcareserviceswereintegratedintotheIRPES.

3.4. Readabilityanalysis

TheWTtextswereverydifﬁculttoreadbothatlexicaland

syn-tacticlevels.Toprovideanideaaboutthelinguisticcompetences

neededtounderstandthetexts,weusedtheGulpEaseindexasa

measuredesignedtotestthereadabilitywithrespecttotheuser

educationallevel(seeMethods).Thetextsissuedbysixregions

werehardtoreadforuserswithprimaryeducation,whilethose

issuedbytheotherregions/APsweredifﬁcultevenforcitizenswith

alowermiddleschooldiploma.TheGulpEaseindexrangesfrom0

(lowreadability)to100(highreadability)andtheWTtextsscores

werebetween50and74points(Fig.2.B),sotheirreadabilityis

“medium”.However,weexpectedthepublishedtexts-whichare

supposedtobeunderstoodbyawidevarietyofreaders-tohave

anoptimallevelofreadability,namelyascorehigherthan80,the

easy-to-readthresholdforpeoplewithaprimaryeducation.We

adoptedtheGlobalREAD-ITindex,areadability-indexrangingfrom

100(highdifﬁculty)to0(lowdifﬁculty),togathermoredetailed

informationonthelinguisticcomplexity.Fortheanalysedtexts,

theGlobalREAD-ITscoresrangedfrom98to70points(Fig.2.A).

Sixty-threepercentofthescoresrangedinbetween98 and92

points.

Theseﬁndingsindicatedthatthetextsshowcomplexlinguistic

characteristics,suchasmultiplesubclauses,complexverbal

pred-icate structures, non-canonicalorders of sentenceconstituents,

andembeddedsequencesofsubordinateclauses.Withrespectto

rawtextfeatures,thereadabilityoftheWTtextsvariednotonly

betweenregionsbutalsowithinthem.Forinstance,the

organisa-tionsinLombardyobtainedscoresrangingfrom99.59to1.78points

fortheGlobalREAD-ITindex,sincethedistributionoflinguistic

characteristicsaffectingreadabilityvariesgreatlyamongtexts.The

analysispointedoutthetextsarequitehardtoread.Concerning

thelexicalaspects,theuseofBIVwordsinthetextswassimilar

(6)

Fig.2. GlobalREAD-ITindex(A),GulpEaseindex(B),PercentageofBasic-Vocabulary words(C)forWTinformationtexts.

Duringthemeetingwiththeregionalkeyinformants,new

read-ability indicators onthe WT websitetexts were discussed and

adoptedintotheIRPES,namely:

• GlobalREAD-ITindex;

• GulpEaseindex;

• PercentageofBIVwords.

4. Discussion

While thepublic disclosure of healthcare performance data

impacts on healthcare systems by affecting the behaviour of

managersandprofessionalsoftenthroughreputationalpressure

[15–17], evidencesuggeststhat citizens rarelyusethepublicly

availabledata,and,whentheydo,ithasalimitedimpactontheir

decision-making[18,19].

Although Italian patients can choose healthcare providers,

wecannot normativelyargue thatpubliclyaccessible and

easy-to-understand WT information changes citizens’ behaviourand

shouldbeprescriptivelyconsideredasatoolforcitizens’

empow-erment or choice. Lay people shouldbe able to interpret such

informationandactonit[37].Thismeans,forinstance,thatpatients

shouldactasconsumers,thoughthisaspecthasnotbeenfoundby

theliteratureonpublicreporting[18,38].Nevertheless,citizensdo

showhighinterestinWTdata[6,7]andinformingpatientsonWTs

canpositivelyaffectpatientbehaviourinbettermanagingtheircare

pathway[6–8].

Our study highlights variability both within and between

regions/APsinItalywithregardtoonlineWTinformationfor

outpa-tientvisits,thereadabilityofthewebpagetextsandtheavailability

of concurrentdigital servicesthat citizensexpect toﬁndwhile

consultingwaitingtimesforselectedoutpatientservices.This

het-erogeneitycanbeahurdleforcitizens[39].

Overall,ourﬁndingsshowthereisroomforimprovingthepublic

reportingofWTdatainseveralaspects.

First,onlyonethirdoftheregionsinvestigatedinourstudyhave

WTinformationontheirhomepage,whilemostregionsusea

web-site sectiondevotedtobureaucraticissuesunderlyingthemere

administrativenatureofinformationreporting.Theeffectivenessof

suchcommunicationstyleisnothighintermsofcitizenawareness

[6].Inthisview,theuseofperformanceinformationproducesa

dis-tortionbecausewhatisactuallyvaluedisthepursuitofthetarget

(WTpublicdisclosure)ratherthantheintendedeffects

(account-abilitytowardscitizenstoimproveservicedelivery).Thiscanleave

theorganizationwithoutastimulusforimprovement,i.e.“hitting

thetargetbutmissingthepoint”[16,40].Wesuggestthatonline

WTreportingshouldbeeasilyavailableonthehomepage-since

thepublicinterestishigh-andshouldbelinkedtootherrelevant

onlinesectionsabouttheuseofsuchinformation.

Second,withregardtothetypeofWTinformation,thewebsites

showalowlevelofhomogeneityintermsofstatistics,frequency

ofupdatesanddataaggregation,oftensuggestingweaknessesin

theaccuracyandreliabilityofthedata.Therefore,citizenscannot

easilyuseWTdatatocompareproviders[11],monitorWTtrends

andgetinformationontheexpectedaccesstimeforselectedcare

visitsortreatments.

These ﬁndings highlightthat the harmonizationof WTdata

needstobeimprovedtogetherwiththeprecisionofdata. This

isparticularlyimportantwhendataisusedforexternalreporting

acrossorganizations,toallowbenchmarking.Indeed,

accountabil-ityis tiedtomeasurable,relevantand comparableindicatorsof

(7)

Third,theWTsshownonlineareoftennotrelevanttopatients,

sincetheyreportstatisticssuchasthepercentageofvisitsprovided

withintherequiredstandards,insteadofmeasuringtheactualWT

(i.e.duration),whichiswhatmattersforpatients[1].These

admin-istrativereportingformatsarenotveryhelpfulincommunicating

withcitizens.WTs needtobemade interesting,valid and

use-fultopatients,alsobylinkingWTswithothersigniﬁcantaspects

ofcare,forexamplebyprovidingreal-timeonlinebooking[11].

Currently,themajorityofItalianregionsorhealthcare

organisa-tionswereviewed,enableuserstoconsultWTswhenbookingan

appointmentwitha specialistinrealtime. Althoughthis digital

serviceisnotbuilttoprovideacomprehensivepictureofWTdata,

itenablespatientstoaccessanduseWTinformationfora

deci-sionprocess.Indeed,onlinebooking(i)reportsWTdatainaneasy

format,usuallywiththeﬁrstdateavailable;(ii)providestimely

accesstoWTsand(iii)isdirectlylinkedtotheoptionofbookingan

appointment.

Althoughwe cannot argue that thecombination of

easy-to-understand,up-to-dateWTdataandonlinebookingsystemscan

reallyempowercitizens,itcanenablethemto‘act’usingWT

infor-mationasoneofthecriteriainmakingachoice.Forthisreason,

wesuggestthatprovidingcitizenswithclear,relevantand

under-standableWTinformation,whilegivingthemtheopportunityto

chooseamongprovidersforappointments,canmakecitizensmore

informedandactive.

Fourth,ourstudyhighlightsthatreadabilityisakeymeasure

fortheefﬁcacyofpublicreporting[41,42].Thereadabilityanalysis

shows thatonline information aboutWTs is difﬁcult to

under-standfor thosewithlow andmedium levelsofeducation,thus

raisingequity issuesin termsofaccessanduseofonline

infor-mation.Besides constructionsof sentences related tolinguistic

complexity, the negative results of readability can be partially

explained by theabsence of a healthcare glossary for

measur-ingthetextreadability(i.e.speciﬁcterms,mostrecurrentwords),

thusoverestimatingthedifﬁcultyofthewrittennarrativesatthe

lexical level. In the light of theseﬁndings, we suggest a

thor-oughrevision of texts to improvethe readability,envisaging a

collaborative workbetweenspecialist writers and practitioners

whostayindirectcontactwithlaypeople,sotoﬁthealth-sector

speciﬁctermswithculturalandlinguisticaspectsoftheaverage

citizen.

Overall,ourﬁndingsrevealthatthepublicdisclosureofWTs

isvariableandfragmentedinItaly. Thissuggeststhatwhatthe

NationalPlanrecommendedintermsofonlinepublicdisclosure

ofWTshasnotyetbeenturnedintoausefulandusableinstrument

forcitizens.Someofthelimitationscouldbeovercomebyadopting

collaborativeandparticipatoryprocesses,involvingregional/local

healthcare organizations and inviting citizens to participate in

deﬁningareportingformat.AsetofperformanceindicatorsonWTs

isalreadyusedatnationalleveltoevaluatetheessentiallevelsof

care,andregionalad-hocanalysestoidentifyWTdeterminantsand

geographicalvariation are available[43,44].Nevertheless,these

performanceindicatorsarenotappropriateforapublicdisclosure

ofWTinformation,fromthepatientperspective.ProperWT

indi-catorsshouldbesharedandendorsedbycitizens,practitionersand

managers.

The publication of performance information should play a

keyrolein supportinghealthcare systemimprovementthrough

benchmarkingandreputation[45].Nonetheless,thereareseveral

instances where public disclosure generates pressure on

orga-nizations, which in somecases maybecome dysfunctional.For

example,thethreatofbeingnamed-and-shamedcancause

repu-tationalconcernsamongtheprovidersand-ifwelldesigned-can

improvetheperformanceofweakorganizationsdifferentlyfrom

the approaches based on command-and-control [16,41,46–49].

However,otherrisksarisewhenintroducingnaming-and-shaming

reformscombinedwithsanctionsandrewardsusingthetraditional

top-downapproaches.

ThisiswelldocumentedbytheNHSinEnglandwhichattempted

toreduceWTsthroughanaming-and-shamingregimecombined

withtargets-and-terror[15].First,thisapproachdoesnotstimulate

excellence,aboveallforprovidersthathaveanacceptable

perfor-mance:thereisnoshameinstayinginthemiddle[15,50].Second,

thetop-downapproach couldlead toa decreasein

responsibil-ityandengagementofhealthcareprovidersatthelocallevel[15].

Third,thehierarchicalapproachoftendoesnotallowﬂexibility,

andissuescanarisewhenpoliciesareimplementedatalocallevel

withoutanyadjustmenttolocalfeatures.

A social process of collegial benchmark competition, which

fostersthe identiﬁcationof best practices and continuous peer

learning, should be embraced when publicly reporting

perfor-manceinformation, especiallyin thehealthcaresectorwhere it

has been proven to improve performance. This is particularly

true when the rewards have reputational effects through the

publicreportingofperformance benchmarking(a sortof

‘repu-tationalcompetition’)andaredesignedforhighperformersonly,

without a ranking system with performance reported

accord-ingtomultiplecriteria[15,51].We encouragepolicymakersto

designWTreportingsystemsgroundedonbenchmarkingtoexploit

thereputationaldrivers,which have already beensuccessfulin

Italy.

Forexample,havinganationalweb-platformcanallowpublic

benchmarking among providers at different levels. The

Min-istryofHealthcouldexploitthepermanentNationalObservatory

on Waiting Lists to provide a common web-platform where

the regions/organisationscan homogeneously uploadWT data,

in addition to the already available local publicly-disclosed

data.

Our ﬁndings open up several new research questions. The

importanceofintegratingWTinformationwithotherindicators,

suchasthequalityof care,couldbeinvestigated.In fact,there

isevidenceabouttheoutput-distortioneffectinmeasuringWTs

withoutincludingqualitystandards[52].Moreover,ina

perfor-manceevaluationperspective,combininganumberofperformance

indicatorsacrossmultipledimensionscanavoidthefocustrickof

thenaming-and-shamingapproach,givenbyproviderstargettheir

energiestowardsimprovingasinglemeasurewhilelosingsighton

theotherindicators.

Additional more analytical studies could be run when

har-monisedandcomparableWTdatawillbeavailable.Itwouldbe

alsointerestingtotestwhetherthecomplexityofwrittentexts(in

termsofreadability)combinedwiththechosenWTmeasures(e.g.

averagetime,numberofpatientswaiting)isassociatedwith

dif-ferentlevelsofonlineaccesstoWTsandcitizens’usageofonline

booking.

Withthisrespect,thechoiceofaspeciﬁcWTdatavisualization

canalsobeenstudied,inordertoverifywhethertheregionsor

healthcareprovidershavemadeanyeffortsinmakingWT

infor-mationmeaningful,easytointerpretandusealsothankstodata

presentation.

Futureresearchcouldanalysethereasonsbehindthehigh

vari-ability and fragmentation of WT public disclosure in Italy. For

example,theimportancegivenbythehealthcareorganisationsto

thepublicdisclosureofWTs,andtherelateddigitalservicesbythe

healthcareorganisations,canplayakeyrole:thiscanbe

investi-gatedbystudyingwhethertheyusespeciﬁcmanageriallevers,such

asincentives[53,54].

Our article presents some limitations, such as the

non-representativesampleconsistingof tworegionsin thesouthof

Italy,threeinthecentreandsixinthenorth.Thisdidnotallow

ustocompletelymapthewholescenarioinItaly.Furthermore,a

(8)

thebestpracticewasoutofthescopeofthiswork.Nevertheless,

someresultsfromaqualitativein-depthstudyongoodpractices

foronlinedisclosureofhealthinformationinItalycanbefoundin

DeRosisandcolleagues[55,56].

5. Conclusions

PubliclyreportingWTs onwebsites isexpectedtofosterthe

transparencyandaccountabilityofthehealthcaresystemandto

achieve performance improvements thanksto thecomparisons

amongprovidersandamoreinformedchoiceforcitizens.

Whilecitizensmaybevery interestedin WTdata,thereare

stillseveralsigniﬁcantbarriersinItaly:thelowreadabilityofWT

onlineinformation,thenon-standardandnoteasy-to-ﬁnd

loca-tionofWTdataonwebsites,thevariabilityinparametersandthe

levelofaggregationadoptedtoreportWTstatistics.Theseﬁndings

suggestmosthealthcareorganisationsinItalyinterprettheonline

publicdisclosureofWTsasmerelycompliancewiththelawrather

thananopportunityforinterfacingwiththepublic.

SeveralregionshaveadoptedpoliciesregardingWTdata

inte-grationwithine-bookingservices.Thistrendcanactasa driver

toreducesomebarriersforcitizensinaccessing,understanding,

contextualizingandusingWTdataformanagingtheircarepaths.

IntegratingonlinebookingwithinWTpublicdisclosuresystems

canbeaneffectivecommunicationchanneltowardscitizens,when

designedtobetransparent,accessible,readableandusable.

Funding

Thisresearchdidnotreceiveanyspeciﬁcgrantfromfunding

agenciesinthepublic,commercial,ornot-for-proﬁtsectors.

DeclarationofCompetingInterest

Theauthorsdeclarethattheyhavenoconﬂictsofinterest.

Acknowledgements

ThisstudywassupportedbytheItalianRegionalPerformance

EvaluationSystem(IRPES)steeredbyLaboratorioManagemente

Sanità,InstituteofManagement,ScuolaSuperioreSant’AnnaPisa

(Italy).TheauthorsacknowledgeProfessorSabinaNutiandallthe

researchersfromLaboratorio ManagementeSanità, Instituteof

Management,ScuolaSuperioreSant’Anna,fortheirconstant

sup-portand useful comments.The authorsaregrateful toMichela

Cassanoforthedatacollection.Theauthorswouldalsoliketothank

allIRPESregionaladministratorsandtheirstaffforparticipatingin

thead-hocmeetingsoncommunicationinhealthcare.

AppendixA. Supplementarydata

Supplementarymaterialrelatedtothisarticlecanbefound,in

theonlineversion,atdoi:https://doi.org/10.1016/j.healthpol.2020.

05.012. References

[1]SicilianiL,BorowitzM,MoranV.Waitingtimepoliciesinthehealthsector: whatworks?OECDHealPolicyStudies2013:1–328.

[2]SicilianiL,HurstJ.Tacklingexcessivewaitingtimesforelectivesurgery:a com-parativeanalysisofpoliciesin12OECDcountries.HealthPolicy(NewYork) 2003;72(2):201–15.

[3]WillcoxS,SeddonM,DunnS,EdwardsRT,PearseJ,TuJV.Marketwatch- mea-suringandreducingwaitingtimes:across-nationalcomparisonofstrategies. HealthAffairs2007;26(4):1078–87.

[4]Conner-SpadyB,EsteyA,ArnettG,NessK,McGurranJ,BearR,etal. Deter-minantsofpatientandsurgeonperspectivesonmaximumacceptablewaiting

timesforhipandkneearthroplasty.JournalofHealthServicesResearch&Policy 2005;10(2):84–90.

[5]HealthCanada.Finalreportofthefederaladvisoronwaittimes[Internet];2006. p.76p.Availablefrom:http://www.hc-sc.gc.ca/hcs-sss/pubs/system-regime/ 2006-wait-attente/index-eng.php.

[6]BruniRA,LaupacisA,LevinsonW,MartinDK.Publicviewsonawaittime man-agementinitiative:amatterofcommunication.BMCHealthServicesResearch 2010:10.

[7]EdwardsRT,BolandA,WilkinsonC,CohenD,WilliamsJ.Clinicalandlay pref-erencesfortheexplicitprioritisationofelectivewaitinglists:surveyevidence fromWales.HealthPolicy(NewYork)2003;63(3):229–37.

[8]Hui MK,Zhou L. Howdoes waiting duration informationinﬂuence cus-tomers’reactionstowaitingforservices?JournalofAppliedSocialPsychology 1996;26(19):1702–17.

[9]Fitzsimons D, Parahoo K, Stringer M. Waiting for coronary artery bypass surgery: a qualitative analysis. Journal of Advanced Nursing 2000;32(5):1243–52.

[10]CrossE,GoodacreS,O’CathainA,ArnoldJ.Rationingintheemergency depart-ment:thegood,thebad,andtheunacceptable.EmergencyMedicineJournal 2005;22(3):171–6.

[11]StoopAP,VrangbækK,BergM.Theoryandpracticeofwaitingtimedataas aperformanceindicatorinhealthcare:acasestudyfromTheNetherlands. HealthPolicy(NewYork)2005;73:41–51.

[12]CashDW,ClarkWC,AlcockF, DicksonNM,EckleyN,GustonDH, etal. Knowledgesystemsforsustainabledevelopment.ProceedingsoftheNational AcademyofSciencesoftheUnitedStatesofAmerica2003;100(14):8086–91.

[13]VanDoorenW,BouckaertG,HalliganJ.Performancemanagementinthepublic sector,secondeditionRoutledge;2015,208p.

[14]HamiltonJT.Regulationthroughrevelation:theorigin,politics,andimpacts ofthetoxicsreleaseinventoryprogram.Regulationthroughrevelation:the origin,politics,andimpactsofthetoxicsreleaseinventoryprogram.Cambridge UniversityPress;2005.p.1–347.

[15]BevanG,EvansA,NutiS.Reputationscount:whybenchmarkingperformance isimprovinghealthcareacrosstheworld.HealthEconomics,PolicyandLaw 2019;14(2):141–61.

[16]BesleyTJ,BevanG,BurchardiKB(January2009).Availablefrom:Namingand shaming:theimpactsofdifferentregimesonhospitalwaitingtimesinEngland andWales;2009http://eprints.lse.ac.uk/33775/.

[17]NutiS,VolaF,BoniniA,VainieriM.Makinggovernanceworkinthehealthcare sector:Evidencefroma“naturalexperiment”inItaly.HealthEconomics,Policy andLaw2016;11(1):17–38.

[18]MarshallMN,ShekellePG,LeathermanS,BrookRH.Thereleaseof perfor-mancedata:whatdoweexpecttogain?Areviewoftheevidence.JAMA 2000;283(14):1866–74.

[19]MannionR,GoddardM.Publicdisclosureofcomparativeclinicalperformance data:lessonsfromtheScottishexperience.JournalofEvaluationinClinical Practice2003;9(2):277–86.

[20]Deloitte, Available from: Global mobile consumer trends [Internet]; 2017 https://www2.deloitte.com/global/en/pages/technology-media-and-telecommunications/articles/gx-global-mobile-consumer-trends.html. [21]ParéG,TrudelMC,ForgetP.Adoption,use,andimpactofE-bookinginprivate

medicalpractices:mixed-methodsevaluationofatwo-yearshowcaseproject inCanada.JournalofMedicalInternetResearch2014;16(9):1–13.

[22]EllingsenG,ObstfelderA.Collectiveexpectations-Individualaction implement-ingelectronicbookingsystemsinNorwegianhealthcare.InternationalJournal ofMedicalInformatics2007;76(SUPPL.1):S104–12.

[23]CromwellDA,GrifﬁthsDA,KreisIA.Surgerydot.com:thequalityofinformation disseminatedbyweb-basedwaitingtimeinformationservices.MedicalJournal ofAustralia2002;177(5):253–5.

[24]VenturiG,BellandiT,Dell’OrlettaF,MontemagniS.NLP–basedreadability assessmentofhealth–relatedtexts:acasestudyonitalianinformedconsent forms;2015.p.131–41.

[25]TediosiF,GabrieleS,LongoF.Governingdecentralizationinhealthcareunder toughbudgetconstraint:WhatcanwelearnfromtheItalianexperience? HealthPolicy(NewYork)2009;90(2–3):303–12.

[26]FerreF,Belvisde,AGiuli,ValerioL,LonghiS,LazzariA,etal.Italy:healthsystem review.HealthSystemTransition2014;16(4):1–168.

[27]NutiS,NotoG,VolaF,VainieriM.Let’splaythepatientsmusic:anewgeneration ofperformancemeasurementsystemsinhealthcare.ManagementDecision 2018;56(10):2252–72.

[28]SeghieriC,BertaP,NutiS.GeographicvariationininpatientcostsforAcute MyocardialInfarctioncare:insightsfromItaly.HealthPolicy2019;123:449–56.

[29]Agenziaperl’Italiadigitale.LineeguidadidesignperiserviziwebdellaPubblica Amministrazione;2019.

[30]Adaptivevs.responsivedesign[internet];2019[cited2019Oct22]. Avail-ablefrom: https://www.interaction-design.org/literature/article/adaptive-vs-responsive-design.

[31]Dell’OrlettaF,MontemagniS,VenturiG,ComputazionaleL,ZampolliA,Cnr ILC,etal.READ–IT:assessingreadabilityofItaliantextswithaviewtotext simpliﬁcation.ProceedingsoftheSecondWorkshoponSpeechandLanguage ProcessingforAssistiveTechnologies(SLPAT’11)2011:73–83.Edimburgo,UK, 30Luglio.

[32]PietroLucisanoeMariaEmanuelaPiemontese.GULPEASE:unaformulaperla predizionedelladifﬁcoltàdeitestiinlinguaitaliana;1988.

[33]DeMauroT,Availablefrom:Nuovovocabolariodibasedellalinguaitaliana; 2016.p.1–18www.dizionario.internazionale.it/nuovovocabolariodibase.

(9)

[34]PortaleSanitàRegionedelVeneto[Internet];2019[cited2019Oct22]. Avail-ablefrom:https://salute.regione.veneto.it/web/rov.

[35]ASURMarche-AziendasanitariaUnicaregionale[Internet];2020.Available from:https://www.asur.marche.it/.

[36]RegioneAutonomaFriuliVeneziaGiulia-Veriﬁcadeitempidiattesae preno-tazioni;2020.Availablefrom:https://servizionline.sanita.fvg.it/prenotazioni/ #/index.

[37]BerwickDM,JamesB,CoyeMJ.Connectionsbetweenqualitymeasurementand improvement.MedicalCare2003;41(Suppl.1):30–8.

[38]FungCh,LimYw,MattkeS,DambergC,ShekellePg.Systematicreview:the evidencethatpublishingpatientcareperformancedataimprovesqualityof care.AnnalsofInternalMedicine2008;148(2):111–23.

[39]FattoreG,FerrèF.ItempidiattesaperleprestazionidelSSN:statodell’artee alcuneriﬂessioni.In:RapportoOASI2012.EGEA;2012.p.109–65.

[40]BevanG,HoodC.What’smeasuredis whatmatters:targetsandgaming intheEnglishpublichealthcaresystem.PublicAdministration2006;84(3): 517–38.

[41]SchneiderEC,EpsteinAM.Useofpublicperformancereports:asurveyof patientsundergoingcardiacsurgery.JAMA1998;279(20):1638–42.

[42]HibbardJH,StockardJ,TuslerM.Doespublicizinghospitalperformance stim-ulatequalityimprovementefforts?HealthAffairs2003;22(2):84–94.

[43]NutiS,VainieriM.Managingwaitingtimesindiagnosticmedicalimaging.BMJ Open2012;2(6):e001255,http://dx.doi.org/10.1136/bmjopen-2012-001255. [44]LunguDA,GrilloRuggieriT,NutiS.Decisionmakingtoolsformanagingwaiting

timesandtreatmentratesinelectivesurgery.BMCHealthServicesResearch 2019;19(1):369,http://dx.doi.org/10.1186/s12913-019-4199-6.

[45]GusmanoMK,HeldermanJK.Frontiersofhealthpolicyresearch.Health Eco-nomics,PolicyandLaw2019;14(2):135–40.

[46]HibbardJH.Whatcanwesayabouttheimpactofpublicreporting? Incon-sistentexecutionyieldsvariableresults.EditorAnnalsofInternalMedicine 2008;148(2):160–1.

[47]FungA,O’RourkeD.Reinventingenvironmentalregulationfromthe grass-rootsup:explainingandexpandingthesuccessofthetoxicsreleaseinventory. EnvironmentalManagement2000;25(2):115–27.

[48]Bevan G, HamblinR. Hittingand missing targetsby ambulance services foremergencycalls:effectsofdifferentsystemsofperformance measure-ment within the UK. Journal of the Royal Statistical Society: Series B 2009;172(1):161–90.

[49]ChassinMR.Achievingandsustainingimprovedquality:lessonsfromNew Yorkstateandcardiacsurgery.HealthAffairs2002;21(4):40–51.

[50]OliverA.Dountoothers:ontheimportanceofreciprocityinpublic adminis-tration.AmRevPublicAdm2017;48(4):279–90.

[51]FreyB.Howshouldpeopleberewardedfortheirwork?In:Oliver,abehavioural publicpolicy.Cambridge:CambridgeUniversityPress;2013.p.165–163.

[52]ElkomyS,CooksonG.Performancemanagementstrategy:waitingtimeinthe englishnationalhealthservices.PublicOrganizationReview2018;20:95–112.

[53]DeRosisS,VainieriM.IncentivizingICTinhealthcare:acomparative analy-sisofincentiveschemesinItalianRegions.InternationalJournalofHealthcare Management2017;10(1):1–12.

[54]VainieriM,LunguDA,NutiS.Insightsontheeffectivenessofrewardschemes from10-yearlongitudinalcasestudiesin2Italianregions.InternationalJournal ofHealthcarePlanningManagement2018;33(2):e474–84.

[55]De RosisS,GuidottiE,Zuccarino S.L’esperienzadellaregionePuglia. In: Dall’individuazioneallavalorizzazionedellebestpracticeL’esperienzadel sis-temadivalutazionedeiservizisanitariregionaliasupportodelladiffusione dell’eccellenza;2019.p.183–9.

[56]DeRosisS,FerreF,GuidottiE,ZuccarinoS.Digitalizzazionenellaprenotazione epagamentodellevisitespecialistiche.In:Dall’individuazionealla valoriz-zazionedellebestpracticeL’esperienzadelsistemadivalutazionedeiservizi sanitariregionaliasupportodelladiffusionedell’eccellenza;2019.p.175–82.