How to set challenging goals and conduct fair evaluation in regional public health systems. Insights from Valencia and Tuscany Regions

Please cite this article in press as: Vainieri M, et al. How to set challenging goals and conduct fair

eval-Contents lists available atScienceDirect

Health

Policy

j o u r n a l h o m e p a g e :w w w . e l s e v i e r . c o m / l o c a t e / h e a l t h p o l

How

to

set

challenging

goals

and

conduct

fair

evaluation

in

regional

public

health

systems.

Insights

from

Valencia

and

Tuscany

Regions

Milena

Vainieri

_,

_Federico

_Vola

_,

_Gregorio

_Gomez

_Soriano

_,

_Sabina

_Nuti

b_{ConselleriadeSanidadUniversalySaludPública,ValenciaRegion,Spain}

a

r

t

i

c

l

e

i

n

f

o

Articlehistory: Received11August2015

Receivedinrevisedform9September2016 Accepted18September2016

Keywords: Incentives Targets Healthcaresector Benchmarking

a

b

s

t

r

a

c

t

Thedefinitionof“therighttargets”andthewaytheevaluationofresultsisperformed affectthewillingnesstocommittonewchallenges,whichisafactorthatinfluencesthe relationshipbetweengoalsettingandperformanceresults.Indeed,someauthorsclaim thatthechoiceofaninappropriategoal-settingprocedureisamajorcauseoffailureof managementcontrolsystems.Goalsettingtheoristsfoundthatassigningaspecificand challenginggoalleadstohigherperformancethan(a)aneasygoal,(b)ageneralgoalor(c) nogoalsetting.Despitethisevidence,yet,fewproposalsconcernthedefinitionofwhatis “challenging”.Thispaperfocusesontwoissues:(a)whatistobeconsideredachallenging goaland(b)whatisa“fairevaluation”inthehealthcaresector.Thisworksuggeststhat benchmarkingisavalidsupporttosolvethepreviousdilemmas.RelyingontwoRegional Europeanadvancedexperiences–ValenciainSpainandTuscanyinItaly–,thispaperaims toprovideconceptualmethodsthatcanhelpmanagersdefinechallenginggoalsand con-ductfairevaluationabouttheirachievement.AlthoughtheseRegionsadopteddifferent governancemodels,bothofthemappliedverysimilartechniques,whichseemtobe asso-ciatedtoanimprovementoftheirperformanceandareductionofunwarrantedvariation. ©2016TheAuthors.PublishedbyElsevierIrelandLtd.Thisisanopenaccessarticleunder theCCBY-NC-NDlicense(http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction

Goalshavepervasiveinfluenceonemployees’ behav-iorandinturnonorganizationalperformance.Thisbasic assumptionofgoalsettingtheory–developedbyLocke andLathamattheendofthe80sfortheindividuallevel–, hasbeenanalyzedfortheorganizationandsystemlevels bycontrolmanagementscholars.Literatureandexperience

∗ Correspondingauthorat:LaboratorioManagementeSanità,Istituto diManagement,ScuolaSuperioreSant’AnnadiPisa.PiazzaMartiridella Libertà,33.56127Pisa.

E-mailaddress:m.vainieri@sssup.it(M.Vainieri).

ongoalsettingshowedthatassigningtargetsisnot suffi-cient.Forinstance,theexperienceofHealthforallprogram, launchedbytheWHOinmid-80s[1],thatsettargetsto memberstatesandrenewedtheminthemid-1990swith theHealth21policyframework[2],flawedinsome coun-triesandinsomeareas[3].Scholarsthatanalyzedthiscase [4]statedthatsomestrategieswerenotmetbecauseof: thelackofinvolvementofkeyactorsatthegrass-roots lev-els;theshiftofpowerandresponsibilitiesfromthecentral totheregionallevel[5];thelackofthe“righttargets”in termsofprioritization,reflectingthespecificityof coun-triesandintermsofidentificationofthecorrectefforttobe required.Alltheseelementsarealsofoundingeneral litera-tureonperformancemanagement[6].Indeed,theadoption http://dx.doi.org/10.1016/j.healthpol.2016.09.011

0168-8510/©2016TheAuthors.PublishedbyElsevierIrelandLtd.ThisisanopenaccessarticleundertheCCBY-NC-NDlicense(http://creativecommons. org/licenses/by-nc-nd/4.0/).

ofaninappropriategoal-settingprocedureisdeemedtobe amajorcauseoffailureofmanagement controlsystems [7].

Scholars of goal-setting theory stated that effective goalsshouldbeassigned consideringthecontent(what havetobesought)and theintensity(howtoattain the goal)[8].Regardingthehealthcaresector,bothcentraland regionallevelsusetargetsintheirgovernancemodelsin differentways,gettingdifferentresults[9–11].

Atthis purpose, Brownet al. believe that successful healthcaresystemshave:apublic,specificstatementof goalswithaplanforreachingthesegoals;apublicreport ofimprovementresultsandstrongphysicianandclinical leadershipin improvement efforts, aligned to improve-mentgoals(again,supportedbyusefuldata)[12].

Inthis scenario, controlmanagement studies mainly discussedwhichindicatorsshouldbeselected,thecriteria tochoosethem[13–16]andsomeelementsoftheprocess, inparticulartheimportanceoffeedbackandinvolvement [6,17,18].Whengoalis specificand challenging,itleads tohigherperformancethan(a)aneasygoal,(b)ageneral goaloranexhortationto“doone’sbest,”or(c)nogoal set-ting[8,19–24].Yet,fewevidenceandproposals concern thedefinitionofwhatis“challenging”,thatisanimportant characteristicgoalsshouldhavetomotivateworkers[25]. Thedefinitionof“therighttargets”and thewaythe evaluationofresultsisperformedaffectthewillingnessto committonewchallenges,whichisafactorthatinfluences therelationship between goal setting and performance results[25].Seekingtorespondtotheaforementionedtwo openissues, it is possible to identifyat leastfour sub-decisionsmanagersandpolicymakersneedtotakewhen theysetandevaluatetargets:

i.Whethertodefinethebenchmarktheactorsareaiming at;

ii.Whethertosethomogeneoustargetsforalltheactors; iii.Whethertoconsidertheagents’pastandrelative

per-formancestosettargets;

iv.Whethertoadjustresultsonthebasisofenvironmental factors.

Goalsettingprocedureneedstoconsiderwhetheragold standardoranormativetargetexist(i).Whenneitherthe goldstandard,northenormativestandardexists,thenthe definitionofthetargetsoftenrequireasubjectivedecision. Thissituationcanjeopardizethelegitimationofgoals. Nev-erthelessoncethestandardisdefined,policy-makershave todecidewhethertoassignthesametargettoallunits (i.e.,healthauthorities,healthdepartmentsor profession-als)(ii).Homogeneousgoalsareoftenassignedtoallunits. Thisdecisionencounterssomedrawbacks.Thefirstone occurswhenthegoalisset,foreveryunit,tothegold stan-dard.Thegoldstandardrequiresextremeeffortforsome agentssothatitcanbeperceivedasunattainable. Seem-inglyimpossiblegoalscanhavetwooppositeeffectsknown as“theparadoxofstretchgoals”.Stretchgoalscould influ-enceorganizationallearningandperformanceinapositive waybyfacilitatingimprovementbecausetheyare seduc-tive,buttheycanalsohaveadisruptiveeffectleadingtono commitmentatall[25,26].

The second drawback is what managerial literature defines the“threshold effect”. Thisoccurswhen a min-imal and equal threshold is set for all the controlled actors.Ontheoneside,thismechanismputssome inten-tionalpressureonunder-performingagents;ontheother side, it instills a perverseincentive for all those agents whoarealreadyperformingoverthethreshold,by stim-ulatingaregressiontowardthethresholdlevel[27].The thresholdmechanismgenerallypenalizesthoseactorsthat perform well but still have single criticalities, while it favorsmediocreagents,whosystematicallyperforminthe thresholdrange.Toovercometheseproblems,individual goalscanbepreferred.

Whenpolicy-makershavetosetindividualgoalsorthey donothavethegoldstandard,awaytosettargetsis con-sideringthepastandrelativeperformanceofagents(iii). Indeed,previousstudiesdemonstratethatgoalshavetobe setconsideringthedifferencebetweentheunitsandtheir startingpoint(baseline)[6,28].Performanceincentiveshad thegreatestimpactonproviderswhoseperformancewas loweratbaseline[29]sothatpolicy-makerscouldaskmore totheworst-performers,consideringthattheeffortshould beperceivedaschallengingbutattainable.Indeed, disrup-tiveeffectsseemtobemorefrequentinthoseorganizations whoserecentperformancewaslow[26].

Inlaboratoryexperiments(largelyappliedinthegoal settingtheory)challenginggoalsareusuallyconsideredto bethosethatarefixedatthe90thpercentileofthe orig-inaldistribution,whileinfieldexperiments“challenging” iswhatagentsperceiveas“difficultyetattainable”goals [25].Thatimpliesthatthedefinitionofwhatischallenging isset,mostofthetime,onsubjectivebasis.

Finally,theevaluationoftheleveloftargetattainment byeachagent(iv)cancorrespondtothesimpledegreeof achievementofthesettargets,butotherfactorsneedtobe considered.Inparticular,somecontextualvariablesmight haveaffectedthedegreeofachievementitself.Thismeans thatsomecorrectiveshavetobeenvisaged[6,8,30].

This paper supports thethesis that theintroduction ofsomebenchmarkingtechniquesmightbethesolution tofacethefourabove-mentionedissues.Indeed, bench-markingtechniqueshavebeenappliedinthepublicsector sincethe1990s[28],becomingthebasisforthe develop-mentofmanagementcontrolsystemsasdominantform ofgovernanceinthehealthcaresector[9–11].The follow-ingparagraphsreporttheconceptualframeworkdrawnby twoEuropeanexperiences–ValenciainSpainandTuscany inItaly–,thatsuggesthowbenchmarkingtechniquescan beleveragedtosetappropriate targetsand conductfair evaluationoftheirachievement.

2. Methodology

The paper offers a comparison of the methods two regionalinstitutions–Tuscany(Italy)andValencia(Spain) –independently developedtosetappropriate targetsto theirhealthcareunitsandtoassesstheirattainment.The studyistheresultofalongitudinalactionresearch pro-cess.The actionresearch approachis a researchmethod thataimstosimultaneouslysolve‘real’problemsinsocial systemsandcontributetothebasicknowledgeofsocial

Please cite this article in press as: Vainieri M, et al. How to set challenging goals and conduct fair eval-science.Thedistinctivestrongholdofactionresearchisthat

theresearchersareinvolvedalongtheflowoflifeofthecase organization,inclosecollaborationwithitsmembers.This hasgoodpotentialforproducingbothpracticallyrelevant andtheoreticallyinterestingcontributions[31–35].

Thefirstoutputofactionresearchisanideaforchange or a design of a solutionto theproblems faced by the hostorganization,bothofthesetypicallyjointlydeveloped with themembers of theorganization. In practice,this usuallymeanstheresearcher’sparticipationinaproject team in charge of takingcare of a changeproject [36]. Actionresearchalsoincludesthetestingoftheideasfor change,typicallybyteamingupwiththemembersofthe hostorganizationandbysupportingtheimplementation ofnewsolutions.Hence,organizationalchange(oratleast anattempttoaccomplishthat)isanimportantoutputof thiskindofstudydesigns.Thisclosecollaborationenables thecollectionofresearchmaterialthatcannotbeusually retrievedbyotherapproaches[31].

Theauthorsofthispaperdirectlycooperatewiththe regionalpublicagenciesthatsettargetsforthehealthcare authorities(forTuscany)andforthehealthcare profession-als(forValencia).Inbothcases,thespiralactionresearch processsuggestedbyBerg[37]wereapplied.Researchers participatedintheperiodicalmeetingsorganizedbythe institutionalbodiesinchargeofthedefinitionand evalu-ationofthehealthcareunitsgoals;thosemeetingswere summarizedininternalreportsandinregionalpublicacts, while single meetingsand exchangeswere recorded in researchers’notes.

WithreferencetotheTuscanycase,LaboratorioMESof ScuolaSuperioreSant’Anna(Pisa)hasbeenactively collab-oratingwiththeregionalhealthcareadministrationsince 2004.Inparticular,researchershavebeencooperatingwith thehealthcareregionaldepartmentindefiningamethod tosetappropriategoalstotheregionalhealthauthorities, quarterlymonitoringthetargetedindicatorsandinhelping evaluatethegoals’achievement.

RegardingValenciaRegion,datacollectionwasmade possiblebythedirectinvolvementofoneoftheauthorsin thedailyactivityoftheAgenciaValencianadeSalud(AVS) asChiefInformationOfficer.

Thankstotheirdirect involvement,theauthorshave beenabletocollectfirsthandresearchmaterialaboutthe proceduresthetwoinstitutionsadoptedtosetchallenging goals,toobservetheapplicationoftheproposedmethod andtointervenealongtheprocessfromthetargets def-initiontotheassessment phase. Quali-quantitativedata arethereforethemainoutputofthelong-time collabora-tionbetweentheauthorsandtheTuscanyRegion/Agencia Valenciana de Salud. In particular, the analyzed period for Tuscany goes from2007 todate while for Valencia the period runsfrom 2007to 2011.Data and informa-tion drawn from action research are complemented by regionalpublicactsandregionalinternalreportsfor Tus-cany; regional strategic plans and internal reports for Valencia.

Theepistemologicallimitationsofactionresearchhave beenthoroughlyinvestigated[38].Weacknowledgethata twocasecomparisonlackingaformalizedprocedureto col-lectandinterpretdatamayraisesomerelevantconcerns

aboutinternalandexternalvalidity.Nonetheless,thetopic this paper inquires remains largely unexploredand we believethatourstudycanmakeanimportantcontribution bothintermsofscientificnoveltyandpolicyimplications.

3. Thetechnicalframeworkforsettingtargetsand assessingperformance

Beforeinvestigatingthespecificexperiencesofthetwo casestudies,wedescribethetechnicalframeworkthose experiences needtobecontextualizedin. Thetwo case studiesactuallysharethesamemethodologytoset chal-lenginggoalsandassesstheirachievement.Theprocess isjointlybasedonsystematiccomparison(benchmarking) andoninformationonpastperformance.

Themethodfollowsareasonableheuristicratherthan statisticalmethods(suchasdataenvelopmentanalysis).In ordertoeasetheprocessandtostreamlinethe commu-nicationtowardstakeholders.Themethodcanbedivided intwophases:(1)settinggoalsand(2)assessing perfor-mances.

Forbothphases,regionalpolicymakersandmanagers havetoidentifytheappropriatekeyperformancemeasures (assuggestedforinstancebyRef.[15]),representingthe goalandthegroupofpeerunits(unitswithsimilarmission, suchasteachinghospitalsorfocusedhospitals).

Theessentialingredientsofthemethodaretwo:(1)past performancemeasureswhichrepresentthebaselinesfor improvement;(2)therelativeperformance.Themethod workswithgoalsexpressedinquantitativeterms. Indica-torsshouldbeeasilymeasurablewithanexplicitandclear formula.

Consideringthetargetsettingphase,literatureshows thattargetshavetobesetonthebasisofpreviousoractual results(thebaseline),byaskingforaninverseeffortrelated tothegoalstandardorthebestperformers:[6,28,39]the greater improvement is required to the units with the poorer performance, whilsta less challenging improve-menttothosethatalreadyregisteredagoodperformance. Thiscanbedonebyexecuting,foreachindicator,the followingsteps:(1)orderingthecomparableunitsonthe basisoftheirbaseline(pastoractualperformance);(2) set-tingthetargetsoftwounits;(3)drawingthelinebetween thetwotargets;(4)calculatingtheexpectedvaluesonthe basisoftheline;(5)proposingthetargetstothegeneral managersoftheunits,tofine-tunethetarget,accordingto localpeculiarities.

Step2isacrucialphase.Thetwocaseswestudied con-sidertwooptions:(a)settingthetargetsofthebestand theworstperformersor(b)settingthetargetsoftheworst performersandthemedianvalue.

A-optionisusedwhenthereisagoldstandardthatthe bestperformermustachieveorthereistheintentionto askthebestperformerforholdingitsposition.B-optionis adoptedwhenthereisnogoldstandard.

FortheA-option,thegoldstandard(derivedfrom liter-atureornational/regionalplans,suchasthevaccination coverage) canbe used as a referencefor thebest per-former(s). It is possible to ask the worst performer to improveuptothe25thpercentile(ortodecreasetothe 75thpercentile,ifthelowerthevaluethebetter).Inthis

Gold

stand

ard

Minimum Observa

on

(worst per

former)

Target

for the

worst

performer: 36.5

°

percenle

Best performer

to

the

gol

d

stand

ard

Minimum Observa

on

(worst per

former)

Target

for the

worst

performer: 36.5

° pe

rce

nle

Best per

former

holds the

posion

or the media

n

shis

towards the 75

° pe

rce

nle

A-opon

B-opon

Fig.1. Thereferencepoints,forindicatorsthathavetoincrease(thehigheristhebetter). way,itisrequiredtotheworstperformertobehaveasthe

1/4oftheunitsregisteringaperformancelowerthanthe median,asshowedinFig.1.

Ifthereisnogoldstandard,thentheB-optionis pre-ferred.In this case, thetwo expected values are set to theworstand themedian values.Inparticular,theunit reportingtheworstvalueisassignedatargetsuchasinthe A-option:toincreaseupthe25thpercentileortodecrease tothe75thpercentile(ifthelowerthevaluethebetter). Regardingthesecondexpectedvalue,theRegion(orthe centralgovernment)couldthinktoshiftthemediantothe 75thpercentile(ortothe25thpercentileforindicators whosevalueisexpectedtodecrease),asreportedinFig.1. Oncethetwoexpectedtargetsareset,twodotshave beenidentified:forincreasingindicator,theactual perfor-manceofthebestpracticeandthegoldstandard(orthe actualmedianandthe25thpercentile)isoneofthedots, whiletheotherdotistheactualperformanceoftheworst practiceandthe36.5thpercentile.Theexpectedvaluesof theotherunitscanbeinferredusingthelinearequation thatresultsfromtheconnectionofthetwodots.

IntheB-option,itispossiblethattheexpectedvalue forthebestperformer(s),comingfromtheequation,tends toworsentheperformance,insteadofimprovingit.Inthis case,theA-optionis tobepreferred,byaskingthebest performer(s)toholdthevalue.

Step 2 determines the range of variability that the Regionmayconsiderasacceptable.Indeed,itcouldsound oddtoacceptacertaindegreeofunwarrantedvariation. Theunderpinningofthischoiceislinkedtotheempirical evidenceprovided bythe goalsetting theorythat chal-lenginggoalslead tobetterperformance.Althoughit is desirablethatallunitsachievetheperformanceofthebest practiceorthegoldstandard,assigningtoeveryunit,for thesamegoal,thesameexpectedtargetmaybeperceived asunfair,thusreducingmotivationtoachieveit.Indeed, incremental goalscan be more motivating than radical

changes,whichcanbeperceivedasstretchgoals.Hence, theassumptionisthatanchoringthetargetdefinitionto ex-anteperformancewouldleadtobetterresultsandfaster reductionofgeographicalvariation(seeforinstanceRef. [40]).

Goal achievement evaluation is thelast crucial pro-cesstobeperformed.Ifthesettargetisreached,thenthe achievementis100%andnofurtherevaluationhastobe done.

Ifthesettargetispartiallyattained,then,inordertogive afairevaluation,itisimportanttocompareperformance withtheothercomparableunitsandthebaseline.Indeed, therelativeperformanceandthebaselinecanhelp under-standifthetargetwasstretchorifenvironmentalfactors occurred.

Howtoassessinafairwaytargetpartiallyattained?It dependsuponcircumstances.Itispossibletoidentifyfive scenarios:

1.Theperformanceoftheunitworsenedaswellasallthe othercomparableunits:itis clearthatsomeexternal factorsoccurred,sothatthesettargetwouldnomore beattainableortheselectedindicatorwouldbe uncon-trollablebytheunits;

2.Theperformanceoftheunitworsenedandtherelative performanceislowerthanthemedian.Inthiscase, per-formanceevaluationisdefinitelynegative;

3.Theperformanceoftheunitworsenedbuttherelative performanceishigherthanthemedian.Inthiscase,the evaluationisnotsonegativeandanincentivecouldbe givenaccordingtoitsrelativeposition;

4.Theperformance oftheunitimproved, butitdidnot achievethesettargetandtherelativeperformanceis lowerthanthemedian;

5.Theperformanceoftheunitimprovedandtherelative performanceishigherthanthemedian.

Please cite this article in press as: Vainieri M, et al. How to set challenging goals and conduct fair eval-Inthefourthandfifthscenarios,itispossibletoapply

thelinearsystemsuggestedbyLocke[41]:thepercentage oftheobtainedimprovementcouldbecreditedtotheunit. Inaddition,forthefifthcase,policymakersmay acknowl-edgeabonusfortherelativeperformance.Inthiscase,they havetochoose:thethresholdforprovidingthebonus(the mean?Themedian?Anotherpercentileuptothemedian? Thebestperformance?),andthebonussharebetweenthe relativeperformanceandthedegreeofimprovement.

4. Thetworegionalcases

The Spanish Health System’s universal coverage is fundedbytaxesanditpredominantlyoperateswithinthe publicsector.ThedevolutionprocesstotheRegionsended upin2002evenifstartedbeforeforsomeRegions[42]. Indeed,ValenciaRegionhashadfullyautonomouspower inmanagingandorganizingitshealthcaresince1987.The AgenciaValencianadeSalud(AVS)isthepublicbodythat provideshealthcareservicestoapproximately5million inhabitants,through24LocalHealthDepartments.

Since2004,AVShasadoptedamanagementcontrol sys-tembasedontargetsandtheirevaluation.Since2005,this systemhasalsobeenalignedwiththevariablesalaryofall itsemployeesand,since2007,ithasbeenlinkedtotheir professionalcareer(decree38/2007).

Objectivesaredeclinedstarting fromthreedomains: responsiveness,healthcareserviceprovisionandfinancial sustainability.

Whengoal-settingwasfirstapplied,mostofthe objec-tivesreferredtoprocessindicators(suchasaveragelength ofstay)then,in2010,outcomeandqualityindicatorswere introducedtoo.

Everyyear,regionalmanagersselectedtheindicators tobeincludedintherewardsystem,onthebasisofthe strategicplan,thecontextualenvironmentandthelistof indicatorsusedatthenationalleveltoassessRegions.

Thedecree38/2007oftheDepartmentofHealthreports thebasicprinciplesandactorsinvolvedinthedefinition ofthevariablepayofprofessionals(seeyearlyplansfrom 2007to2012,andtheperiodicstrategicregionalplans). Results and otherinformation aboutalgorithms can be derivedfrominternalAVSreportsandotherSpanish doc-uments[43].AppendixAshowsthelast goalsavailable. In2013,thelawno.7181/2013determinedtheclosureof theAVS.Itsfunctionsandpersonnelweretransferredto theDepartmentofHealth,thatstillmonitorsandassesses healthunitsandprofessionals.Withthischange,in2013, the newregionalmanagement decided togo backward tothetraditionalgoal-settingprocedures.Thetechnique describedinthispaperisstillusedtomanageprivatehealth careinstitutions.However,itseemsthattheDepartment ofHealthiscurrentlyreconsideringtheapplicationofthe methodforpublicinstitutionstoo.

TheItalianhealthcaresystemensuresuniversal cov-erage and,afterthedevolutionprocessof the90s,each Regionisresponsiblefororganizinganddeliveringhealth services [44–47]. Tuscany’s health care system covers approximately3.7millioninhabitants,delivers95%ofits services through its publicly-owned organizations, and

spendsmorethan6.6billioneurosinhealthcareservices peryear.

In2005,Tuscany’shealthcaresystemadopteda Per-formanceEvaluationSystem(PES)that consistsofmore than100quantitativeindicators,publiclydisclosed[48].In 2006,theregionadministrationdecidedtolinkthePESto theChiefExecutiveOfficers’(CEOs)rewardsystem.Before 2006,mostoftheCEOgoalswerequalitativeandassessed following the“all or none”criterion. Theywere mainly based(morethan50%)onfinancialperformanceandthe averageachievementlevel reachedupto90%,withlow variability.AfterintegrationwiththePES,morethan50%of thegoalsbecamequantitativeandtheweightofthe finan-cialassessmentgoalswasreduced[49].Abovementioned informationcanbefoundintheRegionalactsthatassign thegoalstothehealthcareauthoritiesCEOsandinternal documentsandreports.

Everyyear,regionalmanagersselecttheindicatorsthat are includedin the reward system,onthe basis of the strategic plan and the regional priorities[16],the con-textualenvironmentandthelistofindicatorsusedatthe nationalleveltoassessItalianRegions.Sincethe introduc-tionoftheNationaloutcomeprogram[50],someindicators havebeenrelatingtothehealthcareoutcomestoo(such as30daysmortalityrateforAMI).AppendixAshowsthe listofperformancegoalsofthelastyearavailable.

BothValenciaand Tuscanyapplysimilargoal-setting methods,althoughsomedifferencespersist.

Tosumup,itispossibletogroupthemaindifferences intwoaspects:(1)theprocessofcommunicationand(2) thelevelofimplementation.Asregardsthecommunication, Tuscanypubliclydisclosesalltheinformation –inorder toleveragetheprofessionals’reputation–,whileValencia carefullyselectssomeinformationtobepublicly dissem-inatedandsomeotherinformationtobecommunicated onlytopeers.Concerningthelevelofimplementation, Tus-canyassignshealthcaregoalstothehealthcareauthorities (formallyrepresentedbytheirCEOs),whilstValenciahas acentralizedandpervasivesystem,whichsetstargetsand assessesthemnotonlyatthemacrolevel(health depart-ments),butalsoatthemicrolevel(professionals).

Moreover,therearedifferencesinthetechnical meth-odsthetwoRegionsadopted.Forthetarget-settingphase, Tuscanyuses a globalperformance goal and an overall improvementgoal,withtheaimtomotivatehealth author-itiestopayattentiontoalltheindicators,inordertoreduce potentialoutputdistortions[9,51].Regardingthe assess-mentphase,Valenciausesthemedian(Tuscanyadopted themean),thearctangentfunctiontoadjustforthepast performanceeffectandthepremiumforbestperformers (Tuscanyacceptsasmallvariationtoholdthesame posi-tion).Inparticular,thethreecomponentsthatareapplied totheevaluation phaseare those ofSection 2[43]: (1) thedegreeofachievementofthesettarget(linear com-ponent);(2)theperformanceimprovementorworsening (asymptoticcomponent)and(3)therelativeperformance, comparedtosimilarLocalHealthDepartments’one (expo-nential component). The performance improvement is correctedbyafactorrangingfrom0.5and1.5froman arc-tangentfunction,withtheaimofreducingorincreasing thevalueoftheobjectiveby50%,dependingonwhether

Fig.2. TheyearlypercentageofperformanceimprovementandreductionofvariationinTuscanyandinValencia.

Fig.3.StandardisedhospitalizationrateinTuscany;DTPvaccinationcoverageinValencia.

Fig.4.Hospitallengthofstay.

adeteriorationoranimprovementoccurs.Anadditional andfinalcorrectionisappliedintheValencianmethodto ensurethatthereisalwayssomeonethatachieves100% ofthetarget:ifpisthepercentageofachievementofthe bestperformerandthisvalueislessthan100,acorrection factorof100/pisappliedtorescalealltheotherscores.

Results obtained by the two Regions seem rather encouraging. On the one side, the proposed method requiressomeeffortandstimulatestechnicalcompetences bytheRegion/supportingbodies;ontheotherside,iteases thecommunicationprocessbetweenevaluatorsand eval-uated.Sincethistechniquewasapplied,theclaimsabout goalunfairness(bothfromtheworseandfromthebest performingunits)hasheavilyreduced.

BothinValenciaandinTuscany,thistechnical frame-workhelpedmakethemechanismoffinancialincentives forCEOs(Tuscany)oremployees(Valencia)credibleand acceptable. It led not only to continuous improvement

but alsoto strategy alignmentbetween theRegion and the Health Departments.Finally, most ofthe indicators reported a reduction of variability in the performance of theunits analyzed [48,52]. The following descriptive statistics canoffersomeinsightsabouttheimplications of theadoption ofa coherent managerialstrategy (that includesthegoal-settingprocessproposedinthispaper) onperformanceandvariation:thepercentageofimproved indicatorshasvariedacrossyearsbetween54%and67%in Tuscany;54%and64%inValencia(seeFig.2).Inaddition, morethan50%ofindicatorsreducedtheirvariabilityevery year.

The list of indicators of the last year available are reportedinAppendixA,whilesomeexamplesareshown below.Inparticular,Fig.3reportstheresultsobtainedby the12 TuscanLocalHealth Authoritiesin the standard-izedhospitalization ratefrom2008 to2014andresults obtainedbythe24centersthathavetomonitorwhether

Please cite this article in press as: Vainieri M, et al. How to set challenging goals and conduct fair eval-0–6monthsbabieshavecompletedalltherequestedexams

inoutpatientprimarycarevisits(2008–2010).Forthe Tus-cancase,thenormativestandardforhospitalizationrate was120×1000residents(DGRT1235/2012),whileforthe Valenciancase(DTPvaccinationcoverage)thegold stan-dardis100%(Fig.3).

AsFig.4showstheTuscanLocalHealthAuthoritieshave steadilyreducedtheirhospitalization,whilethereduction ofvariationisunstable,becauseofthedifferentspeedin improvingperformance;however,inthelongrun,every LHA is aligning tothe bestperformers, as 2013results show.InthethreeyearsanalyzedforValencia(2008–2010), themajorityofunitsimproved,reducingdifferencesacross them.

Fig.4reportsanotherexample,forindicatorsthatdonot haveastandard:theaveragelengthofstay.Itiscalculated differentlybyValenciaandTuscany:Valenciaconsidersthe averageDRGweightofthehospital,whileTuscany consid-erstheaveragenumberofdaysaboveorbelowtheregional averagelengthofstay(perDRG).

In the Tuscan case, the average length of stay has reduced on averageof one day: the 16 units (12 Local Health Authoritiesand the4 teachinghospitals) moved froma2008medianofup1.5daystolessthan0in2015. Thereductionofvariationinthiscaseismoreevident.In theValenciancase,theaveragelengthofstayreducedof 0.3days onaveragefrom2006to2010.Inthiscase,the reductionofvariabilityislessevident.

5. Discussionsandconclusions

Thetwocasesthis paperdescribesaimtohelpsolve the dilemma of “how challenging is challenging” and “howtoconductafairassessmentofthetarget achieve-ment”. The methodologies adopted bythe two Regions shareatleasttwocomponents:performance benchmark-ing(relativeperformance)andperformancebaseline(past performance). Theyboth integratea cross-sectional per-spective (benchmarking) with a longitudinal one (past performance).

Ontheoneside,benchmarkingintroducesayardstick competitionamongtheactorsofthehealthsystems,by helping Regions setdifficult, yet attainable targets and avoidproblemslinkedtouncertainty.

Ontheotherside,followingtheassumptionthat incre-mentalgoalscanbemoremotivatingthanradicalchanges, thetechnicalframework (Section3)proposestoset tar-gets byaskingfor aninverseeffortonthebasis ofpast performance.Theexpectedconsequenceisahighdegree ofsuccess,whichleadstoanoverallregionalimprovement andareductionofgeographicvariationaswell.

TheresultsobtainedbythetwoRegionsseem promis-ing,asboth ofthemregisterperformance improvement and variability reduction. The process we described is aimed not only at guaranteeing high-quality services (definedbythechosentargets),butalsoatreducingthe distancebetweenbestandworstperformers:itis there-foredesignedtosupportRegionsincopingwithpotential inequalitiesinservices.

Moreover,reportsoftheMinistryofHealthonthe per-formanceoftheItalianRegionstoguaranteeessentiallevel

ofcarefrom2007to2014highlightthatTuscanysteadily improveditsperformanceresults(comparedtotheother ItalianRegions)becomingthebestperformingRegionin 2013 and in 2014, also registering the highest level of improvement.

Results obtainedby thetwo Regions(both in terms of performance and in terms of variability)might have significantimplications,ifweconsiderthatTuscanyand Valenciahaveadoptedverydifferentgovernancemodels.If wetakeupBevanandWilson’sclassification[11],Valencia canbedefinedtohaveadoptedthecentrallydriven “hier-archyandtargets”model,withsomecharacteristicsthat arelinkedtothe“choiceandcompetition”model. Actu-ally,thepublic-private partnershipthat wasintroduced intothesystemenablesquasi-marketmechanisms. Tus-cany,instead,combinesthe“hierarchyandtargets”model withpublicranking[9].Despitedifferentgovernance mod-els,thetechnicalsolutionsadoptedbythetwoRegionsare verysimilar.Bothofthemusepastperformanceand rel-ativeperformance tosetchallenging goalsand tofairly assesstheirachievement.Hence,theoperational frame-workseemstobeusefulandadaptabletodifferentcontexts andseemstobeapplicabletodifferentlevels(CEOs,heads ofdepartments,individuals).

Thetwocasestudieswedescribeddifferinarelevant characteristic, that might deserve some further consid-eration:asmentionedabove,thegoalsettingprocedure is addressed to the CEOs of Local Health Authorities and Teaching Hospitals in Tuscany,while it focuses on thehealthdepartmentsandonprofessionalsinValencia. Hence,themethodismainlyproposedatthemacrolevel toregionaladministrations,butitcanalsobeappliedatthe microlevel,aslongasunitscanbecomparedandthereisa sharedperformancemeasurementsystemalreadyinuse.

Itisbeyondthescopeofthispapertoinquirewhether theapplicationofthegoalsettingtechniqueswediscussed provetobedifferentlyeffectiveaccording tothe differ-entagentsitisaddressedto(CEOs,headsofdepartments, professionals).TheyareonlyapartoftheTuscanand Valen-cianmanagerialstrategiesandfurtherresearchisneeded tounderstandif(and howmuch of)thepositive afore-mentioned preliminary resultscan be attributedto the methodsappliedinthetwoRegions.However,wewant toemphasizethattheunderlyingassumptionsthatsome oftherecommendationsthispaperinfers–theimportance ofconsideringthebaselineinthegoal-settingprocedure, forinstance–canbegeneralizedacrossthedifferent lev-els.Moreover,thepreliminaryevidenceoftheeffectiveness ofthismethodareinfluenced alsoby theapplicationof broadergovernancetools.

We deem it necessary to dedicate some conclusive remarks to thereorganization that affectedthe AVS in 2013.Thisspecificeventgivesustheopportunityto high-lighttheneedofsystematicallycombiningthescientific accuracyofthemanagementtoolswiththeirtransparent disclosure:allegedly,themainreasonbehindthechoice ofdismissingthegoal-settingproceduredescribedinthe paperandtooptforasimpleronewastheneedtoadopt a methodology that could be easily understood by the newmanagementofthedepartmentofhealth,thattook in charge thedefinition of goalsand its assessment in

2013.The policy-makersmainlyembracedtransparency toleveragetheprofessionals’andthemanagers’ reputa-tioninimprovingperformances,butitalsocouldhavea (positive)side-effect:oncegranted,transparencycannot beeasilywithdrawn.Thismeansthatitanchorsthe policy-makersthemselvestoanirrevocablepublicaccountability logic.MaybeadifferentchoiceofValenciaonpublic disclo-surecouldhaveactedasadeterrenttogobackwardtothe traditionalwayofsettingtargets.

Indeed,publicdisclosure of datacontributes to fully exploittheopportunities aperformance evaluation tool (targetsetting,performancemeasurementsystemor eval-uation)mightoffer.Onceallthestakeholdershavebecome familiarwiththeregular useof theevaluationtool, the same endorsement by policy-makers may become less relevant,asthelegitimizationwouldcomefromthe stake-holders’expectationsthemselves.Itthereforeturnsintoa “commonlanguage”thatthevariousagentsadopttobe accountablewithanevidence-basedapproach.

Inconclusion, the transparentprocess, reinforcedby publicdisclosureofdata,canconvertthetechniquefroma governancetoolinthepolicy-makers’handsintoan“open asset”,attheservice ofallthestakeholders thatjointly constitutethehealthcaresystem.

Acknowledgements

Apreviousversionofthis manuscriptwaspresented atthe2014EHPGautumnmeeting.Theauthorswantto thankEHPGmembersfortheirhelpfulcommentsand Bar-baraBini,forherpreciouscontributionincollectingsome materials.

AppendixA. Supplementarydata

Supplementary data associated with this arti-cle can be found, in the online version, at http://dx.doi.org/10.1016/j.healthpol.2016.09.011.

References

[1]WorldHealthOrganization.Ottawacharterofhealthpromotion. WorldHealthOrganization;1986.

[2]WorldHealthOrganization.Thehealthforallpolicyframeworkfor theWhoEuropeanRegion.Health21;1999.

[3]WismarM,McKeeM,ErnstK,SrivastanaD,BusseR.Healthtargets inEurope.ObservatoryStudiesSeries2008:13.

[4]BusseR,WismarM.Healthtarget programmesandhealthcare services—anylink?Aconceptualandcomparativestudy(part1). HealthPolicy2002;59:209–21.

[5]Decentralizationinhealthcare.SaltmanRB,BankauskaiteV, Vrang-baekK,editors.Strategiesandoutcomes.London:OpenUniversity Press;2007.

[6]FerreiraA, Otley D. Thedesign and use ofperformance man-agement systems: an extended framework for analysis. Mana 2009;20:263–82,http://dx.doi.org/10.1016/j.mar.2009.07.003. [7]IttnerCD,LarckerDF.Comingupshortonnonfinancialperformance

measurement.HarvardBusinessReview2003:81.

[8]LockeEA,LathamGP.Atheoryofgoalsettingandtaskperformance. EnglewoodCliffs,NJ:PrenticeHall;1990.

[9]NutiS,Vola F, Bonini A,Vainieri M.Making governance work in the health care sector: evidence from a natural experi-mentinItaly.HealthEconomics,PolicyandLaw2015;11:17–38, http://dx.doi.org/10.1017/S1744133115000067.

[10]LeGrandJ.Motivation,agencyandpublicpolicy:ofknightsand knaves,pawnsandqueens.Oxford,UK:OxfordUniversityPress; 2003.

[11]BevanG,WilsonD.Doesnamingandshamingworkforschoolsand hospitals?Lessonsfromnaturalexperimentsfollowingdevolutionin EnglandandWales.PublicMoney&Management2013;33:245–52, http://dx.doi.org/10.1080/09540962.2013.799801.

[12]Brown A, ClossonT, SullivanT,Baker GR. Thejourney toward high performance and excellent quality. Healthcare Quarterly 2012;15:6–9.

[13]PencheonD.Thegoodindicatorsguide:understandinghowtouse andchooseindicators.London:NHSInstituteforInnovationand Improvement;2007.

[14]Gagliardi AR,Fung MKF,Langer B, Stern H,Brown AD. Devel-opment of ovarian cancer surgery quality indicators using a modifiedDelphiapproach.GynecologicOncology2005;97:446–56, http://dx.doi.org/10.1016/j.ygyno.2004.12.059.

[15]JacobsR,GoddardM,SmithPC.Publicservices:arecomposite mea-suresarobustreflectionofperformanceinthepublicsector.Centre forHealthEconomicsPapern.16:2006.

[16]NutiS,VainieriM,VolaF.Prioritiesandtargets:amethodologyto supportthepolicy-makingprocessinhealthcare.PublicMoney& Management,2017;February.

[17]MuranteAM,VainieriM,RojasD,NutiS.Doesfeedbackinfluence patient—professional communication? Empirical evidence from Italy.HealthPolicy(NewYork)2014;116:273–80.

[18]Vandewalle D. A goal orientation model of feedback-seeking behavior.HumanResourceManagementReview2003;13:581–604, http://dx.doi.org/10.1016/j.hrmr.2003.11.004.

[19]Latham GP, Borgogni L, Petitta L. Goal setting and performance management in the public sector. Inter-national Public Management Journal 2008;11:385–403, http://dx.doi.org/10.1080/10967490802491087.

[20]LockeEA,ShawKN,SaariLM,LathamGP.Goalsettingandtask per-formance:1969–1980.PsychologicalBulletin1981;90:125–52. [21]Lunenburg FC. Goal-setting theory of motivation. International

Journal of Management Science Business and Administration 2011;15:1–6.

[22]Locke EA, Latham GP. New directions in goal-setting the-ory.CurrentDirections inPsychologicalScience2006;15:265–8, http://dx.doi.org/10.1111/j.1467-8721.2006.00449.x.

[23]LathamGP.Workmotivation:theory,researchandpractice. Thou-sandOaks,CA:Sage;2007.

[24]Mitchell TR, Daniels D. Motivation. In: Borman WC, IlgenDR, KlimoskyRJ,WeinerIB,editors.Compr.Handb.Psychol.Ind.Organ. Psychol.NewYork:Wiley&Sons;2003.p.225–54.

[25]LockeEA,LathamGP.Newdevelopmentsingoalsettingandtask performance.NewYorkandLondon:RoutledegeTaylor&Francis Group;2013.

[26]Sitkin S, See K, Miller C, Lawless M, Carton A. The para-dox ofstretchgoals: organizations inpursuitof theseemingly impossible. Academy of Management Review 2011;36:544–66, http://dx.doi.org/10.5465/AMR.2011.61031811.

[27]BirdS,CoxD,FarewellV,GoldsteinH.Performanceindicators:good, bad,andugly.JournaloftheRoyalStatisticalSociety2005;168:1–27. [28]AnthonyR,YoungD.Managementcontrolinnonprofitorganizations,

VIed.Boston,MA:McGraw-Hill;1999.

[29]GreeneJ,HibbardJH,OvertonV.Largeperformanceincentiveshad greatestimpactonproviderswhosequalitymetricswerelowestat baseline.HealthAffairs2015;34:673–80.

[30]FlamholtzEG,DasTK,TsuiAS.Towardanintegrativeframework oforganizationalcontrol. Accounting,OrganizationsandSociety 1985;10.

[31]Jonsson S, Lukka K. There and back again: doing interven-tionist research in management accounting. In: Chapman CS, Hopwood AG, Shields MD, editors. Handbooks of manage-ment accounting research, vol. 1. Elsevier; 2006. p. 373–97, http://dx.doi.org/10.1016/S1751-3243(06)01015-7.

[32]SusmanGI,EveredRD.Anassessmentofthescientificmeritsof actionresearch.AdministrativeScienceQuarterly1978;23:582–603, http://dx.doi.org/10.2307/2392581.

[33]Lewin K. Action research and minority prob-lems. Journal of Social Issues 1946;2:34–46, http://dx.doi.org/10.1111/j.1540-4560.1946.tb02295.x.

[34]Reason P, Bradbury H, editors. Handbook of action research. London–ThousandOaks–NewDelhi:SAGEPublications;2006. [35]Rapoport RN. Three dilemmas in action research:

expe-Please cite this article in press as: Vainieri M, et al. How to set challenging goals and conduct fair eval-rience. Human Relations 1970;23:499–513,

http://dx.doi.org/10.1177/001872677002300601.

[36]EeroK,KariL,ArtoS.JournalofManagementAccountingResearch 1993;5:243–64.

[37]BergBL.Qualitativemethodsforthesocialscientist.Boston:Pearson Education;2004.

[38]MctaggartR.Participatoryactionresearch:issuesintheoryand prac-tice.EducationalActionResearch1994;2:313–37.

[39]DruckerP.Thepracticeofmanagement.NewYork:Harper&Row; 1954.

[40]Nuti S, Seghieri C. Is variation management included in regional healthcare governance systems? Some propos-als from Italy. Health Policy (New York) 2014;114:71–8, http://dx.doi.org/10.1016/j.healthpol.2013.08.003.

[41]LockeEA.Linkinggoalstomonetaryincentives.TheAcademyof ManagementExecutive2004;18.

[42]García-armestoS,Abadía-tairaMB,Hernández-quevedoC, Bernal-delgadoE.HealthSystemsReview2010:12.

[43]GòmezG,CarrilloE.Evaluaciòndelcummplimientodeobjetivosen lossistemasdeincentivaciòndeldesempe ˜no.Barcelona:2009. [44]FerréF,GiulioA,ValerioL,LonghiS,LazzariA,FattoreG,etal.Health

systemreview:Italy.HealthSystemsinTransition2014;16:1–168. [45]FattoreG.ClarifyingthescopeofItalianNHScoverage.Isit

fea-sible?Isitdesirable?HealthPolicy(NewYork)1999;50:123–42, http://dx.doi.org/10.1016/S0168-8510(99)00068-8.

[46]FranceG,TaroniF,DonatiniA.TheItalianhealth-caresystem.Health Economics2005;14:187–202,http://dx.doi.org/10.1002/hec.1035. [47]TediosiF,GabrieleS,LongoF.Governingdecentralizationinhealth

care under tough budget constraint: what can we learn from theItalianexperience?HealthPolicy(NewYork)2009;90:303–12, http://dx.doi.org/10.1016/j.healthpol.2008.10.012.

[48]Nuti S, Seghieri C, Vainieri M. Assessing the effectiveness of a performance evaluation system in the public health care sector: some novel evidence from the Tuscany region experience. Journal of Management & Governance 2012, http://dx.doi.org/10.1007/s10997-012-9218-5.

[49]NutiS.Lavalutazionedellaperformanceinsanità.Bologna:IlMulino; 2008.

[50]AgabitiN,DavoliM,FuscoD,StafoggiaM,PerucciCA.Valutazione comparativadiesitodegliinterventisanitari.Epidemiologiae Pre-venzione2011;(Suppl.1):1–80.

[51]BevanG,HoodC.What’smeasurediswhatmatters:targetsand gam-ingintheEnglishpublichealthcaresystem.PublicAdministration 2006;84:517–38.

[52]Revista deMedicina de Familiay AtenciónPrimaria.Vol. 15(1 Suppl):50p.ISSN:1989-6832.