Measuring research in the Big Data Era: the evolution of Performance Measurement Systems in the Italian Teaching Hospitals

(1)

ContentslistsavailableatScienceDirect

Health

Policy

jo u rn al h om ep a g e :w w w . e l s e v i e r . c o m / l o c a t e / h e a l t h p o l

Measuring

research

in

the

big

data

era:

The

evolution

of

performance

measurement

systems

in

the

Italian

teaching

hospitals

Frank

Horenberg

a,b,∗

_,

_Daniel

_Adrian

_Lungu

a,b

_,

_Sabina

_Nuti

a,b

a_Health_and_Management_Laboratory_(MeS_Lab),_Institute_of_Management_and_Department_EMbeDS,_Scuola_Superiore_Sant’Anna,_Piazza_Martiri_della

Libertà,33,Pisa,Italy

b_Sant’Anna_School_of_Advanced_Studies,_Health_and_Management_Laboratory_(MeS_Lab),_Piazza_Martiri_della_Libertà,_33,₅₆₁₂₇_Pisa_PI,_Italy

a

r

t

i

c

l

e

i

n

f

o

Articlehistory: Received2July2019 Receivedinrevisedform 21September2020 Accepted4October2020 Keywords: Teachinghospitals Researchproductivity Performanceevaluation Impactfactor

Field-weightedcitationimpact

a

b

s

t

r

a

c

t

Background:Inthehealthcaresystem,TeachingHospitals(THs)notonlyprovidecare,butalsotrain health-careprofessionalsandcarryoutresearchactivities.ResearchisafundamentalpillarofTHs’missionand relevantforthehealthcaresystemmonitoredbyPerformanceEvaluationSystems.Researchactivities canbemeasuredusingcitationindexservicesandthispaperhighlightsdifferencesbetweentwo ser-vicesbasedonbibliometrics,describesopportunitiesandriskswhenperformanceindicatorsrelyon datacollected,controlledandvalidatedbyexternalservicesanddiscussesthepossibleimpactonhealth policyatasystemandproviderlevel.

Methods:Abibliometricanalysiswasdoneondatabetween2014−2016fromISIWebofScienceand Scopusof18.255physiciansworkingin26ItalianTHs.Quantitywasdeﬁnedasthenumberofpublications andqualityasImpactFactororField-WeightedCitationImpact.

Results:Overall,41.233and66.409documentswereextractedfromrespectivelyISIWebofScienceand Scopus.Whilebenchmarkingresults,signiﬁcantdifferencesinrankedpositionbothinmetricsemerged. Discussion:UtilizingsecondarydatasourcestomeasureresearchactivitiesofTHsallowsbenchmarking atan(inter)nationallevelandovercomingself-referment.Toutilizeindicatorsformultiplegovernance purposesatthesystemandproviderlevel,indicatorsneedtobeprofoundlyunderstood,require for-malizationsindata validation,internalanalysis andasharingprocess amonghealthprofessionals, managementandpolicymakers.

1. Background

Performanceevaluationsystems(PESs)arecrucialfor

account-abilityandserveasafeedbackandguidancetoolforthemanagerial

leveloforganizations[1].PESsareusedtoevaluatehowwell

orga-nizationsaremanagedandtomeasurethevaluethatorganizations

delivertocustomersandotherstakeholders[2,3].Fromthe1980s,

PESshavebroadenedthekindofindicatorsmonitored,but

main-tainedthefocusonﬁnancialones[4,5].Alongside,developmentsin

Abbreviations:IRPES,Inter-Regional-PerformanceEvolutionSystem;LHA,local healthauthority;PES,performanceevaluationsystem;TH,teachinghospital;WoS, ISIWebofScience.

∗ Correspondingauthorat:Sant’AnnaSchoolofAdvancedStudies,Healthand ManagementLaboratory(MeSLab),PiazzaMartiridellaLibertà,33,56127PisaPI, Italy.

E-mailaddresses:horenbergfrank@gmail.com(F.Horenberg),

danieladrian.lungu@santannapisa.it(D.A.Lungu),sabina.nuti@santannapisa.it

(S.Nuti).

informationandcommunicationstechnology(ICT)facilitateddata

availability,completeness,andaccessibilityandtheevolutionof

theso-calledBigDataturnedusefultoenrichthePESinformation

[6–8].

StillnowPESsineconomicsectorsthatareproﬁt-orientedare

mainlyfocusedonmeasuresregardingproﬁtandrevenues,while

thisisnotthecaseinhealthcarewherethegoalistoproducevalue

forpatientsandthepopulation[9–11].Withinthehealthcare

sec-tornon-ﬁnancialindicatorsarecrucialandPESs,mostlyinpublic

universalcoveragehealthcaresystemswhererevenuesarebased

onapercapitaquota,aredesignedandimplementedtobeable

tomeasureononesideoutcomes,qualityofcareandlife,identify

issues,andontheotherhandresourcesmadeavailablebysociety.

Inorderfor PESstobeeffectiveinpublicuniversalcoverage

healthcaresystem,itshouldbecharacterizedbythefollowing

ele-ments[12,13]:

• Multi-dimensionality:Indicatorsshouldincludemultiple

dimen-sions(process,qualityofcare,equity,etc.);

https://doi.org/10.1016/j.healthpol.2020.10.002

(2)

• Evidence-based:onresearchorclinicalpractice;

• Shareddesign:allstakeholders,andespeciallyhealth

profession-als,shouldbeinvolvedinthedesignandtheﬁne-tuningprocess

ofthePESandtheindicators;

• Systematicbenchmarking:allowstoovercomeself-referentiality

andtomeasureavoidablevariationandspaceforimprovement;

• Transparentdisclosure:stimulatesdatapeer-reviewandmakes

professionalreputationleveragepossible;

• Timeliness:allowspolicymakerstomakedecisionspromptly.

Withthesepremises,PESs inhealthcare have beenevolving

overcomingtheorganizationalboundariesofsingleproviders[14].

Moreover,amongtheaboveelements,themostcrucialistorely

on benchmarking which facilitates and triggers organizational

improvementprocessestoincreaseeffectivenessbasedon

repu-tation[15].

Consideringtheserelevantfeaturesfor thehealthcaresector,

thispaperfocusesonthePESsadoptedbyteachinghospitals(THs)

withinthehealthcaresystem.Thesekindofinstitutions,evenif

theymaybenodifferentfromotherhospitalsintermsofquality

ofcare[16],fulﬁllaspecialroleinthehealthcaresystembecause

theirmissionisnotjustprovidingcare,butalsototrainhealthcare

professionalsandtocarryoutresearchactivities.

Asmedicalknowledgecontinuouslyevolves,THsareatthe

cen-terofinnovationsinhealthcare withrespecttotreatments and

cures.Theyareinchargeofconductingresearchandaddingnew

knowledgetoscientiﬁc literature.For this reason, researchis a

fundamentalpillarof theTHs’missionand thereforea relevant

componenttobeincludedinPESs,notjustforthesingleprovider

butforthewholehealthcaresystem.TheabilityofTHstoperform

researchactivitiesguaranteesthemaroleofreferenceandguide

intheprocessesofimprovingcareatregionalornationallevel.The

morehealthcareprofessionalsknowhowtobeonthefrontiersof

clinicalresearch,themorelikelyitisthatcarewillbealignedwith

thebestandmostupdatedclinicalprotocolsbeneﬁtingforpatients.

Itisinfactproventhatthebesthospitalsarethosewheremore

researchiscarriedoutandinturnhealthcareoutcomesimprove

[17,18]and patientsbeneﬁtfromaccesstonewandinnovative

treatmentsthatwouldnotbeotherwiseavailable[19,20].

Further-more,researchactivitiesshouldguideprocessestoimprovequality

ofcaregenerationknowledgewhichleadstoupdatedandtrained

stafftoestablishteamsofexpertsandcentersofexcellence[23–26].

Measuringperformanceofresearchactivitiesthusbecomesa

relevanttopicforthewholehealthcaresystemandforeachTHthat

operatesinit.However,isanendeavorasresearchactivitiesresult

inbothintangible(e.g.knowledge,experience)andtangible

out-puts(e.g.scientiﬁcarticles,products)andaccuratemeasurements

dependonmanypreconditions[27,28].

Ameasurethatisfrequentlyusedisscholarlyoutput[27],by

countingthenumberofpublisheddocuments.Thisproxycanbe

accessedviareadilyavailabledatasourcesfrompublishers,

jour-nals,citationindexingservices,andothersecondarydatasources.

ItisasimplemeasurethatcanbedetectedinternallybyeachTH,but

whichalsohasanexternalvalue:thearticleshavebeenpublished

andthereforerecognizedbythescientiﬁccommunityassigniﬁcant

contributionstotheevolutionofscience.

Quantifyingresearchusingthenumberofpublishedarticlescan

thereforebetheﬁrststeptomeasureresearchperformancebut

doesnotprovideanyinformationaboutthevalueandimpactof

thesepublishedworks.Othermetricsarethereforeneededto

mea-surethequalityoftheseworkstoprovidecontextandtheimpact

withintheresearchcommunity[29].Inthisperspective,citations

canbeareferenceelement,sinceitisanindirectpositive

evalua-tionthattheworkhasbeenreadandtakenintoconsideration-and

thereforementioned-bycolleagues.

Inordertoprovidecontextandassesstheimpactofscholarly

output,aquantitativemethodcanbeemployed,namelya

biblio-graphicanalysis[30].Nowadays,variouscommonlyusedmetrics

canbeusedtoassess valuei.e. downloadsand views,citations,

impactfactor,h-indexandﬁeld-weightedcitationimpact(FWCI),

altmetrics (storage,links, bookmarks, conversations) and many

others[31–34].Thesemetricscanbeutilizedbothonajournalor

individualresearcherlevel.Mostbibliometricsarecalculated,

man-aged,andtrackedusingcitationindexservicessuchasScopus,and

ISIWebofScience(WoS)andcanbeaccessedviatheInternet.

Thesedifferentcitationindexservicesareanaccesspointto

differentrepositorieswhichstoreandcategorizescholarlyoutput.

However,eachoftheseservicesdiffersintheircoverage,method

of tracking, and available metrics [35]. Since data provided by

theseservicesaremanagedandcontrolledbyexternalparties,new

opportunitiesandchallengesariseforPESswhenusedtoevaluate

andbenchmarkperformance.

• What are the main differences between the commonly used

scholarly metrics extracted from citation index services and

derivedusingbibliographicanalyses?

• What are the opportunities and risks when PESs and their

correspondinggovernancetools arerelying ondatacollected,

controlled,andvalidatedbyexternalsources?

• Whatistheimpactintermsofhealthpoliciesatasystemlevel?

Thispaper,therefore,describesthedifferencesbyfocusingon

thequantityandqualityofresearchperformanceofTHsthatresult

fromtheuseoftwodifferentscientiﬁccitationindexingservices

ontheweb,ScopusandISIWebofScienceCoreCollection,by

per-formingabibliometricanalysisof26THsinItaly.Moreover,results

arecontextualizedbydiscussingthepossibleimplicationsonPESs

whenmeasuringresearchperformancethroughalternative

exter-nalcitationindexservices.

Thenextsectionconceptualizesdifferentmetricsusedto

eval-uatescientiﬁcproductionprovidedbybothScopusandWoSand

adescriptionoftheimplementationofresearchperformancein

aregionalperformanceevaluationsysteminItaly.Thethird

sec-tiondescribesthemethodologyandcomparingmetricsfromboth

services.Findingsarecontextualizedin theﬁnaldiscussionand

conclusionsections.

1.1. Measuringscientiﬁcperformance

Asearlyasin1927Grossetal.identiﬁedtheproblemof

dissem-inatingliteratureandin1955EugeneGarﬁeldproposedtoutilize

a citationindex forscientiﬁc literatureto eliminatethe

uncrit-icalcitationof fraudulent,incomplete,or obsoletedata [31,36].

Later,in 1961,asa founderof theInstitutefor Scientiﬁc

Infor-mation(ISI)GarﬁeldlaunchedtheScienceCitationIndexasatool

forresearchers,librariansandscholarstomanagethelarge

num-beroflibrarycollections.Overtime, thepurposeofthecitation

indexchanged,nowknownasimpactfactorwhichwasintended

todescribejournalimpactbasedonthenumberofcitations[37].

Althoughthismetricwasneverdesignedorintendedtobeusedas

anevaluationindicator,inpracticeitisoftenusedtoindicatethe

qualityofindividualscientiﬁcwork[38].Thescientiﬁccommunity

hasoftenexpressedconcernsregardingthebiasedimpactfactorof

journalsderivingfromasymmetric-leftskewed-distributionof

papercitations[39,40]andcausingquitesomecontroversywithin

theresearchcommunity[38,41–43].

Althoughmanyalternativesevaluationmetricshavebeen

pro-posedbytheresearchcommunity,impactfactorremainsdominant

inusage.However,Larivièreetal.proposedasimpleandrobust

methodologytodeferthecitationdistributionsthatunderliethe

(3)

pro-posedmethodwasadoptedbytheJournalCitationReports(JCR)

in2018andseemstobeanearnestﬁrstattempttoaddressthe

concernsofthescientiﬁccommunity[45].

Anotherwell-knownproblemwithimpactfactoristhe

skew-nessinspeciﬁcresearchﬁelds.Forexample,Narinetal.reported

thatresearchinbiochemistry andmolecularbiologywerecited

aboutﬁvetimes asoftenaspharmacy articles[46].In orderto

correctforthisphenomenon,theDutchpublisherElsevier

imple-mented their own metric, namely the ﬁeld-weighted citation

impactmetric. TheFWCIshowshowthenumberofcitationsof

asinglepapercompareswiththeaveragenumberofcitationsby

similarpublicationsindexedinScopus[47]resolvingtheissueof

differentresearchbehavioracrossdisciplines.

AnotherwellknowmetricgrouparetheAltmetricswhichuse

multipledatasourcessuchassocialmedia,numberofreadingsand

downloadstoassesstheimpactofthepaperbothinsideandoutside

thescientiﬁccommunity.Theabilitytomeasureimpactofscientiﬁc

workoutsidethescientiﬁccommunityisavaluabletrait.However,

theactualuseofAltmetricsneedstobefurtherconceptualizedto

becomeametriconitsownwhilestilllackingacleardeﬁnition,an

ever-evolvingframework,lowdatatransparency,andorigin[48].

Evaluatingthelargebodyofavailablebibliography,itbecomes

clearthatitisnotpossibletomeasurescientiﬁcperformanceby

simplyusingonemeasureandwhileothermeasuressuchasthe

h-indexandg-indexareincreasinglyusedtoevaluateresearchers’

performance,usingdifferentmetrics,emphasizingboth

productiv-ity,qualityandcontext,insideandoutsidetheresearchcommunity

isimperative.

1.2. EvaluatingresearchperformanceinItaly

The Italian National Healthcare System follows a Beveridge

model,mainlyﬁnancedthroughgeneraltaxationandbasedonthe

principleofuniversalcoverage.Resourcesarecollectedatanational

levelandallocatedtothetwentyregionsonage-adjustedpercapita

basis.Theresponsibilityfortheorganizationandprovisionofcare

hasbeendecentralizedat a regionallevel, and regions allocate

resourcestoLocalHealthAuthorities(LHAs)whoareresponsible

forthedeliveryofallhealthcareservicesintheirgeographicalarea,

directlythroughpublicprovidersoraccreditedprivatehospitals.

THsareautonomousbodiesfromtheLHA,canbepublicorprivate,

andareusuallymanagedjointlybytheregionaladministrationand

auniversity.Thissharedresponsibilityinmanaginghasarelevant

impactontheirorganizationalcultureastheycanbeconsidered

doubleprofessionalbureaucracies[49].Withintheregional

health-caresystem,theyplayarelevantrolebecausetheyoverseetraining

offuturehealthprofessionalsandbecausetheyareinchargeof

leadinginnovationprocessesbasedontheresearchactivitiesthat

theycarryout.

Starting from2005, theManagementand Health Lab ofthe

Scuola Superiore Sant’Anna has developed a multidimensional

healthcareperformanceevaluationsystem(PES),initiallyadopted

by Tuscany’s regional administration. Over time, the PES was

adoptedbyanincreasingnumberofregionsandin2008the

Net-workofItalianRegionswasformed.TheNetworkexpandedand

nowadaystwelveregionshaveadoptedthesame

Inter-Regional-Performance Evolutionsystem (IRPES) [13] tobenchmark their

performanceusingmorethanthreehundredsharedindicators.In

2014,theNetworkofTHswasfounded,aimedatbenchmarkingTHs

performanceusingaPEStotakeintoaccountthespeciﬁc

charac-teristicsandmissionofTHswithintheregionalhealthcaresystem

consideringaroundsixtyindicators[16,50,51].

Reportingontheseindicatorsareconsideredanimportant

man-agementtoolbyallTHsandtheItaliangovernmentandregionsuse

themforseveralissuesasmonitoringandassessingperformance,

allocateﬁnancialresourcesforresearch,andalsotoevaluatethe

Table1

IndicatorsincludedintheIRPESregardingresearchevaluationofteachinghospitals.

Number Descriptionofindicator

1 Averageimpactfactorperphysician 2 Averagenumberofpublicationsperphysician 3 Percentageofpublicationswithanaverageimpact

factorhigherthanthebenchmarkspecialtyimpact factorreportedinISI

4 Percentageofpublicationswithamedianimpact factorhigherthanthebenchmarkspecialtyimpact factorreportedinISI

5 Medianimpactfactorperspecialty 6 Medianimpactfactorvariationperspecialty

GeneralManagers’performance[52].Giventhatresearchisoneof

thethreepillarsofTHs’mission,withinthesesixtyindicatorssome

arefocusedonresearchperformance.

Table1providesanoverviewoftheindicatorsincludedinthe

IRPESwhichareusedtoevaluateresearchactivities.

2. Methods

The bibliometricanalysis canbe performedusing two

well-knownpubliclyaccessiblecitationindexingservices,namely,ISI

webofknowledgeandScopus.Metricsaboutthescholarlyoutput

canbeextractedfromtheserepositoriesbysimplyprovidingauthor

ﬁrstname,lastname,andoptionallytheirafﬁliatedorganization.

Thepossibilityofusingthesesearchengines,externaltothe

internal detection systems, has always been perceived as an

opportunitytohave ¨certain ¨andvalidateddata,afundamental

char-acteristictoguaranteethestrength,rigorandreputationofthePES

itself.

However,eventhesesystemsshowsomecriticalities.For

exam-ple,whenan authoris afﬁliatedwithmultipleorganizationsor

duplicatenamesareafﬁliatedwiththesameorganizations

man-ualcorrectionisrequired.Dataextraction,articlede-duplication

of(co-)authors,andvalidationweredoneintwodifferent

man-ners for both repositories including all scientiﬁc documents in

thesedatabaseswhichhavebeenpublishedbetween2014−2016.A

detailedprotocolofthedataextractioncanbefoundas

supplemen-tarymaterial;AppendixA(inSupplementarymaterial)–Extraction

ofdata.THswereresponsibleforprovidingnamesofresearchers

afﬁliatedwiththeirorganization.

2.1. DataextractionWoS

DocumentspublishedinISIWebofSciencebetween2014–2016

were extracted and validated by an external afﬁliated party,

ResearchValueSRL,inMay2018.Datawassentto

correspond-ingauthorsforinternalvalidationoncompletenessusingrandom

samplingmethods.Allavailablemetricswereextractedfrom

ISI-WoS,including;title,discipline,documenttype,afﬁliatedauthors,

ISSN,ISBN,year,edition,pagenumbers,subjectcategories,DOI,

PubMedidentiﬁcationnumber,numberofcitationsandjournal’s

impactfactor.

2.2. DataextractionScopus

Scholarlyoutputproductionbetween2014−2016inScopuswas

extractedutilizing internallywrittenscriptsusingElsevier’sAPI

developers’program.Thescriptwasdividedintotwomain

func-tionalities,MatchandExtract,andwasexecutedinDecember2018.

TheﬁrstpartqueriestheScopusdatabaseauthornamestoobtain

auniqueidentiﬁcationcodeusedinallElsevier’sproductssuch

asScopusandScival.Onlywhenauniquematchwasfoundbased

(4)

Table2

NumberofpublisheddocumentsinbothScopusandWoSdatabasewiththeirrespectivedifferenceandchangeinrankingwhenbenchmarkedperTH.

Teachinghospital Numberof physicians Numberof documents Scopus Numberof documentsWoS Differencein documentsn(%) Differencein positionwhen benchmarked AOPadova 854 5854 3579 2275(38,9%) 1 AOUBologna 868 4605 2727 1878(40,8%) 1 AOUCareggi 1026 4343 2684 1659(38,2%) 1 S.Raffaele-MI 526 4140 3583 557(13,5%) -3 AOUVerona 876 3910 2247 1663(42,5%) 1 Fondaz.IRCCSCaGranda 780 3818 2470 1348(35,3%) -1 IRCCSS.Martino 862 3334 2223 1111(33,3%) 0 AOUPisana 952 3295 2222 1073(32,6%) 0

P.O.SpedaliCiviliBrescia 1039 3184 1766 1418(44,5%) 0

Ist.Clin.Humanitas-Rozzano 679 2928 1305 1623(55,4%) 4

AOUPol.Bari 895 2878 1708 1170(40,7%) -1

IRCCSPoliclinicoSanMatteo 571 2304 1485 819(35,5%) -1

AOUParma 696 2208 1374 834(37,8%) -1

AOUOsp.Riun.Ancona 758 2049 1184 865(42,2%) 3

Osp.S.Gerardo-Monza 735 1961 871 1090(55,6%) 5

AOUSenese 573 1958 1309 649(33,1%) -3

AOPerugia 576 1882 1304 578(30,7%) -2

AOUModena 489 1850 1199 651(35,2%) -2

Osp.L.Sacco-Milano 559 1623 859 764(47,1%) 2

ASUIUdine 745 1573 1017 556(35,3%) -1

AOUFerrara 523 1542 1020 522(33,9%) -3

Osp.S.Paolo-Milano 736 1441 742 699(48,5%) 1

Osp.diCircoloeFond.Macchi 589 1274 801 473(37,1%) -1

ASUITrieste 539 1165 640 525(45,1%) 0

OO.RR.Foggia 412 811 570 241(29,7%) 0

AOTerni 397 479 344 135(28,2%) 0

Total 18.255 66.409 41.233 25.176(37,9%)

searchresultedinmultiplepossibleauthorsamanualvalidation

wasdonebytheauthors,selecting,ormergingtheresearcher

pro-ﬁle(s).

Thesecondpart,extractspublishedworkfromScopusandScival.

AllavailablemetricswereextractedfromScopus,including;title,

DOI,ISSN,Journalname,typeofpublication,coverdata,number

ofcitations,afﬁliationorganization.AsScopusdoesnotallowto

trackanyvaluemetrics,theFWCIperauthorusingtheElsevier

identiﬁcationnumberwasextractedfromScival.

Adetailed description ofthe fullscript can befoundin the

supplementarymaterial;AppendixA(inSupplementarymaterial)–

Extractionofdata.FullscriptdetailsusedtoobtaindatafromScopus

andstatisticalprocedurescanberequestedviathecorresponding

author.StatisticalanalysiswasperformedusingRversion3.5.2.

3. Results

Afterextractingthescholarlyoutputofall26THs,atotal of

66.409and41.233documentsareincludedforanalysisfromScopus

andrespectivelyWoSfromatotalof18.255authors.Descriptive

statistics about the THs can be found in Appendix B (in

Sup-plementarymaterial)–DetailsTeachingHospitals.Documentsare

categorizedasarticles(69,2%Scopus;75,3%Wos),reviews(13,6

%Scopus;14,29%WoS),Letters(7,1%Scopus;9,4%Wos),

editori-als(1,89%Scopus),book(chapters)(2,97%Scopus)orother(5,2%

Scopus;1,0%WoS).Theprecedingtwocategoriesareonlyindexed

inScopus.Table2andFig.1comparethetotalnumberof

docu-mentsperTHinWoSandScopuspublishedbetween2014−2016.

Table3showsanoverviewofthetotalnumberofdocumentsper

THsbetweenWoSandScopusexcludingbook(chapters)and

edi-torialswhicharenotindexedinWoStoprovideamoreaccurate

comparison.

AWilcoxonsigned ranktest wasperformedtocomparethe

differenceofindexeddocumentsinbothdatabases,indicating a

signiﬁcantdifference(p<0.005)inthedocumentsindexedin

Sco-pus(M=2.060,SD=1.122)andWoS(M=1.267,SD=863).When

rankingTHsbasedonthescholarlyoutputasshowninTable2,

almostallorganizationsarebenchmarkedatadifferentposition.

Onaverage,instituteschangetwopositionseitherpositiveor

neg-ative.Thebiggestpositivechangeintherankingwhenlookingat

ScopusistheTeachinghospital“AOSanGerardodiMonza”moving

fromthe20thpositionthe15thposition.

AlthoughthequalitymetricextractedfromScopusandWoS

can-notbe directlycompared witheach othersince WoSmeasures

impactfactorandScopusmeasuresFWCI,investigatingthe

qual-ityofthepublisheddocumentsshowsadifferenceinrankingwhen

benchmarked.AdetailedoverviewcanbefoundinTable4,showing

bothimpactfactor andFWCIoftheinstitutesandtheir

respec-tiverankingwhenbenchmarked.Onaverage,instituteschangeﬁve

positionseitherpositiveornegative.Thebiggestpositivechangein

therankingcanbeseenwith“AOSanGerardodiMonza”moving

fromthe19thpositionthe3rdposition.However,some

organiza-tionsalsomovedownintheranking.AOUCareggiisplacedon18th

positionwhenrankingtheorganizationwithFWCIbutisranked7th

whenbenchmarkingwithimpactfactor.Noneoftheorganizations

remainatthesamepositionwhencomparingthebenchmarkon

ImpactfactororFWCI.

Finally,Fig.2showstherelationshipbetweenqualityand

quan-tity between the published works. Calculating the Spearman’s

rho shows a low but positive correlation between the quality

(FWCI)ofproduceddocumentsandthenumberofdocumentsper

researcher.

4. Discussion

Thispaperdescribesthedifferencesinperformanceof26THsin

Italybyfocusingonthequantityandqualityoftheirresearch.The

goalofthispaperwastoperformabibliometricanalysisfocusingon

twocommonlyusedperformancemetrics,impactfactorandFWCI

usingScopusandISIWebofScienceCoreCollectiontoidentifymain

differencesandpotentialopportunitiesandchallengesforPESsas

(5)

Fig.1.NumberofpublisheddocumentsinbothScopusandWoSdatabasewiththeirrespectivedifferencewhenbenchmarkedperTH.

Table3

NumberofpublisheddocumentsinbothScopusandWoSdatabasewiththeirrespectivedifferenceandchangeinrankingwhenbenchmarkedperTH,excludingbook(chapters) andeditorials.

ISIWoS Scopus(articles&reviews) Scopus(articles,reviews&conferencepapers) Scopus(alldocuments) Numberofpublisheddocuments(2014−2016) 41.233 55.805 56.962 66.409

Difference(%) 14.572(26,1%) 15.729(27,6%) 25.176(37,9%)

Table4

QualityofpublisheddocumentsinbothScopusandWoSdatabasewiththeirrankingandrespectivechangeinrankingwhenbenchmarkedperTH.Qualityisdeﬁnedasthe averageimpactfactororField-weightedcitationimpactofallauthorsafﬁliatedtotheTH.

Teachinghospitalname ImpactFactor Field-WeightedCitationImpact RankingScopus RankingWoS Differenceinpositionwhenbenchmarked

AOUBologna 13,49 2,66 1 4 3

Ist.Clin.Humanitas-Rozzano 10,66 2,54 2 9 7

Osp.S.Gerardo-Monza 5,84 2,38 3 19 16

S.Raffaele-MI 35,70 2,14 4 1 -3

AOPerugia 10,57 2,07 5 10 5

AOPadova 18,03 1,97 6 2 -4

AOUPisana 8,99 1,93 7 12 5

AOUModena 10,19 1,91 8 11 3

IRCCSPoliclinicoSanMatteo 12,06 1,88 9 5 -4

Fondaz.IRCCSCaGranda 14,27 1,85 10 3 -7

P.O.SpedaliCiviliBrescia 7,41 1,84 11 15 4

AOUSenese 8,68 1,82 12 13 1

AOUVerona 10,96 1,79 13 6 -7

AOUOsp.Riun.Ancona 6,23 1,78 14 18 4

AOUFerrara 7,42 1,77 15 14 -1 ASUIUdine 5,40 1,77 16 21 5 IRCCSS.Martino 10,68 1,73 17 8 -9 AOUCareggi 10,69 1,62 18 7 -11 AOTerni 3,52 1,60 19 24 5 OO.RR.Foggia 4,38 1,59 20 23 3

Osp.diCircoloeFond.Macchi 5,03 1,57 21 22 1

Osp.L.Sacco-Milano 5,67 1,56 22 20 -2

AOUParma 6,99 1,54 23 17 -6

Osp.S.Paolo-Milano 3,43 1,52 24 26 2

AOUPol.Bari 7,00 1,48 25 16 -9

ASUITrieste 3,51 1,33 26 25 -1

4.1. Bibliometricanalysis

Extractingthescholarlyoutputshowedasigniﬁcant

discrep-ancybetweentheextracteddatafromthetworepositories.When

extractingthefullscholarlyoutputofthe18.255authorsinthe

sample,Scopus resultedin 37,9 %moredocuments (66.409Vs.

41.233).Toacertaindegree,thisdifferencecanbeexplained.First,

apartfromreviews,articles,conferencepapers,bookchapters,

Sco-pusalsoindexesbooksandeditorialsintheirdatabase.WoSdoes

notincludethesetwo categoriesintotheircorecollection,thus

explaining9,7%ofthevariation.Second,Scopusisknowntobe

moreextensiveintheircoverageincludingover71millionrecords

andcoveringover23,700peer-reviewedjournals[53]whileWoS

includesjustover20.000peer-reviewedjournals[54].Itis,

there-fore,possiblethatsomearticlesarenotindexedinbothdatabases.

Third,sinceauthorsaresearchedusingonlyname,surname,and

afﬁliationand itispossiblethatauthorsareindexeddifferently

inbothdatabasesresultinginamismatchwhenextracting

infor-mation.However,atthisstage,weareunabletoprovideanexact

quantiﬁcationofthisobservedvariation.

Whencomparingthequalityindicatorofbothdatabasesand

benchmarking THs based on impact factor and FWCI, none of

theorganizationsremainatthesameposition.Interestingly,data

(6)

Fig.2. Correlationbetweenquality(FWCI)andpublisheddocumentsperresearcherinScopus.

workspublishedbytheTHs.Haslam&Laham,hypothesizedthat

researchersinmoreprestigiousinstitutionsmayfollowastrategy

wherethefocusismoreonthequalityofpublishedpapersandless

onthequantity[55].Ourresultscontradictthishypothesisproving

apositiverelationshipbetweenquantityandqualityofpublished

worksindicatingthatTHspublishingmoredocumentsalso

pro-ducehigherqualitydocuments.However,itshouldbenotedthat

withthecurrentnumberofobservationstherelationshipisweak

andmightnotsustainwithanincreasedsamplesize.

Archambaultetal.,2009providesevidencethatindicatorsof

scholarlyproductionandcitationsatthecountrylevelarestable

and largely independent of the database reported and no

sig-niﬁcantbibliographicdifferences betweenScopus and WoSare

found [35]. We were able to compare results on the

individ-ualresearcher,nowsuggestingthatasigniﬁcantdifferenceexists

betweenbothrepositories,rejectingtheﬁndingsofArchambault.

However,Archambaultwasunabletoinvestigatethescholarly

out-putonanindividuallevelandfocusedonaninstitutionallevel.

4.2. Impactonperformanceevaluationsystems

Thepresentedﬁndingscanhaveimportantimplicationsforthe

currentuseofperformanceevaluationsystemsinthehealthcare

sector.

IntraditionalPESsdataaremeasured,calculatedandvalidated

bytheorganizationsthemselves,usingbenchmarkinginorderto

compareresultswithothers,ondifferentlevelssuchas

individu-als,departments,andorganizations[14].Usingsecondarybigdata

sourcesopensnewopportunitiestobenchmarkoutsidethe

orga-nizationalboundarieswithotherorganizationsonanationaland

internationallevel.

ThisreducestheroleofeachsingleTHinthecollectionofdata

andreassurestheRegional orNationalHealthSystemaboutthe

reliabilityofthedataitself,asthereisareducedriskof

opportunis-ticdatamanipulation.Thebenchmarkingprocess,ataﬁrstglance,

appearsmorerobust.

Servicestoconsultbibliographicinformationarepublicly

avail-ableandeasilyaccessibleviatheInternet.However,datainthese

systemsarenotmanaged,owned,andoftennotvalidatedbythe

organizationthemselves,butbyexternalpartiessuchasClarivate

Analyticsand Elsevierwhichhavepartly acommercialinterest.

Especiallysincenumerousstudieshaveprovidedevidenceabout

inaccurate information, falsiﬁcation, and fabrication of data in

citationindexserviceswhichaffectandinﬂuencethe

bibliomet-ricmeasures[56–59]. Additionally,metricsmeasuringthesame

constructnamelyqualityoftendifferfromeachotherandareall

subjectedtotheirownadvantagesanddisadvantagesmaking

com-parisonschallenging.

Theseindicatorsareanimportantmanagement toolusedby

theItaliangovernment,theRegionsandtheTHs.Theyusethem

for monitoring and assessing performance, allocating ﬁnancial

resources for research, and evaluating the General Managers’

performance.We want tounderlinethatchoosing oneofthese

databasesisnotsufﬁcient norreliabletobaseimportanthealth

policydecisionsonwithoutincludingcontextualinformation.

Thedifferencesbetweenthetwometricsfoundintheresults,in

fact,highlighttheintrinsicweaknessofthesemetricswhich,tobe

effective,requireasigniﬁcantworktocriticallyassessthemeaning

usingcontextualinformation.Validationoftheoriginofthemetric

isakeystepintheageofBigDatabeforeassessingthemeaningof

themetricitself.Moreover,increasingthescopeofbenchmarked

organizationsprovidesnewinsightstopolicymakers,andcan

sup-port beneﬁcialstrategies when using PESssuchas namingand

shaming[60]orrewardingorganizationsforhigherperformance

[61].Thisgoalcanonlybeachievedifdataarereliable.Thefact

thattheresearchindicatorsarebasedonsystemssuchasWoSand

Scopusdoesnotguaranteepersethepursuitofthiscondition.

Healthsystemsmustaccompanytheuseofthesemetricswith

acontinuoussharingprocesswithallthestakeholdersofthe

sys-temandﬁrstofallwiththeresearchersthemselves[62].Thissame

sharingprocessrepresentstheﬁrstmechanismtoaligneffortsand

commitmenttowardspursuingtheoverallmissionofthe

health-caresystemanditisthebasisoftherelationshipoftrustandesteem

thatallowstofeedandpromoteimprovementprocesses.

Finally, other several issues should be mentioned possibly

inﬂuencingthepresentedresults.First,althougharepresentative

samplesizeof 18.255authorswasused,authorswerenotable

tovalidateeachindividualresearcher.Namesofresearcherswere

providedbyallTHsintheIRPESnetwork,butauthorswerenotable

tovalidatetowhatextenttheseresearcherswereactivelyworking

fortheTHsorprovideanydescriptivestatisticsabouttheseauthors.

Second,datafromWoSwasextractedandprimarilyvalidatedbyan

externalcommercialparty,ResearchValueSRL.Duetocommercial

interests’authorswereunabletoassesstheextractionprocedure

tovalidateaccuracy.Authorswereabletovalidatetheextraction

fromScopusbyaccessingthedevelopers’platformfromElsevier.

Sincetheauthorswerenotabletocomparetheextractionaccuracy

itispossiblethatthedifferenceinthenumberofarticlesfrom

Sco-pusisattributedtoahigheraccuracywhenqueryingScopus.Third,

thescholarlyoutputfromWoSwasextractedinMay2018and

Sco-pus9monthslater.Theeffectonthenumberofpaperswouldbe

(7)

thisdelaysinceimpactfactorandFWCIrelyonthetotalnumberof

citations.Itis,therefore,possiblethattheFWCIispositivelyskewed

comparedwithIF.

Futureresearchshouldaddressthetopicsmentionedaboveby

aligningtheextractionmethodofWoSandScopusandperform

extractionsimultaneously.Additionally,byexpandingtheIRPES

moreTHscanbeincludedtoimprovegeneralizabilityonanational

level.Next,adetailedstudyshouldbeperformedtoanalyzeeach

documenttypeseparately,sincereviewshave,ingeneral,ahigher

impactthanmostotherdocumenttypessuchasarticles,letters,

notes[41].Finally,otherqualitymetricscanbeincludedintothe

analysistofurthercontextualizeFWCIwithotherqualityindicators

bylookingat,butnotlimitedto,cross-checkinggrants,

collabo-rationswithotherresearchinstitutes,andpercentage ofpapers

publishintop5percentilejournals.

5. Conclusion

Toourknowledge,nopriorresearchwasperformedto

iden-tifyandhighlightthedifferencesofresearchperformanceofTHs

withrespecttoquantityandqualitymetricsusingtheirpublished

workswhileincludingalargesampleofindividualphysicians.

Uti-lizingsecondary BigDatasourcesforperformance management

is,ontheonehand,usefulbecausetheyallowbenchmarkingat

anational andinternationallevel,but ontheotherhand,using

differentdatasources tomeasurethesameconstructofquality

andquantity,clearlyleadtodifferentresultswhenbenchmarked

againsteachother.

Researchactivitiesareanobjectivetobepursuedandispart

ofthemissionofboththeHealthcareSystemasawholeandthe

providerswhooperatewithintheSystem.Amongtheproviders,

intheﬁrstplacetherearetheTHs,withtheirtriple-foldmission

ofresearch,careandtraining.Followingtheirmission,THshavean

intrinsicmotivationtodeliverhighperformanceonallthreepillars.

Measuringtheperformanceoftheresearchactivitiesisessential

butcomplex.Web-basedtoolsallowtoensureabenchmarking

pro-cessondifferentlevelsandcanbeeffectivelyusedataHealthcare

Systemlevelfordifferentgovernancepurposessuchasplanning,

designingincentivesforresearch,andallocatingresources.

Web-basedtoolshaveweaknessesandrequireaformalinternaldataand

validationprocesstoavoidself-referral.Thiscanbeovercomeby

settingupatransparentprocesssharedamonghealth

profession-als,hospitalmanagementandpolicymakers,whichcancontribute

andinturnimproveresearchperformance.

Ethicsapproval

NotApplicable.

Consenttoparticipate

Participating teaching hospitals in the network of

measur-ingperformanceprovidedauthorswiththeirafﬁliatedemployed

researchers.Final analysis wasperformed onan organizational

levelandemployedresearcherswerenotinvolved,contactedor

analyzedonanindividuallevelinanywayduringthestudy.

Consentforpublication

Responsible region representatives have approved the ﬁnal

results.

Availabilityofdataandmaterial

Thedatasets,scriptsoranyothersupplementarymaterialused

andanalyzedduringthecurrentstudyareavailablefromthe

corre-spondingauthoronreasonablerequest.DataobtainedfromSciVal®

database,ElsevierB.V.,http://www.scival.com

Funding

FH is working as a fellow in a project

(www.healthPros-h2020.eu)thathasreceivedfundingfromtheEuropeanUnion’s

Horizon2020researchandinnovationprogrammeundertheMarie

Skłodowska-CuriegrantagreementNo765141.Theoverallproject

ispartlyﬁnancedbyItalianregionswithintheIRPES.

Authors’contributions

StudyconceptionwascreatedbySN;studydesignwascreated

byFHandDAL.AcquisitionofdatawasperformedbyFH,DAL.

Anal-ysisandinterpretationofdatawasperformedbyFH.Draftingof

themanuscriptwasperformedbyFHandDAL.SNwasinvolved

incriticalrevisionsofthemanuscriptandcontributedinwriting

thebackground,discussionandconclusionparagraphs.Allauthors

havereadandapprovedthesubmittedmanuscript.

DeclarationofCompetingInterest

Theauthorsreportnodeclarationsofinterest.

Acknowledgments

Authorswouldliketothanktheparticipationoftheregional

networkinprovidinguswithinputfordatacollection.Thispaper

ispartofaproject(www.healthpros-h2020.eu)thathasreceived

fundingfromthe EuropeanUnion’sHorizon 2020researchand

innovationprogrammeundertheMarieSkłodowska-Curiegrant

agreementNo765141.

AppendixA. Supplementarydata

Supplementarymaterialrelatedtothisarticlecanbefound,in

theonlineversion,atdoi:https://doi.org/10.1016/j.healthpol.2020.

10.002.

References

[1]MarrB,GrayD.Strategicperformancemanagement.TaylorFrancis;2006,240 p.

[2]MoullinM.Performancemeasurementdeﬁnitions:linkingperformance mea-surementandorganisationalexcellence.InternationalJournalofHealthCare QualityAssurance2007;20(3):181–3.

[3]MoullinM.Deliveringexcellenceinhealthandsocialcare:quality,excellence, andperformancemeasurement.OpenUniversityPress;2002.

[4]Bourne M, Mills J. Designing, implementing and updating performance measurementsystems.InternationalJournalofOperationsProduction& Man-agement2000;20(7):754–71.

[5]KaplanD,RobertS,NortonD.Thebalancedscorecard:translatingstrategyinto action.Boston:HarvardBusinessSchoolPress;1996.

[6]NeelyA. Theperformancemeasurementrevolution:why nowand what next? International Journal of Operations Production & Management 1999;19(2):205–28.

[7]KaleS,TamakuwalaH,VijayakumarV,YangL,RawalKshatriyaBS.Bigdatain healthcare:challengesandpromise.In:Smartinnovation,systemsand tech-nologies.Springer;2020.p.3–17.

[8]NgiamKY,KhorIW.Bigdataandmachinelearningalgorithmsforhealth-care delivery.LancetOncology2019;20:e262–73.LancetPublishingGroup.

[9]PorterME.Whatisvalueinhealthcare?NewEnglandJournalofMedicine [Internet]2010;363(26):2477–81,http://dx.doi.org/10.1056/NEJMp1011024. Dec23[cited2019Sep19];Availablefrom:.

[10]GrayM.Populationhealthcare:designingpopulation-basedsystems.Journal ofRoyalSocietyMedicine[Internet]2017;110(5):183–7,http://dx.doi.org/10. 1177/0141076817703028.May12[cited2019Jul3];Availablefrom:. [11]BerwickDM,NolanTW,WhittingtonJ.Thetripleaim:care,health,andcost.

HealthAffairs2008;27(3):759–69.

[12]NutiS,NotoG,VolaF,VainieriM.Let’splaythepatientsmusicsystemsin healthcare.ManagementDecision2018;56(10):2252–72.

(8)

[13]NutiS.Makinggovernanceworkinthehealthcaresector:evidencefroma ‘naturalexperiment’inItaly.PolicyLaw2016;11(February2015):17–38.

[14]BititciU,GarengoP,NudurupatiS.Performancemeasurement:challengesfor tomorrow.InternationalJournalofManagementReviews2012;14:305–27.

[15]BevanG.Reputationscount:whybenchmarkingperformanceisimproving healthcareacrosstheworld.HealEconomicsPolicyLaw2019;(14):141–61.

[16]NutiS,RuggieriT,PodettiS.Douniversityhospitalsperformbetterthan generalhospitals?AcomparativeanalysisamongItalianregions.BMJOpen 2016;6(011426).

[17]PrasadV,GoldsteinJA.USnewsandworldreportcancerhospitalrankings:do theyreﬂectmeasuresofresearchproductivity?PLoSOne2014;9(9):1–6.

[18]KrzyzanowskaMK,KaplanR,SullivanR.Howmayclinicalresearchimprove healthcare:outcomes?AnnalsOncology2011;22(Suppl.7):10–5.

[19]MajumdarSR,ChangWC,ArmstrongPW.Dotheinvestigativesitesthattake partinapositiveclinicaltrialtranslatethatevidenceintopractice?American JournalofMedicine2002;113(November(2)):140–5.

[20]KanavosP,SullivanR,LewisonG,SchurerW,EckhouseS,VlachopiotiZ.The roleoffundingandpoliciesoninnovationincancerdrugdevelopment. Ecan-cermedicalscience2010;Vol.4:1–139.

[23]JanniW,KiechleM,SommerH,RackB,GaugerK,HeinrigsM,etal.Study participationimprovestreatmentstrategiesandindividualpatientcarein par-ticipatingcenters.AnticancerResearch2006;26(September(5B)):3661–7.

[24]Birkmeyer JD,Stukel TA, SiewersAE, GoodneyPP, Wennberg DE, Lucas FL. Surgeonvolume and operative mortality in the United States. New EnglandJournalofMedicine[Internet]2003;349(November(22)):2117–27,

http://dx.doi.org/10.1056/NEJMsa035205 [cited 2020 May 9]; Available from:.

[25]BirkmeyerJD,DimickJB,BirkmeyerNJO.Measuringthequalityofsurgicalcare: structure,process,oroutcomes?11Nocompetinginterestsdeclared.Journalof AmericanCollegeSurgery2004;198(April(4)):626–32.

[26]duBoisA,RochonJ,PﬁstererJ,HoskinsWJ.Variationsininstitutional infrastruc-ture,physicianspecializationandexperience,andoutcomeinovariancancer: asystematicreview.GynecologyOncology2009;112:422–36.

[27]AbramoG,D’AngeloCA.Howdoyoudeﬁneandmeasureresearchproductivity? Scientometrics2014;101(2):1129–44.

[28]KreimanG,MaunsellJHR.Ninecriteriaforameasureofscientiﬁcoutput.Front ComputingNeuroscience2011;5(48):1–6.

[29]DewettT,DenisiA.Exploringscholarlyreputationit’smorethanjust produc-tivity.Scientometrics2004;60(2):249–72.

[30]Broadus RN. Toward a deﬁnition of “bibliometrics”. Scientometrics 1987;12(5–6):373–9.

[31]Gross EM. College libraries and chemical education. Science (80-) 1927;66(1713):385–9.

[32]SternbergRJ,StatesU.Journalofappliedresearchinmemoryandcognition evaluatingmeritamongscientists.JournalofAppliedResearchMemoir Cogni-tive2018;7(2):209–16.

[33]GrechV.Increasingimportanceofresearchmetrics:journalImpactFactorand h-indexH-index.InternationalUrogynecologicalAssociation2018;29:619–20.

[34]CockrielWM,McdonaldJB.Theinﬂuenceofdispersiononjournalimpact mea-sures.Scientometrics2018;116(1):609–22.

[35]ArchambaultE,CampbellD,GingrasY,LarivièreV.Comparingbibliometric statisticsobtainedfromthewebofscienceandScopus.JournalofAmerican SocietyInformationScienceTechnology2009;60(7):1320–6.

[36]GarﬁeldE.Citationindexesforscience.Science(80-)1955;6:31–2.

[37]GarﬁeldE.Thehistoryandmeaningofthejournalimpactfactor.Journalof AmericanMedicalAssociation2006;295(1):1–4.

[38]GarﬁeldE.Howcanimpactfactorsbeimproved?BMJ1996;313:411–3.

[39]MutzR,DanielH.Skewedcitationdistributionsandbiasfactors:solutions totwocoreproblemswiththejournalimpactfactor.JournalofInformatics 2012;6(2):169–76.

[40]SeglenP.Theskewnessofscience.JournalofAmericanSocietyInformation Science1992;43(9):628–38.

[41]VanLeeuwenT,MoedHF,ReedijkJ.CriticalcommentsonInstitutefor Scien-tiﬁcInformationimpactfactors:asampleofinorganicmolecularchemistry journals.JournalofInformationScience1999;25(6):489–98.

[42]SeglenPO.Whytheimpactfactorofjournalsshouldnotbeusedforevaluating research.BMJ1997;314:498–513.

[43]AlbertsB.Impactfactordistortions.Science(80-)2013;340(6134):787.

[44]LariviereV,KiermerVV,MacCallumCJ,McnuttM,PattersonM,PulvererB,etal. Asimpleproposalforthepublicationofjournalcitationdistributions.bioRxiv 2016:1–23.

[45]MinnickJ.The2018JCRishere!Clarivateanalytics;2018.

[46]NarinF, HamiltonSK.Bibliometricperformancemeasures.Scientometrics 1996;36(3):293–310.

[47]Researchmetricsguidebook.Elsevier;2018.

[48]HausteinS.Grandchallengesinaltmetrics:heterogeneity,dataqualityand dependencies.Scientometrics2016;108(1):413–23.

[49]MintzbergH,UpSaddleRiverStructureinFives;DesigninEffective Organiza-tions;1992.

[50]NutiS,D’OrioG,GambaMP.Ilsistemadivalutazionedellaperformancedei sistemisanitariregionali;IrisultatidelleAziendeOspedaliero-Universitariea confronto;2017.

[51]NutiS,RuggieriT.LavalutazionedellaperformancedelleAziende Ospedaliero-Universitarie.Finalità,metodierisultatiaconfronto2016:109.

[52]VainieriM,VolaF,GomezG,NutiS.Howtosetchallenginggoalsandconduct fairevaluationinregionalpublichealthsystems.InsightsfromValenciaand TuscanyRegions.HealthPolicy(NewYork)2016;120(11):1270–8.

[53]Aneyeonglobalresearch.Elsevier;2018.

[54]CarloniM.Webofsciencecorecollectiondescriptivedocument;2018.

[55]HaslamN,LahamSM.Quality,quantity,andimpactinacademicpublication. EuropeanJournalofSocietyPsychology2010;40:216–20.

[56]FranceschiniF,MaisanoD,MastrogiacomoL.Empiricalanalysisand classiﬁca-tionofdatabaseerrorsinScopusandWebofScience.JournalofInformatics 2016;10(4):933–53.

[57]FranceschiniF,MaisanoD.Themuseumoferrors/horrorsinScopus.Journal ofInformatics2016;10(1):174–82.

[58]López-cózarED,Robinson-garcíaN,Torres-salinasD.Thegooglescholar exper-iment:howtoindexfalsepapersandmanipulatebibliometricindicators. JournalofAssociationInformationScience&Technology2014;65(3):446–54.

[59]Bartneck C,KokkelmansS.Detectingh-indexmanipulationthrough self-citationanalysis.Scientometrics2011;87:85–98.

[60]BevanG,WilsonD.Does‘namingandshaming’workforschoolsandhospitals? LessonsfromnaturalexperimentsfollowingdevolutioninEnglandandWales. PublicMoneyManagement[Internet]2013;33(July(4)):245–52,http://dx.doi. org/10.1080/09540962.2013.799801[cited2019Jul26];Availablefrom:. [61]VainieriM,LunguDA,NutiS.Insightsontheeffectivenessofrewardschemes

from10-yearlongitudinalcasestudiesin2Italianregions.International Jour-nalofHealPlanManagement2018;33(2):474–84.

[62]NutiS,BiniB,RuggieriTG,PiaggesiA,RicciL,GrilloRuggieriTG,etal.Bridging thegapbetweentheoryandpracticeinintegratedcare:thecaseofthediabetic footpathwayinTuscany.InternationalJournalofIntegrativeCare2016;16(May (2)):9.