Hidden connections: Network effects on editorial decisions in four computer science journals

(1)

ContentslistsavailableatScienceDirect

Journal

of

Informetrics

jo u r n al hom e p ag e :w w w . e l s e v i e r . c o m / l o c a t e / j o i

Regular

article

Hidden

connections:

Network

effects

on

editorial

decisions

in

four

computer

science

journals

Giangiacomo

Bravo

a,b,∗

_,

_Mike

_Farjam

a,b

_,

_Francisco

_Grimaldo

_Moreno

c

_,

Aliaksandr

Birukou

e

_,

_Flaminio

_Squazzoni

d

a_Department_of_Social_Studies,_Linnaeus_University,_Växjö,_Sweden

b_Linnaeus_University_Centre_for_Data_Intensive_Sciences_&_{Applications,}_Växjö,_Sweden c_Department_of_Informatics,_University_of_Valencia,_Valencia,_Spain

d_Department_of_Economics_and_Management,_University_of_Brescia,_Brescia,_Italy e_Springer_Nature,_Heidelberg,_Germany

a

r

t

i

c

l

e

i

n

f

o

Articlehistory:

Received8September2017 Receivedinrevisedform 13November2017 Accepted2December2017 Keywords: Editorialbias Networkeffects Authorreputation Peerreview Bayesiannetwork

a

b

s

t

r

a

c

t

Thispaperaimstoexaminetheinfluenceofauthors’reputationoneditorialbiasinscholarly journals.Bylookingateightyearsofeditorialdecisionsinfourcomputersciencejournals, including 7179 observations on 2913 submissions, we reconstructed author/referee-submissionnetworks.Foreachsubmission,welookedatreviewerscoresandestimatedthe reputationofsubmissionauthorsbymeansoftheirnetworkdegree.BytrainingaBayesian network,weestimatedthepotentialeffectofscientistreputationoneditorialdecisions. Resultsshowedthatmorereputedauthorswerelesslikelytoberejectedbyeditorswhen theysubmittedpapersreceivingnegativereviews.Althoughthesefourjournalswere com-parableforscopeandareas,wefoundcertainjournalspecificitiesintheireditorialprocess. Ourfindingssuggestwaystoexaminetheeditorialprocessinrelativelysimilarjournals withoutrecurringtoin-depthindividualdata,whicharerarelyavailablefromscholarly journals.

1. Introduction

Peerreviewisadecentralised,distributedcollaborationprocessthroughwhichexpertsscrutinisethequality,rigourand noveltyofresearchfindingssubmittedbypeersbeforepublication.Theinteractionbetweenallfiguresinvolved,including editors,refereesandauthors,helpstofilteroutworkthatispoorlydoneorunimportant.Atthesametime,bystimulating aconstructivedialoguebetweenexperts,thisprocesscontributestoimproveresearch(e.g.,Casnici,Grimaldo,Gilbert,& Squazzoni,2016;Righi&Takács,2017).Thisiscrucialforjournals,scienceandknowledgedevelopment,butalsoforacademic institutions,asscientistreputationandcareerarelargelydeterminedbyjournalpublications(Bornmann&Williams,2017; Fyfe,2015;Squazzoni&Gandelli,2012,2013).

Undertheimperativesofthedominant“publishorperish”academicculture,journaleditorsarecalledeverydaytomake delicatedecisionsaboutmanuscripts,whichnotonlyaffecttheperceptionofqualityoftheirjournalsbutalsocontribute tosetresearchstandards(Petersen,Hattke,&Vogel,2017;Siler,Lee,&Beroc,2015)bypromotingcertaindiscoveriesand

∗ Correspondingauthorat:DepartmentofSocialStudies,LinnaeusUniversity,Växjö,Sweden.

E-mail addresses: giangiacomo.bravo@lnu.se (G. Bravo), mike.farjam@lnu.se (M. Farjam), francisco.grimaldo@uv.es (F. Grimaldo Moreno), aliaksandr.birukou@springer.com(A.Birukou),ﬂaminio.squazzoni@unibs.it(F.Squazzoni).

https://doi.org/10.1016/j.joi.2017.12.002

(2)

Table1

Reviewrounddistributionbyjournalintheoriginaldatabase.

Reviewround Journal

J1 J2 J3 J4 1 4025 3233 3534 1098 2 1117 675 563 194 3 144 121 112 21 4 6 11 11 3 5 1 0 0 0 6 1 0 0 0

methodswhilerejectingothers(Bornmann&Daniel,2009;Lin,Hou,&Wu,2016;Resnik&Elmore,2016).Withthehelp

ofreferees,theyhavetomakethewholeeditorialprocessaseffectiveaspossiblewithoutfallingintothetrapofcognitive, institutionalorsubjectivebiases,mostlyhiddenandevenimplicit(Birukouetal.,2011;Casnici,Grimaldo,Gilbert,Dondio, &Squazzoni,2017;Lee,Sugimoto,Zhang,&Cronin,2013;Teele&Thelen,2017).

Unfortunately,althoughlargelydebatedandalwaysunderthespotlight,peerreviewandtheeditorialprocesshave beenrarelyexaminedempiricallyandquantitativelywithin-depth,across-journaldata(Batagelj,Ferligoj,&Squazzoni, 2017;Bornmann,2011).Whileeditorialbiashasbeeninvestigatedinspecificcontexts(e.g.,Hsiehchen&Espinoza,2016; Moustafa,2015),theroleofbiasduetohiddenconnectionsbetweenauthors,refereesandeditors,whicharedetermined bytheirreputationandpositioninthecommunitynetwork,hasbeenexaminedrarelyempirically(García, Rodriguez-Sánchez,&Fdez-Valdivia,2015;Grimaldo&Paolucci,2013;Squazzoni&Gandelli,2013).Anoteworthyexceptionhasbeen Sarigöl,Garcia,Scholtes,andSchweitzer(2017)whorecentlyexaminedmorethan100,000articlespublishedinPLoSONE between2007and2015tounderstandwhetherco-authorshiprelationsbetweenauthorsandthehandlingeditoraffected themanuscripthandlingtime.Theirresultsshowedthateditorshandledsubmissionsco-authoredbypreviouscollaborators significantlymoreoftenthanexpectedatrandom,andthatsuchpriorco-authorrelationsweresignificantlyrelatedtofaster manuscripthandling.Inthesecases,editorialdecisionswerespeduponaverageby19days.However,thisanalysiscould notlookatwholeeditorialprocess,includingrejectionsandrefereeselection,andcouldnotdisentangleeditorialbiasfrom submissionauthors’strategiesofeditors’targeting.

Ourstudyaimstofillthisgapbypresentingacomprehensiveanalysisofeightyearsoftheeditorialprocessinfour computersciencejournals.First,thesejournalswerecomparableintermsofscopeandthematicareas,soprovidingan interestingpictureofacommunityanditsnetworkstructure.Secondly,welookedatthewholeeditorialprocess,witha particularfocusoneditorialdecisionsandrefereerecommendationsfromallsubmissions.Whilewedidnothavedataon allcharacteristicsofauthorsandsubmissionsandsocouldnotdevelopintrinsicestimatesofamanuscript’squality,we usednetworkdatatotracethepotentialeffectofauthors’centralityoneditorialdecisionsoncecheckedfortheeffectof reviewscores.Inthisrespect,ourstudyoffersamethodtoexamineeditorialbiaswithoutin-depthandcompletedata, whicharerarelyavailablefromjournals(Squazzoni,Grimaldo,&Marusic,2017).Moresubstantively,ouranalysisrevealed thatalthoughthesefourjournalswereinthesamefield,theireditorialprocesseswerejournal-specific,e.g.,influencedby rejectionratesandimpactfactor.Secondlyandmoreinterestingly,ourfindingsshowthatmorereputedauthorswereless penalisedbyeditorswhentheydidnotpresumablysubmitbrilliantwork.

Therestofthepaperisorganisedasfollows.Section2presentsourdataset,includingdataanonymisationproceduresand theconstructionofourmainvariables.Section3presentsourﬁndings,includingaBayesiannetworkmodelthatallowedus todisentanglebiasthroughouttheeditorialprocess,whileSection4discussesthemainlimitationsofourstudyandsuggests futuredevelopments.

2. Datasetconstruction

2.1. Dataextractionandpreparation

DatawereacquiredbyfollowingaprotocoldevelopedbyanetworkofscientistsandpublishersaspartoftheTD1306 COSTActionNewfrontiersofpeerreview(hereinafter,PEERE),whichallowedustosharedataonpeerreviewbyprotecting theinterestsofallstakeholdersinvolved(Squazzonietal.,2017).

Ourdataincluded14,870observationsencompassingseveralyearsofeditorialactionsinfourcomputersciencejournals (hereinafterJ1–J4).Eachobservationcorrespondedtooneactionperformedbytheeditor,suchasaskingarefereetoreview apaper,receivingareport,ortakinganeditorialdecisionoverthepaper.Thetimeframechosendependedonthedata availabilityforeachjournal.Morespeciﬁcally,weused10yearsofobservationsforJ1,12forJ2,7forJ3(amorerecently establishedoutlet),and9yearsforJ4.Foralljournals,observationswerelimitedtoJanuary2016.About80%ofthecollected datareferredtotheﬁrstroundofreviews(Table1).

Thetargetjournalsrevolvedaroundcoherentthematicareas,thoughtheyvariedintheirrankingofimpact.Notably,J1 wasrankedintheﬁrstquartileofthe2016JournalCitationReports(JCR)impactfactor(IF)distribution,J2wasincludedin thefourthquartileoftheIFdistribution,J3inthethird,whileJ4wasnotindexedintheJCR.Foreachjournal,weextracted

(3)

dataonmanuscriptssubmitted,reviewerrecommendationsandeditorialdecisions.Datawerecleanedandanonymised, withscientistnamesandsubmissiontitlesreplacedbysecurehashidentiﬁers(IDs)oftheSHA-256type,whichdigested theoriginalstringsremovingaccentsandnonalphanumericcharacters(Schneier,1996).TheseautomaticallygeneratedIDs allowedustotrackalltheentitieswhilepreservingtheirprivacy.Notethatthesehashcodespreventedustodisambiguate authororrefereenamesbutalsothatnamesakeswereeventuallyassignedthesameIDwhereasdifferentspellingsofthe samenamewouldendupindifferentIDS.Notthatthisonlyintroducedamarginaldistortioninourdataset.First,such homonymswereratherunlikelygiventhatweconsideredalimitednumbersofjournals.Inaddition,asthesejournalsuse thesamejournalmanagementsystem,scientistshadauniqueproﬁleandonlyonespellingforeachnamewasavailable.

WerestrictedourattentiontosubmissionswhichhadatleastoneactiverefereeIDinthedataset,meaningthatatleast onerefereewasinvitedbytheeditor(i.e.,thepaperwasnotdesk-rejectedorretractedbeforereview).Thisledto11,516 observationson3018submissions.However,4250ofthesecasesdidnothaveanyrefereerecommendationsbecausethe refereeanswerednegativelytotheeditor’sreviewrequest,didnotsendareportortherecommendationwassentlaterthan January2016.Theseobservationswerenotconsideredintheanalysis.

Furthermore,giventhepurposeofourwork,weconsideredonlyobservationswithanunequivocaleditorialdecision basedatleastonerefereerecommendation.Weeliminated61submissionsforwhich,althoughtheIDofoneormorereferee existed(andcouldhencebeconsidered,forinstance,inthenetworkanalysisbelow),nodecisionwasrecorded,alongwith 44submissionswithnon-standarddecisions(e.g.,“TerminatedbyEditor-in-Chief”or“Skipped”).Thisledtoaﬁnaldataset including7179observationson2913submissions.Theseincludedonereviewreportandrecommendationperobservation, alongwithonecleareditorialdecisionpersubmission.

2.2. Refereerecommendations

Refereerecommendations(whichsometimesappearedasnon-standardexpressionsinthedatabase)wereﬁrstrecoded intothestandardordinalscaleaccept,minorrevisions,majorrevisions,reject.Inordertousetherefereerecommendations efﬁcientlyandtesttheireffectoneditorialdecisions,weestimatedanumericalscoreforanyactualsetofreferee recom-mendations.Beingthisanordinal-andnotaninterval-scalevariable,simplycomputingthesumofranksofthedifferent recommendationswasnotcorrect.Weinsteaddecidedtoderivethescorestartingfromthereviewdistributionthatwe wouldexpectifwehadnopriorsonhowcommoneachofthefourrecommendationswere(i.e.,weassumedthattheywere equiprobable).Inpractice,wederivedthesetofallpossiblerecommendationsforagivennumberofreviewsandsimply countedhowmanywereclearlybetterorworsethantheoneactuallyreceived.Forinstance,incasetherewasonlyone refereereportandtherecommendationwasaccept,threepossiblelessfavourableandnoonemorefavourablecasesexisted. Whentherecommendationwasmajorrevision,weassumedtherewereoneworseandtwobettercases.

Moregenerally,weusedthefollowingproceduretocalculatethereviewscores.Weﬁrstderivedthesetofallpossible uniquecombinationsofrecommendationsforeachsubmission(henceforth,potentialrecommendationset).Usingthisset, wecountedthenumberofcombinationsthatwereclearlylessfavourable(#worse)ormorefavourable(#better)thantheone actuallyreceivedbythesubmission(e.g.,{accept,accept}wasclearlybetterthan{reject,reject}).Notethatathirdgroupof combinationsexisted,whichcouldbeﬁrmlyconsideredeitherbetterorworsethanthetarget(e.g.,{majorrevisions,major revisions}wasneitherclearlybetternorworsethan{accept,reject}).

Thisallowedustoassignan“optimistic”estimateofthevalueofanyactualcombinationofrecommendations,considering thewholesetofpossiblerecommendations,ora“pessimistic”one,whenconsideringonly“clear”cases.Thetworesulting reviewscoreswerecomputedusingEqs.(1)and(2)respectively.

reviewScoreoptimistic=

#worse

#better+#worse (1)

reviewScorepessimistic=

#worse

#better+#worse+#unclear (2)

Table2showshowthereviewscoreestimationworksincaseoftworecommendations,whilesimilartablescanbe producedforanynumberofrecommendations.Oneofthemostinterestingaspectsofthisprocedureisthattheresulting scoreisalwaysboundedinthe[0,1]interval,whichmakeseasytocomparepapershavingadifferentnumberofreferees.

Itisworthnotingthattheranksproducedbyour“optimistic”and“pessimistic”scoresdonotchangeinTable2,nordothey foradifferentnumberofrefereerecommendations(notethat,given(1)and(2),reviewScorepessimistic≤reviewScoreoptimisticas longas#unclear≥0).Furthermore,giventhatinallouranalysesthetwoscoresledtoqualitativelysimilarestimates,from nowonwereportedonlyresultsobtainedbythe“optimistic”scoreonly,calledreviewscore.

Finally,weestimatedarefereedisagreementscoreasthenumberofrefereerecommendationsthatshouldbechangedto reachaperfectagreementamongthereferees,dividedbythenumberofrefereessothatcomparabilityacrosspaperswith adifferentnumberofreviewscouldbeachieved.Forinstance,incaseofthreereviewsrecommendationssuchas{accept,

accept,minorrevisions},thedisagreementscorewouldbe1/3;incaseof{accept,majorrevisions,minorrevisions},itwould be2/3.

(4)

Table2

Reviewscoreestimationforallpossiblecombinationsforatwo-recommendationset.

Recommendations Potentialrecommendationset Reviewscore

#better #worse #unclear Pessimistic Optimistic

{accept,accept} 0 9 0 1.00 1.00

{accept,minorrevisions} 1 8 0 0.89 0.89

{accept,majorrevisions} 2 6 1 0.67 0.75

{minorrevisions,minorrevisions} 2 5 2 0.56 0.71

{minorrevisions,majorrevisions} 4 4 1 0.44 0.50

{accept,reject} 3 3 3 0.33 0.50

{majorrevisions,majorrevisions} 5 2 2 0.22 0.29

{minorrevisions,reject} 6 2 1 0.22 0.25

{majorrevisions,reject} 8 1 0 0.11 0.11

{reject,reject} 9 0 0 0.00 0.00

Fig.1.Author–papernetwork.(a)Completeafﬁliationnetwork.Papersaredrawnasbluesquares,authorsasredcircles.(b)Bipartiteprojectiononpapers. Coloursindicatejournalsinwhichpapersweresubmitted.(Forinterpretationofthereferencestocolourinthisﬁgurelegend,thereaderisreferredtothe webversionofthearticle.)

2.3. Networkcentralitymeasures

Welookedatsubmissionsasthetargetofournetworkanalysisasthesecouldbeconsideredas“objects”thatlinkedall

theﬁguresinvolvedalthoughindifferentroles,suchasauthors,refereesandeditors.Wewantedtoestablishtheposition

ofeachpaperintheauthorsandreferees’networks.Asboththeauthor(s)andthereferee(s)ofagivenpaperhadadirect

relationwithit,atripartiteafﬁliationnetwork1_exists_in_principle,_where_both_authors_and_referees_are_{“afﬁliated”}_(i.e.,_hold

links)withthepapertheywroteorreviewed.Nevertheless,asingleresearchercouldactbothasauthorandrefereeover time.Forsakeofsimplicity,weconsideredtwoseparatebipartitenetworks,theoneincludingsubmissionsandauthorsand theoneincludingsubmissionsandreferees,andseparatelyderivecentralitymeasuresforeachofthem.Bothnetworkswere derivedfromthedatasetfollowingtheproceduresbelow.

2.3.1. Author–paperafﬁliationnetwork

Authorsandpapersincludedinthedatasetformanetworkwhereauthorsholdadirectedlinktothepaper(s)theywrote. Thisresultedinabipartitenetworkwith10,049nodes(7031authorsand3018papers)and9275links(Fig.1a).Two-thirdsof thepapersinthenetworkhadthreeorlessauthorsand10.7%onlyoneauthor.Inaddition,only18%oftheauthorssubmitted morethanonepapertothejournalsincludedinthedatabase,whichresultedinaverylownetworkdensity(0.00009).

Thebipartiteprojectionofthenetworkonpapers(alsoknownasco-membershipnetwork)consideredpapersaslinked iftheysharedatleastoneauthor.Italsohadlowdensity(0.00009),withover40%ofpaperswhichwereisolated.Onthe otherhand,wefoundalargeclusterofwellconnectedpapers,mostofthemsubmittedtotwojournals(redandyellowin Fig.1b).Weestimatedthepositionofthepaperinthenetworkbyusingstandardcentralitymeasures,suchasthedegree andeigenvectorcentrality,thelatteroftenusedintheanalysisofafﬁliationnetworks(Faust,1997).Thedegreeofagiven

1 _This_should_not_be_confused_with_the_academic_affiliation_of_authors_and_referees._Following_the_standard_definition_used_in_network_analysis,_here authorsandrefereesare“affiliated”tothepapertheyrespectivelywroteorreviewed.

(5)

Fig.2.Referee–papernetwork.(a)Completeafﬁliationnetwork.Papersaredrawnasbluesquares,authorsasredcircles.(b)Bipartiteprojectiononpapers. Coloursindicatejournalsinwhichpapersweresubmitted.(Forinterpretationofthereferencestocolourinthisﬁgurelegend,thereaderisreferredtothe webversionofthearticle.)

nodesimplyreportsthenumberoflinksitholds,whiletheeigenvectorcentralityisamorecomplexmeasuretakinginto accountnotonlythepositionofthenodeinthenetworkbutalsothecentralityoftheothernodestowhichitisconnected.2 Thecorrespondingstatisticswerecomputedandsavedforallnodesinthepaperprojection,i.e.,foreachpaperincludedin thedataset.

2.3.2. Referee–paperafﬁliationnetwork

Similarly,werepresentedrefereeandpapersasanetworkwhererefereesheldadirectedlinktothepapertheyreviewed. Afterapplyingthisruletothedataset,wefoundabipartitenetworkwith8546nodes(5528refereesand3018papers) and11,516links(Fig.2a).Only11.9%papersinthenetworkwerereviewedbyonlyonereferee,and52.7%bythreeor more.Furthermore,33%oftherefereesreviewedmorethanonepaper.Here,thenetworkdensitywashigherthaninthe author–papernetworkcase(0.0002).

Inthiscase,theprojectionofthenetworkonpapersconsideredtwopaperslinkedwhentheywerereviewedbyatleast acommonreferee.Resultsshowedthatthisnetworkhadahigherdensitythantheoriginalbipartitenetwork(0.08),with only8.6%ofthepapersthatwereisolated.Mostofthemwereactuallyincludedinaverylargecomponent,evenifthefour separatejournalclusterwerestillidentiﬁable(Fig.2b).Asbefore,refereecentralitymeasureswerecomputedandsavedfor allnodesinthepaperprojection.

3. Results 3.1. Descriptives

Thefinaldatasetaggregatingfirstroundreviewsandeditorialdecisionsforeachsubmissionincluded2957observations. Papersweresubmittedtofourjournals:29.2%tothefirstone(J1),33.0%toJ2,30.3%toJ3,and7.5%toJ4.Table3showsthe correspondingfrequencydistributionofrefereerecommendations(a)andeditorialdecisions(b).

Resultsindicatethateditorialdecisionsandthecombinedscoresofrefereerecommendationswerealigned(Table4). Itisalsoworthnotingthatthedisagreementamongrefereeswaslowerwhenthesubmissionwasacceptedbytheeditor, whereasitsigniﬁcantlyincreasedincaseofmajorrevisions(t=3.10,p=0.004,differencesbetween“accept”andtheother groupswerenotsigniﬁcant).ConsistentlywithCasnicietal.(2017),reportslinkedtorejectionandmajorrevisiondecisions werepredominantlylongerthanincaseofacceptanceandminorrevisions,soindicatingthewillingnessofrefereestojustify theiropinionwithdetailsorhelpauthorsimprovingtheirwork.

Thetwonetworkspresentedasimilarunderlyingstructurealthoughwithsigniﬁcantdensitydifferences.Degreesinboth theauthor–paperandreferee–papernetworkswereexponentiallydistributedwithcoefﬁcients<1,meaningthatamajority ofpapershadnoorjustafewlinks,whileasmallnumberofpapershadmanymorelinks(upto55fortheauthor–paperand to183forthereferee–papernetwork).Thefactthatthedensityofthenetworkwashigherforthereferee–papernetworkdid

2_In_a_preliminary_stage_of_the_research,_we_also_computed_the_closeness_and_betweenness_centrality_measures_for_each_node._A_more_{comprehensive}_index ofcentralitywasthenderivedfromthesecorrelatedmeasuresthroughprincipalcomponentanalysis.However,thisindexwasneithermoreinformative norlessskewedthanthemuchsimplerdegreenumber.Wethereforedecidedtousethelatterinouranalysis.

(6)

Table3

Frequencydistribution(%)ofrefereerecommendations(a)andeditorialdecisions(b)byjournal.

(a) Journal Refereerecommendation J1 J2 J3 J4 Accept 4.4 2.4 6.1 10.4 Minorrevisions 23.1 18.4 26.8 29.9 Majorrevisions 39.8 32.8 30.1 32.0 Reject 32.7 46.4 37.0 27.8 (b) Journal Editorialdecision J1 J2 J3 J4 Accept 1.2 0.1 1.2 3.6 Minorrevisions 10.4 7.2 18.4 27.7 Majorrevisions 38.5 26.7 30.2 37.3 Reject 49.9 66.1 50.1 31.4 Table4

Averagereviewscores(notconsideringuncertaincases),refereedisagreement,refereereportlengthandreviewtimebyeditorialdecision.

Editorialdecision Reviewscore Disagreement Reportlength(characters) Reviewtime(days)

Accept 0.89 0.19 1729 34

Minorrevisions 0.71 0.24 2602 36

Majorrevisions 0.39 0.32 3636 45

Reject 0.09 0.21 3358 36

Table5

Correlations(Spearman’s)betweentheauthordegreeandthenumberofrefereesanddegreeofreferees,respectively,with95%bootstrapconﬁdence intervals.

Numberofreferees Degreeofreferees

Journal1 0.13[0.06,0.19] 0.20[0.13,0.27]

Journal2 0.27[0.22,0.33] 0.11[0.04,0.17]

Journal3 0.06[−0.01,0.12] 0.02[−0.04,0.08]

Journal4 0.10[−0.03,0.23] 0.12[−0.01,0.24]

notdependonalargernumberofrefereesperpaper.Theaveragenumberofrefereeperpaper(1.83)wasactuallysmaller

thanthenumberofauthors(2.33).Thehigherdensitywasinsteadduetothelargershareofrefereewhoreviewedmultiple

papers,whichishigherthantheoneoftheauthorswhosubmittedtwoormoreoftheirworksinthedatasetjournals.

3.2. Refereeandeditorialbias

Weusednetworkmeasuresandothervariablespresentedabovetoanalysethepotentialsourcesofbiasinthereview

process.Theﬁrststepwastolookattheeditor’schoiceofreferees.Atleasttwodecisionswerepotentiallybiasedhere.

First,dependingonthecharacteristicsoftheauthor,theeditorcouldinvitedifferentreferees(Ganguly&Mukherjee,2017),

who,inturn,couldproducesystematicallydifferentrecommendations.Secondly,alsodependingonauthors’characteristics, theeditorcouldinviteadifferentnumberofreferees.Therefore,thesedecisionscouldrevealcertainbiasesoftheeditorial process,whichcouldtreatcertainauthorsdifferentlyduetotheircharacteristics,e.g.,reputation.

Tochecktheﬁrstofthispoints,weestimatedthecorrelationbetweencharacteristicsofauthorsandrefereesofeach paperderivedfromtheafﬁliationnetworksshowedinFigs.1band2b.Giventhattheestimatedcentralitymeasureswere highlycorrelated,withthedegreeofauthorsandrefereesshowingthehighestvariance,wefocusedonthismeasurewhen discussingauthors’andreferees’properties.Table5showsthatthedegreeofauthorsandrefereeswaspositively(though weakly)correlatedinJ1,J2andpossiblyJ4(seealsoSection3.3).

Giventhatthedegreecouldreﬂectjobexperienceandseniority,thiswouldindicatethateditorspreferentiallyselected experiencedrefereesforpaperwrittenbyexperiencedauthors.Otherwise,thiscouldindicatethatexperiencedreferees acceptedtoreviewpreferablypapersfrompresumably“important”authors.Wedidﬁndasimilarpatternwithcorrelations betweenthedegreeofauthorsandthenumberofreferees(Table5),wheresmallcorrelationscanbefoundinallbutthe thirdjournal.Althoughweak,thefactthatacertaindegreeofcorrelationispresentissurprisinggiventhat,atthisstage,we didnotincludeintheanalysisanyothersourceofinformationabletoreducethevarianceofthedata.

Afterlookingattheeditors,thesecondstepwastofocusontheworkofreferees.Giventhatrefereeswerenot ran-domlyassignedtosubmissions,apotentialsourceofbiasherecouldbeduetorefereeexperience(measuredasdegreein thereferee–papernetwork).Therefore,wetriedtotestwhetherthereferees’positioninthenetworkcouldpredicttheir recommendations.Weestimatedanordinaryleastsquares(OLS)modelusingthemeandegreeoftherefereesforagiven paper,togetherwiththenumberofrefereesandthejournalIDascontrolvariablesandcalculatedifthispredictedthe

(7)

cor-Table6

OLSestimatesforthereviewscoremodel.

Coefﬁcient 95%CI Std.coefﬁcienta (Intercept) 0.047 [−0.002,0.097] – Refereedegree −0.017 [−0.030,−0.005] −0.063 Numberofreferees 0.078 [0.063,0.093] 0.218 Journal2 −0.018 [−0.049,0.013] – Journal3 0.102 [0.073,0.131] – Journal4 0.129 [0.090,0.168] – R2 _0.084 F(5,2907) 53.460

a_When_and_how_{coefﬁcients}_of_categorical_variables_should_be_standardized_represents_a_{controversial}_issue_(e.g.,_Gelman,_2008)._We_hence_decided_to proceedbyonlystandardisingcontinuousvariables.Asaconsequence,thetabledoesnotshowstandardizedcoefﬁcientsforthe“Journal”variable. Table7

Logisticmodelestimatesontheprobabilityofrejection.

Coefﬁcient 95%CI Std.coefﬁcienta (Intercept) 3.468 [2.770,4.185] – Reviewscore −11.840 [−12.725,−10.999] −3.243 Numberofreferees −0.350 [−0.573,−0.129] −0.268 Authordegree −0.302 [−0.522,−0.093] −0.161 Refereedegree −0.037 [−0.251,0.179] −0.116 Disagreement 1.255 [0.716,1.799] 0.287

Authordegree×disagreement 0.567 [0.089,1.074] 0.130

Refereedegree×disagreement −0.320 [−0.821,0.178] −0.073

Journal2 0.327 [−0.072,0.728] –

Journal3 −0.029 [−0.416,0.359] –

Journal4 −0.729 [−1.235,−0.219] –

Pseudo-R2_(McFadden) _0.553

a_When_and_how_{coefﬁcients}_of_categorical_variables_should_be_standardized_represents_a_{controversial}_issue_(e.g.,_Gelman,_2008)._We_hence_decided_to proceedbyonlystandardisingcontinuousvariables.Asaconsequence,thetabledoesnotshowstandardizedcoefﬁcientsforthe“Journal”variable.

respondingreviewscore.Table6showsthatrefereeswithahigherdegreetendedpreferablytoassignlowerreviewscores. However,thecorrespondingeffectisrathersmall:resultsfromamodelincludingonlythereferees’degreeaspredictor (i.e.,excludingthecontrols)indicatedthatthisvariablealoneexplainedabout2%ofthereviewscorevariance.Inaddition, giventhatingeneralrefereeshadassigneddifferentpapersbyeditors,wecannotexcludeatthisstagethatthedifference inrecommendationswasduetoadifferenceinqualityofthesubmissionsinsteadofrevealingasevereattitudeofmore experiencedreferees(onthis,moredetailinSection3.3).Alsonoticethatsubmissionswithalargernumberofrefereestend toobtainhigherscores.

Thelaststepwastolookattheeditorialdecision.Ideally,theonlyelementinfluencingtheeditor’sdecisionshouldbethe reviewscore.Wetriedtopredicttheeditorialdecisionthroughamodelincludingthereviewscoreasafixedeffect.Tolook atpotentialsourcesofbias,incaseofauthorandrefereedegrees,wetestedthenumberofrefereesandtheirdisagreement. Wealsoaddedinteractiontermsbetweenreferees’disagreementandrefereeandauthordegrees.Thiswastounderstandif editorscouldbeinfluencedmorelikelybyextra-elements,suchasthedegree(i.e.,experience)ofauthorsorreferees,when dealingwithcontradictoryreviews.Finally,giventhatthedistributionofeditorialdecisionswasdifferentacrossourfour journals,weaddeddummiesforeachjournalbutthefirstaspredictor.Beingtheeditorialdecisionanordinalvariable,using OLSregressionwasnolongerappropriate.Weestimatedourmodelusingtwodifferentstrategies:(i)logisticregression, and(ii)orderedlogisticregressionwithacumulativelinkfunction.

AsTable3showedthataroundhalfofthepaperswererejectedinthefirstreviewround,toefficientlyuseourdatawe splittheeditorialdecisionvariableatthislevel.Alogisticmodelwasthenusedtopredictwhetherapaperwasrejectedvs. invitedforrevisionsoraccepted.Asexpected,thereviewscoreresultedthestrongestpredictorfortheeditorialdecision (Table7).Furthermore,consistentlywithTable3,journaldummiesshowedsignificantdifferencesintherejectionrate. Moreinterestingly,thefactthateditorialdecisionswereinfluencedbytheauthors’degree–bothaspureeffectandin interactionwithdisagreement–suggeststhateditorsinterpretedrefereerecommendationsdifferentlydependingonthe author’sreputationinthescientificcommunity.Inaddition,havingmoreandmoreexperiencedrefereesdecreasedthe probabilityofbeingrejectedindependentlyoftheotherfactors.

Tochecktherobustnessofourresultsandfullyexploitourdataonallthepossiblelevelsoftheeditorialdecision, wetestedaorderedlogisticmodelwithacumulativelinkfunctiontopredictwhetherapaperwasaccepted,invitedfor resubmissionwithminorrevision,withmajorrevision,orrejected.Table8showsthatmostestimates,includingtheeffect oftheauthor’sdegree,werequalitativelysimilartowhatweobtainedfromthesimplerlogisticmodelabove.However, theeffectofsomefactorsslightlyvaried.Mostnotably,(i)theeffectofthenumberofrefereestronglydecreased,witha correspondingconﬁdenceinterval(CI)thatincludedzerointhenewmodel;(ii)thepositiveeffectoftheinteractionbetween

(8)

Table8

Cumulative-linkorderedlogisticestimationspredictingpaperacceptance(referencecategory),minorrevision,majorrevisionorrejection.

Coefﬁcient 95%CI Std.coefﬁcienta Reviewscore −11.388 [−12.021,−10.779] −3.119 Numberofreferees −0.169 [−0.349,0.010] −0.129 Authordegree −0.204 [−0.363,−0.045] −0.113 Refereedegree 0.006 [−0.176,0.189] −0.109 Disagreement 1.667 [1.219,2.121] 0.382

Authordegree×disagreement 0.368 [−0.028,0.776] 0.085

Refereedegree×disagreement −0.463 [−0.892,−0.036] −0.106

Journal2 0.321 [−0.007,0.652] –

Journal3 −0.099 [−0.413,0.215] –

Journal4 −0.780 [−1.160,−0.399] –

(Accepted|Minorrevisions) −12.163 [−12.989,−11.336] –

(Minorrevisions|Majorrevisions) −7.413 [−8.077,−6.748] –

(Majorrevisions|Reject) −2.834 [−3.399,−2.270] –

Pseudo-R2_(McFadden) _0.509

a_When_and_how_{coefﬁcients}_of_categorical_variables_should_be_standardized_represents_a_{controversial}_issue_(e.g.,_Gelman,_2008)._We_hence_decided_to proceedbyonlystandardisingcontinuousvariables.Asaconsequence,thetabledoesnotshowstandardizedcoefﬁcientsforthe“Journal”variable.

theauthordegreeanddisagreementwasalsoreducedclosetozero;(iii)thenegativeeffectoftheinteractionbetweenthe refereedegreeanddisagreementincreased(inabsoluteterms)andthecorrespondingCInolongerincludedzero.Sinceonly 14%ofpapersinthedatasetreceivedanacceptorminorrevision,wedidnotconsiderotherlogisticmodelstofurthertestif theestimatesoftheexplanatoryvariablesareconsistent(proportional)acrossdifferentthresholdsfortheeditorialdecision. 3.3. Acomprehensiveanalysisofthereviewprocess

Althoughinteresting,themodelsabovecouldnotprovideacomprehensivepictureofthereviewprocessyet.Furthermore, theauthors’degree(whichpossiblyreflectedexperienceandseniority)couldsystematicallyaffectthequalityofapaper.In otherwords,evenifweobservedasignificantrelationbetweenthisvariableandthereviewscore,thiscouldsimplyreflect thefactthatauthorswithahigherdegreeweremorelikelytosubmitpapersofhigherquality.Togetamorecomprehensive pictureofthewholeprocessandbetterdistinguishbetweenbiasedandunbiasedpathsleadingtotheeditorialdecision,we trainedaBayesiannetwork(Friedman,Geiger,&Goldszmidt,1997)toestimatetheprobabilityofapaperrejectiononthe basisofallthevariablespreviouslyconsidered.3

Bayesiannetworksmodelinterdependenciesamongasetofvariablesasconnectionsinadirectedacyclicgraph,where nodesrepresentrandomvariables–e.g.,thereviewscore–andedgesrepresentconditionaldependencies–e.g.,howmuch thereviewscorehadaneffectontherejectionprobabilityofapaper.Weoptedforthismethodfortworeasons.First,the structureofaBayesiannetworkislearnedinductivelyfromthedata,withnoneedfortheresearcherstoprovideprior assumptionsontherelevantcausaleffects.Secondly,oncelearnedthenetworkcouldbeusedtoderiveprobabilitiesofthe eventofinterestgivenasetofconditionsontheothervariables–e.g.,howlikelyistherejectionofapapergiventhatits authorhasanetworkdegreehigherthanacertainvalue.

Fig.3showsthestructureoftheBayesiannetworkinductivelylearnedfromthedatathroughmaximumlikelihood estimation.4_The_algorithm_is_fully_data-driven_and_used_the_training_set_(80%_of_the_data)_to_learn_both_the_network_structure andthedirectionofeachedge.Non-significantpathswereautomaticallyexcluded.Theonlyexternalconstraintonthe networkstructurewasthattherecouldbenolinkfromrejecttoanyoftheothernodes.Notethat,toincreasethereadability ofthefigure,thejournalvariable,whichsignificantlyaffectedallothernodes,wasnotincluded.

AllparametersoftheBayesiannetworkwerelearnedonatrainingsetconsistingofarandomsampleof80%ofourdata, whiletheremaining20%wereusedformodelvalidation.Theresultingnetworksuccessfullypredicted84%ofthevalidation setcaseswhengivenalltheinformationbuttheeditorialdecision.Table9showsthestandardizedcoefficientsforthe networkpaths.Pathcoefficientsexpresstheeffectthateachupstreamnodehastothedownstreamnodesitisconnectedto. Thesecoefficientswerelearnedbythealgorithmwhileignoringtheinformationonthejournal,whichimpliesthattheycan bebeinterpretedasaggregatedacrossthefouroutlets.Notethatweomittedthecoefficientslinkingthejournalvariables totheothernodesbecause,beingjournalacategoricalvariable,theanalysisledtoestimateonedifferentcoefficientper journal,withnostraightforwardwaytosummarizetheminasinglevalue.

Tointerprettheresultingnetwork,itisimportanttofollowallthedifferentpathsgoingfromtheauthors’degreetothe editorialdecision.Theonegoingthroughthereviewscore(ingreeninFig.3)doesnotnecessarilyrepresentbiases,asit

3 _For_technical_reasons,_it_is_not_possible_to_have_a_ﬁnal_node_that_has_more_then_two_levels,_when_parent_nodes_are_continuous_variables._However,_the logisticandordinalregressionmodelsintheprevioussectionimplythatresultswerenotdramaticallydifferentwhentheeditorialdecisionisincludedas abinaryvariableinsteadofconsideringallfourlevels.

(9)

Fig.3. StructureoftheBayesiannetworkresultingfromatrainingsetbasedon80%ofourdata.Red(dashed)arrowsmarkthe“biased”pathslinking authordegreetorejectionwithoutgoingthroughthereviewscore.Plusandminussymbolsindicatethedirectionofeffect.Notethatthecategoricaljournal variable,whichsignificantlyaffectsallothernodes,isnotincludedinthefigureforbetterreadability.(Forinterpretationofthereferencestocolourinthis figurelegend,thereaderisreferredtothewebversionofthearticle.)

Table9

StandardizedpathcoefﬁcientsforthenetworkinFig.3.Beingjournalcategorical,nocoefﬁcientsareshownforpathsinvolvingthisvariable.

Path Standardizedcoefﬁcient

Authordegree→rejected −0.041

Authordegree→refereedegree 0.066

Refereedegree→reviewscore −0.129

Reviewscore→rejected −0.724

Refereedegree→numberreferees 0.144

Disagreement→numberreferees 0.413

Disagreement→reviewscore 0.263

Numberreferees→rejected −0.047

couldsimplyreﬂectthefactthatmoreexperiencedauthorsaremorelikelytoproducehigh-qualitypapers.Ontheother

hand,anypathlinkingtheauthors’degreewiththeeditorialdecisionandnotgoingthroughthereviewscore(redinFig.3)

canbeconsideredasapotentialsourceofbiassinceitexcludesthemostcentral–andinprincipleonly–elementthat shouldcountinthereviewprocess.

Themostnoticeableinsightfromthelearnednetworkwasthedirectpathfromtheauthors’degreetotheeditorialdecision (the“reject”node)(Fig.3),whichcanbeinterpretedasadirectbiasoneditorialdecisionsduetotheexperience/reputation ofauthors.However,thestrengthofthispathisonlyabout6%ofthestrengthofthepathlinkingthereviewscoretothe rejectvariable.Asecondindirectpathfromtheauthors’degreetotheeditorialdecisionnodewentthroughthedegreeand numberofreferees.Thissuggestedthepresenceafurtherbiasduetotheselectionofdifferent(innumbersandquality) refereesfordifferentauthorswhosubmittedtothesamejournal.Finally,wefoundthatthejournalnode(notshownin theﬁgure)affectedallothervariables,whichwasconsistentwiththeintuitionthatdifferentjournalshad,amongothers, differenteditorsandrejectionrates.

Usingthetrainednetwork,weestimatedhowtheprobabilityofbeingacceptedorinvitedforrevisions(i.e.,nottobe rejected)varieddependingonthereviewscoreandtheauthordegree.Fig.4showsthatitmonotonicallyincreasedwith thereviewscore.Furthermore,authorsbelongingtothetopquartilewithregardtothenetworkdegreesystematicallyhad a5–10%higherchanceofbeingacceptedwhencomparedwiththeonesinthebottomquartile.Itisworthnotingthatthis probabilitywasestimatedaftercontrollingforreviewscores,whichsuggestedthattheseeditorialdecisionswerenotdue tothequalityofsubmissions–resultinginmorepositivereviews–buttothereputationofauthors.Thisbiaswasmore prominentwhenreviewscoreswererelativelylow,whichsuggestedthatmoreimportantauthorsmoreeasilyescapeda presumablydeservedrejectionwhentheydidnotsubmitbrilliantpapers.

4. Discussionandconclusions

Inourwork,wetriedtousenetworkdatacreativelytoestimateeditorialbiasinfourcomputersciencejournals,without recurringtoin-depthinformationonmanuscripts,authorsorreviewersinvolved.Itisworthnotingthatcross-journal, large-scaledataontheeditorialprocessofscholarlyjournalsthatincludefullinformationonmanuscripts,authorsandreview scoresareunfortunatelyonlyrarelyavailable(Squazzonietal.,2017).Needforpreservingauthoranonymity,unavailability ofex-ante,objectivemeasuresofasubmission’squality,lackofdatatoreconstructnetworkeffectsduetotheprestigeof scientists,areallfactorsthatimpedetoquantitativelyassesstherigourandimpartialityoftheeditorialprocessinscholarly journals(Batageljetal.,2017;Casnicietal.,2017).Whiledatainprincipleexist,thankstothefulldigitalisationoftheeditorial

(10)

Fig.4. Relationbetweenthereviewscoreofapaperandtheprobabilityofbeingacceptedorinvitedforresubmission.Thetwolinesshowpapersfrom authorsbelongingtothetopandbottomquartilewithregardtotheauthors’degree.

processinalmostalljournals,allquantitativeindicatorswhich haverecentlyincreasedourunderstandingofresearch, citationandpublicationpatternsanddynamicsarerarelyappliedtomeasurepeerreviewandeditorialprocessesonalarge scale(Mutz,Bornmann,&Daniel,2017).Inthisrespect,ourstudyhastheadvantageofmeasuringcertaincharacteristicsof theeditorialprocessacrosscomparablejournalswithouttheneedforin-depth,individualdata,whichareoftenunavailable. Themainlimitationofthestudyresidesinitsrelianceonasmallsampleofjournalsfromasinglediscipline.Whetherits conclusionscouldbegeneralizedtoothercasesandscientiﬁctraditionshenceremainsanopenquestion.Futureresearch shouldtrytoapplyandextendtheproposedmethodologytolargerandmorevariedsamples,takingalsointoaccountthat severalvariablesmayinteracttoproduce(orprevent)thereputationbiashighlightedbythedataanalysedhere.

Thissaid,ourfindingsclearlyshowedthateditorscouldbebiasedtowardsmoreprestigiousauthorsbygivingthem higherchancesofpublicationevenwhentheypresumablydidnotdeserveaspecialtreatment.Althoughinourcasethe editorialbiaswasnotsolargetodrasticallyaltertheoutcomeofthereviewprocess,itwasstillsignificant,especiallyforlow reviewscores.Theresulting“implicit”premiumtomoreprestigiousauthorscanbeduetoseveralreasons.First,editorscould rationallypredicthighercitationsofarticlesfromprestigiousauthors,whoprobablyareevenmoreprolific.Assuggestedby Petersenetal.(2014),whiletheeffectoftheprestigeofauthorsonarticlecitationsmayevaporateinthelongrun,thiseffect iseffectiveespeciallyindrivingcitationcountearlyinthetypicalcitationlifecycle.Giventhatajournal’simpactfactoris calculatedinatwoyearswindow,andthatattentiondecaybythecommunityseemsmorepronouncednowadays(Parolo etal.,2015),editorscouldbemoreindulgentwithmoreprestigiousauthorsasameanstomaximisetheirjournal’simpact factor.

Secondly,prestigiousauthorscouldreactmorestrategicallyagainstamanuscriptrejectionbyrefusingtoreviewor avoidingtosubmitfurtherwork,sopunishingeditorsandtheirjournalsafterwards(Squazzoni&Gandelli,2013).Onthe otherhand,giventhecomplexityofanyeditorialdecisionandperhapstheunavailabilityoffullyinformativereviews,editors couldintentionallytrustmoretheconﬁdenceofprestigiousauthorsthantheopinionofimprecisereviewersatleasttoavoid falsenegatives(Leeetal.,2013).Thiswouldsuggestthatalthoughtheeditorialmanagementofasubmissionshouldbein principlean“one-shotgame”,thepastperformanceofauthorsorthepredictiveimpactoftheirresearchcouldimplicitly inﬂuencetheprocess.

However,thereisalsoanotherinterpretationofourfindings,whichdoesnotconveyanynegativemessageontheeditorial processofthesefourjournals,whoseestimatedlevelofbiaswasinanycaseonlyminimal.Ontheonehand,whilereviewers andeditorsareincreasingunderthespotlightforavarietyofexplicitorevenimplicitbias(Leeetal.,2013;Pinholster,2016), itmustbesaidthattheyoperateinasituationofhyper-competition,timetrade-offsandpressuresduetotheincreasing numberofsubmissions,journalsandpublications(Edwards&Roy,2017;Kovanis,Porcher,Ravaud,&Trinquart,2016).On theotherhand,thiscriticismimplicitlysuggeststhateditorialdecisionshaveanoptimalreferencepoint,withreviewersand editorsbeingeventuallyresponsiblefordeviationsandmistakes.Theseperspectivestendtodenythefactthatpeerreview andeditorialprocessesareintrinsicallycomplexandimperfectbecausetheycannotbutreflectwhatscienceitselfessentially is(Cowley,2015).Whileitisimportanttoexploremeasurestoassesstheeditorialprocessandimproveitstransparencyand efficacy,theideathatanoptimalworldwouldexist,ifonlyscientistswerenotexposedtoexternalinfluencesanddistortive incentivesorhadnotmaliciousintentions,isnaive.

(11)

Authorcontributions

GiangiacomoBravo:Conceivedanddesignedtheanalysis;Performedtheanalysis;Wrotethepaper. MikeFarjam:Contributeddataoranalysistools;Performedtheanalysis;Wrotethepaper.

FanciscoGrimaldoMoreno:Collectedthedata;Contributeddataoranalysistools;Wrotethepaper. AliaksandrBirukou:Contributeddataoranalysistools.

FlaminioSquazzoni:Conceivedanddesignedtheanalysis;Wrotethepaper;Othercontribution. Acknowledgements

TheauthorsgratefullyacknowledgeAnnetteHinze,AlfredHofmann,KatarinaKreissigandTamaraWelschotfortheirhelp withthedataextractionandtheanonymisationprocedure,andRalfGerstnerforbothhelpingwiththedataextractionand hisusefulcommentsonanearlierversionofthepaper.ApreliminaryversionofthisarticlewaspresentedataWG2PEERE meetingatUvAinAmsterdamonFebruary2017andbeneﬁtedfromcommentsbyAnaMaruˇsi ´c,BaharMehmani,Virginia DignumandMichaelWillis.ThisworkwaspartiallysupportedbytheCOSTActionTD1306“Newfrontiersofpeerreview” (www.peere.org)andbytheSpanishMinistryofScienceandInnovationProjectTIN2015-66972-C5-5-R.Lastbutnotleast, theauthorsgratefullyacknowledgethecommentsmadebytwoanonymousreferees,whichnicelyhelpedtoimprovethe qualityofthepaper.

References

Batagelj,V.,Ferligoj,A.,&Squazzoni,F.(2017).Theemergenceofaﬁeld:Anetworkanalysisofresearchonpeerreview.Scientometrics,113(1),503–532. http://dx.doi.org/10.1007/s11192-017-2522-8

Birukou,A.,Wakeling,J.R.,Bartolini,C.,Casati,F.,Marchese,M.,Mirylenka,K.,Osman,N.,Ragone,A.,Sierra,C.,&Wassef,A.(2011).Alternativestopeer review:Novelapproachesforresearchevaluation.FrontiersinComputationalNeuroscience,5http://dx.doi.org/10.3389/fncom.2011.00056 Bornmann,L.(2011).Scientiﬁcpeerreview.AnnualReviewofInformationScienceandTechnology,45(1),197–245.

http://dx.doi.org/10.1002/aris.2011.1440450112

Bornmann,L.,&Daniel,H.-D.(2009).ExtentoftypeIandtypeIIerrorsineditorialdecisions:AcasestudyonAngewandteChemieInternationalEdition. JournalofInformetrics,3(4),348–352,pii:S1751157709000406.

Bornmann,L.,&Williams,R.(2017).Canthejournalimpactfactorbeusedasacriterionfortheselectionofjuniorresearchers?Alarge-scaleempirical studybasedonResearcherIDdata.JournalofInformetrics,11(3),788–799,pii:S1751157717300378.

Casnici,N.,Grimaldo,F.,Gilbert,N.,Dondio,P.,&Squazzoni,F.(2017).Assessingpeerreviewbygaugingthefateofrejectedmanuscripts:Thecaseofthe journalofartiﬁcialsocietiesandsocialsimulation.Scientometrics,http://dx.doi.org/10.1007/s11192-017-2241-1

Casnici,N.,Grimaldo,F.,Gilbert,N.,&Squazzoni,F.(2016).Attitudesofrefereesinamultidisciplinaryjournal:Anempiricalanalysis.Journalofthe AssociationforInformationScienceandTechnology,68(7),1763–1771.http://dx.doi.org/10.1002/asi.23665

Cowley,S.J.(2015).Howpeer-reviewconstrainscognition:Onthefrontlineintheknowledgesector.FrontiersinPsychology,6,1706. http://dx.doi.org/10.3389/fpsyg.2015.01706

Edwards,M.A.,&Roy,S.(2017).Academicresearchinthe21stcentury:Maintainingscientiﬁcintegrityinaclimateofperverseincentivesand hypercompetition.EnvironmentalEngineeringScience,34(1),51–61.http://dx.doi.org/10.1089/ees.2016.0223

Faust,K.(1997).Centralityinafﬁliationnetworks.SocialNetworks,19(2),157–191.http://dx.doi.org/10.1016/s0378-8733(96)00300-0 Friedman,N.,Geiger,D.,&Goldszmidt,M.(1997).Bayesiannetworkclassiﬁers.MachineLearning,29(2–3),131–163.

Fyfe,A.(2015).Journals,learnedsocietiesandmoney:Philosophicaltransactions,ca.1750–1900.NotesandRecordsoftheRoyalSociety,69(3),277–299. http://rsnr.royalsocietypublishing.org/content/69/3/277

García,J.A.,Rodriguez-Sánchez,R.,&Fdez-Valdivia,J.(2015).Theauthor–editorgame.Scientometrics,104(1),361–380. http://dx.doi.org/10.1007/s11192-015-1566-x

Gelman,A.(2008).Scalingregressioninputsbydividingbytwostandarddeviations.StatisticsinMedicine,27(15),2865–2873.

Grimaldo,F.,&Paolucci,M.(2013).Asimulationofdisagreementforcontrolofrationalcheatinginpeerreview.AdvancesinComplexSystems,16(07), 1350004.http://dx.doi.org/10.1142/S0219525913500045

Hsiehchen,D.,&Espinoza,M.(2016).Detectingeditorialbiasinmedicalpublishing.Scientometrics,106(1),453–456. http://dx.doi.org/10.1007/s11192-015-1753-9

Kovanis,M.,Porcher,R.,Ravaud,P.,&Trinquart,L.(2016).Theglobalburdenofjournalpeerreviewinthebiomedicalliterature:Strongimbalanceinthe collectiveenterprise.PLoSONE,11(11),1–14.http://dx.doi.org/10.1371/journal.pone.0166387

Lee,C.J.,Sugimoto,C.R.,Zhang,G.,&Cronin,B.(2013).Biasinpeerreview.JournaloftheAmericanSocietyforInformationScienceandTechnology,64(1), 2–17.http://dx.doi.org/10.1002/asi.22784

Lin,Z.,Hou,S.,&Wu,J.(2016).Thecorrelationbetweeneditorialdelayandtheratioofhighlycitedpapersinnature,scienceandphysicalreviewletters. Scientometrics,107(3),1457–1464.http://dx.doi.org/10.1007/s11192-016-1936-z

Moustafa,K.(2015).Istherebiasineditorialchoice?Yes.Scientometrics,105(3),2249–2251.http://dx.doi.org/10.1007/s11192-015-1617-3

Mutz,R.,Bornmann,L.,&Daniel,H.-D.(2017).Arethereanyfrontiersofresearchperformance?Efﬁciencymeasurementoffundedresearchprojectswith theBayesianstochasticfrontieranalysisforcountdata.JournalofInformetrics,11(3),613–628,pii:S1751157716303741.

Parolo,P.D.B.,Pan,R.K.,Ghosh,R.,Huberman,B.A.,Kaski,K.,&Fortunato,S.(2015).Attentiondecayinscience.JournalofInformetrics,9(4),734–745, pii:S1751157715200442.

Petersen,A.M.,Fortunato,S.,Pan,R.K.,Kaski,K.,Penner,O.,Rungi,A.,Riccaboni,M.,Stanley,H.E.,&Pammolli,F.(2014).Reputationandimpactin academiccareers.ProceedingsoftheNationalAcademyofSciences,111(43),15316–15321.http://dx.doi.org/10.1073/pnas.1323111111

Petersen,J.,Hattke,F.,&Vogel,R.(2017).Editorialgovernanceandjournalimpact:Astudyofmanagementandbusinessjournals.Scientometrics,112(3), 1593–1614.http://dx.doi.org/10.1007/s11192-017-2434-7

Pinholster,G.(2016).Journalsandfundersconfrontimplicitbiasinpeerreview.Science,352(6289),1067–1068.http://mfkp.org/INRMM/article/14042738 Resnik,D.B.,&Elmore,S.A.(2016).Ensuringthequality,fairness,andintegrityofjournalpeerreview:Apossibleroleofeditors.ScienceandEngineering

Ethics,22(1),169–188.http://dx.doi.org/10.1007/s11948-015-9625-5

Righi,S.,&Takács,K.(2017).Themiracleofpeerreviewanddevelopmentinscience:Anagent-basedmodel.Scientometrics,113(1),587–607. http://dx.doi.org/10.1007/s11192-017-2244-y

Sarigöl,E.,Garcia,D.,Scholtes,I.,&Schweitzer,F.(2017).Quantifyingtheeffectofeditor–authorrelationsonmanuscripthandlingtimes.Scientometrics, http://dx.doi.org/10.1007/s11192-017-2309-y

(12)

Ganguly,N.,&Mukherjee,A.,2017,Inﬂuenceofreviewerinteractionnetworkonlong-termcitations:Acasestudyofthescientiﬁcpeer-reviewsystemof theJournalofHighEnergyPhysics.http://adsabs.harvard.edu/abs/2017arXiv170501089S.

Siler,K.,Lee,K.,&Beroc,L.(2015).Measuringtheeffectivenessofscientiﬁcgatekeeping.ProceedingsoftheNationalAcademyofSciences,112(2),360–365. Squazzoni,F.,&Gandelli,C.(2012).SaintMatthewstrikesagain:Anagent-basedmodelofpeerreviewandthescientiﬁccommunitystructure.Journalof

Informetrics,6(2),265–275,pii:S1751157711001179

Squazzoni,F.,&Gandelli,C.(2013).Openingtheblack-boxofpeerreview:Anagent-basedmodelofscientistbehaviour.JournalofArtiﬁcialSocietiesand SocialSimulation,16(2),3.http://jasss.soc.surrey.ac.uk/16/2/3.html

Squazzoni,F.,Grimaldo,F.,&Marusic,A.(2017).Publishing:Journalscouldsharepeer-reviewdata.Nature,546,352.http://dx.doi.org/10.1038/546352a Teele,D.L.,&Thelen,K.(2017).Genderinthejournals:Publicationpatternsinpoliticalscience.PS:PoliticalScience&Politics,50(2),433–447.

GiangiacomoBravoisprofessorattheDepartmentofSocialStudiesandcoordinatesthecomputationalsocialsciencegroupattheCentreforData IntensiveSciencesandApplications,LinnaeusUniversity,Sweden.Hismainresearchinterestscovercollectiveactionproblemsandenvironmental sustainability.

MikeFarjamisapost-docattheDepartmentofSocialStudiesatLinnaeusUniversity,Sweden.Hismaininterestistheperceptionofriskinvolved wheninteractingwithothers/machinesandcooperationingeneral.

FranciscoGrimaldoMorenoreceivedhisPh.D.incomputersciencefromtheUniversityofValenciain2008.HeisassociateprofessorattheUniversity ofValenciaandresearchfellowattheItalianNationalResearchCouncil.Hisresearchinterestsareagent-basedmodellingandsimulation,machine learninganddataanalysisandvisualization.

AliaksandrBirukoureceivedhisPh.D.incomputersciencefromtheUniversityofTrentoin2009.HeisnowExecutiveEditorforComputerScience atSpringerNature.

FlaminioSquazzoniisassociateprofessorofeconomicsociologyattheDepartmentofEconomicsandManagement,UniversityofBrescia,Italy,where heleadstheGECS-ResearchGrouponExperimentalandComputationalSociology.