Selecting key quality indicators in public transport systems by a robust method

6(/(&7,1*.(<48$/,7<,1',&$7256,138%/,&

75$1632576<67(0686,1*$52%8670(7+2'

5HFHQW LQWHUHVWV LQ WUDQVLW VHUYLFHV KDYH FDSWXUHG DWWHQWLRQ RI H[SHUWV RQ WKH PRQLWRULQJ RI SXEOLF WUDQVSRUWTXDOLW\3UHYLRXVUHVHDUFKKDVIRFXVHGRQWKHGHYHORSPHQWRIUHOHYDQWPRGHOVDQGPHWKRGV IRUPRQLWRULQJWUDQVLWVHUYLFHTXDOLW\DQGLGHQWLILHGZKHUHDQGZKHQGLIIHUHQWOHYHOVRIVHUYLFHTXDOLW\ RFFXU +RZHYHU OLWWOH DWWHQWLRQ KDV EHHQ JLYHQ WR REMHFWLYHO\ GHWHUPLQLQJ D SRRO RI NH\ TXDOLW\ LQGLFDWRUV.4,VIRUPRQLWRULQJSXUSRVHV,QWKLVSDSHUZHDGGUHVVWKLVJDSE\SURSRVLQJDUREXVW PHWKRGRORJ\WKDWLGHQWLILHVDOHQJWK\OLVWRISRWHQWLDO.4,VDQGGHILQHVWKHLUSURSHUWLHVWKHQHQOLVWVWKH MXGJHPHQW RI ERWK UHVHDUFKHUV DQG SUDFWLWLRQHUV UHJDUGLQJ HDFK .4, HYDOXDWHV WKH LQLWLDO OLVW DQG UHFRPPHQGVWKHPRVWSURPLVLQJ.4,VHW:HGHPRQVWUDWHWKHHIIHFWLYHQHVVRIWKLVPHWKRGRORJ\XVLQJ DQLQWHUQDWLRQDOVXUYH\DQG0RQWH&DUORVLPXODWLRQPHWKRGV7KHRXWFRPHVZLOOSURYHXVHIXOLQKHOSLQJ SUDFWLWLRQHUVWRPRQLWRUWUDQVLWVHUYLFHTXDOLW\DFFRUGLQJWRDVHWRIUHFRJQLVHG.4,V

Keywords: key quality indicators, transit service quality monitoring, quality robust methodology.

,1752'8&7,21

7KHWHUP³TXDOLW\RIH[SHULHQFH´4R(UHIHUVWRWKHGHJUHHRIVDWLVIDFWLRQRUGLVDSSRLQWPHQW DXVHUH[SHULHQFHVZKHQXVLQJDQDSSOLFDWLRQRUVHUYLFH4R(UHVXOWVIURPWKHIXOILOPHQWRI D XVHU¶V H[SHFWDWLRQV ZLWK UHVSHFW WR WKH XWLOLW\ RI DQGRU HQMR\PHQW GHULYHG IURP WKH DSSOLFDWLRQRUVHUYLFHLQOLJKWRIWKHXVHU¶VSHUVRQDOLW\DQGFXUUHQWVWDWH>@,QWKHWUDQVLW VHUYLFHFRQWH[W4R(LVDEURDGDQGFRPSOH[FRQFHSWWKDWFDQEHGHILQHGDVWKHFDSDELOLW\RI WUDQVLWRSHUDWRUVWRSURYLGHDVHUYLFHEDVHGRQWKHGHVLUHVRIFXUUHQWDQGSRWHQWLDOXVHUV,W FDQEHGHVFULEHGE\VHYHUDOLQGLFDWRUVWKDWUHSUHVHQWYDULRXVIHDWXUHVRIWUDQVLWVHUYLFHV7KH LQWHUHVWLQLQGLFDWRUVLVGXHWRWKHLUDELOLW\WRSURYLGHLQIRUPDWLYHVLJQDOVUHJDUGLQJDUHDVRI H[FHOOHQFH DQG VHUYLFH SULRULW\ LQ WUDQVLW VHUYLFHV 0RUHRYHU LQGLFDWRUV FDQ UHSUHVHQW D VWUDWHJLFDVSHFWRIWKHGHVLUHGTXDOLW\DQGVHUYHDVGDWDLQSXWIRUWKHQH[WVWHSLQWKHVHUYLFH TXDOLW\PRQLWRULQJSURFHVVi.e.,PHDVXUHPHQWDQGPDQDJHPHQW7KHUHIRUHWKHVHOHFWLRQRI NH\TXDOLW\LQGLFDWRUV.4,VLVFUXFLDOLQRUGHUWRHYDOXDWHWKHSHUIRUPDQFHRQTXDOLW\RI WUDQVLWVHUYLFHVDQGSUHVHQWVIRXUNH\FKDOOHQJHV )LUVWWKHUHDUHPDQ\SRWHQWLDOVHUYLFHTXDOLW\LQGLFDWRUVLQWKHWUDQVLWVHUYLFHFRQWH[WVR VHOHFWLQJDFRPSDFWSRRORINH\LQGLFDWRUVFDQEHFKDOOHQJLQJ>@

6HFRQG SUHYLRXV UHVHDUFK RQ WUDQVLW VHUYLFH TXDOLW\ KDV IRFXVHG SULPDULO\ RQ WKH GHYHORSPHQW RI PRGHOV DQG PHWKRGV IRU ³PDQDJLQJ´ LQGLFDWRUV UDWKHU WKDQ RQ WKH GHWHUPLQDWLRQRIZKDWWRPHDVXUH>@0RUHSUHFLVHO\SUHYLRXVUHVHDUFKHUVKDYHUDUHO\XVHG REMHFWLYHPHWKRGVWRGHYHORSLQGLFDWRUVHOHFWLRQPHFKDQLVPV7KDWLVVHWVRIVXLWDEOHTXDOLW\ LQGLFDWRUVXVHGWRSHUIRUPDQDO\VLVKDYHSUHYLRXVO\EHHQVHOHFWHGDVIROORZV x %DVHGRQWKHOLWHUDWXUHDQGRUVSHFLILFNQRZOHGJHRIYDULRXVHFRQRPLFDQGLQGXVWULDO ILHOGV>@–>@ x 6XUYH\VRIRSHUDWRUVXVHUVDQGIRFXVJURXSV>@±>@ x 6SHFLILFRUJDQLVDWLRQDOREMHFWLYHV>@>@ x $GKRFWKHRUHWLFDOPHWKRGRORJLHV>@>@

7KLUG VLQFH LQGLFDWRUV DUH PHUHO\ DEVWUDFWLRQV RI WKH V\VWHP DQG ZLOO QHYHU SHUIHFWO\ UHSUHVHQWLWLWLVFKDOOHQJLQJWRVHOHFWWKHPRVWDSSURSULDWHLQGLFDWRUVWKDWZLOOGHVFULEHWKH V\VWHPPRVWHIIHFWLYHO\ )RXUWKDFDUHIXOLQGLFDWRUVHOHFWLRQSURFHVVFDQLPSURYHWKHLUDFFHSWDELOLW\DQGFUHGLELOLW\ DPRQJH[SHUWV7KLVHQKDQFHGDFFHSWDELOLW\LVUHIHUUHGWRDVPHWKRGUREXVWQHVV ,QWKLVSDSHUZHSURSRVHDUREXVWPHWKRGRORJ\IRULGHQWLI\LQJDQGVHOHFWLQJ.4,VXVLQJ ERWKGDWDFROOHFWHGWKURXJKLQWHUQDWLRQDOVXUYH\VDQG0RQWH&DUORVLPXODWLRQPHWKRGV7KLV PHWKRGRORJ\EXLOGVRQWKHILUVWEORFNRI75$1648$/ZKLFKUHFRPPHQGVWKHXVHRID UREXVW PHWKRGRORJ\ IRU .4,V VHOHFWLRQ >@ 7KLV PHWKRGRORJ\ GLIIHUV IURP (/$67,& ZKLFKLVDIUDPHZRUNIRUWKHLGHQWLILFDWLRQDQGVHOHFWLRQRIVXVWDLQDEOHWUDQVSRUWLQGLFDWRUV GHYHORSHGE\8.H[SHUWVRQO\>@)LQDOO\LWIDFLOLWDWHVDQGDGYDQFHVSULPDU\UHVHDUFKE\ SURYLGLQJDFRPSDFWSRRORI.4,VUDWKHUWKDQWKHILYH.4,VUHSRUWHGLQ>@ 7KHUHPDLQGHURIWKLVSDSHULVRUJDQLVHGDVIROORZV,Q6HFWLRQZHSURSRVHDPHWKRG IRULGHQWLI\LQJDQGVHOHFWLQJWKHPRVWVXLWDEOH.4,VIRUGHVFULELQJWUDQVLWVHUYLFHTXDOLW\,Q 6HFWLRQZHSUHVHQWDQGGLVFXVVWKHDSSOLFDWLRQRIWKLVPHWKRGDQGLWVUHVXOWV)LQDOO\LQ 6HFWLRQZHGUDZRXUFRQFOXVLRQVDQGVXJJHVWIXWXUHUHVHDUFKGLUHFWLRQV 0(7+2'2/2*< ,QWKLVVHFWLRQZHSURSRVHDUREXVWPHWKRGRORJ\IRUWKHLGHQWLILFDWLRQDQGVHOHFWLRQRID SRRORI.4,V7KHPHWKRGRORJ\FRPSULVHVVL[VWHSVDVVKRZQLQWKHVFKHPDWLFRI)LJ $FRQFLVHGHVFULSWLRQRIWKHVHVWHSVLVDVIROORZV ,GHQWLILFDWLRQRIDORQJOLVWRISRWHQWLDOTXDOLW\LQGLFDWRUV )LUVWDOHQJWK\OLVWRI.4,VLVLGHQWLILHG6HYHUDOVRXUFHVPD\EHFRQVLGHUHG,QWKLVPHWKRG UHOHYDQWOLWHUDWXUHLVLGHQWLILHGZLWKUHVSHFWWRERWKUHVHDUFKDQGSUDFWLFHVXFKWKDWDPRUH FRPSUHKHQVLYHLQLWLDOVRXUFHRI.4,VLVREWDLQHG 'HILQLWLRQRIFRPSRQHQWVDQGDWWULEXWHVRI.4,V ,QWKHVHFRQGVWHSWKHFRPSRQHQWVDQGDWWULEXWHVRIWKH.4,VDUHHVWDEOLVKHGWRVHOHFWDSRRO RI.4,VWKDWDUHGHWHUPLQHGWREHWKHPRVWDSSURSULDWHIRUGHVFULELQJWUDQVLWVHUYLFHTXDOLW\ 7KHRYHUDOOJRDORIWKLVVWHSLVWRSURYLGHHIIHFWLYHGLUHFWLRQIRUWKHHYDOXDWLRQSURFHVV 7RDFKLHYHWKLVJRDOZHFRQVLGHUWZRPDQDJHDEOHFRPSRQHQWV x 7KHPHWKRGRORJLFDOIHDWXUHVRIWKHLQGLFDWRU x 7KHUHOHYDQFHRIWKHLQGLFDWRUWRWKHFRQFHSWRIVHUYLFHTXDOLW\ 7KHVHFRPSRQHQWVZKLOHPRUHVSHFLILFWKDQWKHRYHUDOOJRDOPD\VWLOOEHWRREURDGWR JXLGHWKHVHOHFWLRQSURFHVV7KHUHIRUHZHPXVWIXUWKHUGHFRPSRVHWKHVHFRPSRQHQWVLQWR PRUHPDQDJHDEOHDWWULEXWHV$FFRUGLQJWR>@WKHPHWKRGRORJLFDODWWULEXWHVDUHDVIROORZV x 0HDVXUDELOLW\IRUFRQVLGHULQJWKHWKHRUHWLFDODVSHFWVRIHDFKLQGLFDWRU x (DVHRIDYDLODELOLW\IRUFROOHFWLQJGDWDDWDUHDVRQDEOHFRVW x 6SHHGRIDYDLODELOLW\IRUPDNLQJUHJXODUDVVHVVPHQWXSGDWHV x ,QWHUSUHWDELOLW\WRDYRLGDPELJXRXVRUFRQIXVLQJRXWFRPHV 7KHVDPHUHIHUHQFH>@GRFXPHQWVWKDWWKHUHOHYDQFHRILQGLFDWRUVWRWKHFRQFHSWRI VHUYLFHTXDOLW\LQFOXGHVIRXUPDLQDWWULEXWHV 

 )LJXUH 3URSRVHGUREXVWPHWKRGRORJ\IRULGHQWLI\LQJDQGVHOHFWLQJ.4,VIRUPRQLWRULQJ WUDQVLWVHUYLFHTXDOLW\ x ,QWHJUDWLRQRIXVHUV¶DQGSXEOLFWUDQVSRUWFRPSDQ\37&¶VYLHZVWRUHRULHQWWKH RUJDQLVDWLRQDVRQHWKDWLVFXVWRPHUFHQWUHG x 8VHURULHQWDWLRQWRDYRLG37&RULHQWHGSDUDPHWHUV x 6XEMHFWLYHDQGREMHFWLYHPHDVXUHPHQWWRHYDOXDWHWKHLPSDFWRQWKHVDWLVIDFWLRQRQ WKHSHUIRUPDQFH x 6WDWHPHQWRIWKHDPRXQWRISDVVHQJHUVWRSULRULWLVHDFWLRQV 0HWKRGRORJLFDODWWULEXWHVDUHJHQHUDODVSHFWVRIWKHDSSOLFDWLRQRIWKHPHWKRGZKHUHDV TXDOLW\DWWULEXWHVDUHPRUHVSHFLILFLQWKDWWKH\DFFRXQWIRUKRZZHOOWKH\UHIOHFWWKHLPSDFW RQXVHUVRIFUXFLDODVSHFWVRIVHUYLFHTXDOLW\,QDGGLWLRQWKHOHYHORILPSRUWDQFHDVVLJQHGWR HDFKFRPSRQHQWDQGDWWULEXWHFDQGLIIHUWRUHIOHFWWKHYLHZSRLQWVRIH[SHUWV ,QYROYHPHQWRIH[SHUWVWRHOLFLWWKHLUMXGJHPHQWV ,QWKHVHOHFWLRQRI.4,VWKHWKLUGVWHSLVNH\LQWKDWLWUHTXLUHVWKHLQYROYHPHQWRIDFDGHPLF UHVHDUFKHUVZKRFDQJXLGHWKHFRQFHSWRITXDOLW\DQGSUDFWLWLRQHUVZKRZRUNHYHU\GD\WR HQVXUH VHUYLFH TXDOLW\ +HUHLQDIWHU ZH UHIHU WR UHVHDUFKHUV DQG SUDFWLWLRQHUV DV H[SHUWV

'HILQLWLRQRIFRPSRQHQWVDQGDWWULEXWHVRI .4,VIRUWUDQVLWVHUYLFH ,GHQWLILFDWLRQRIDORQJOLVWRISRWHQWLDO .4,V ,QYROYHPHQWRIH[SHUWVWRHOLFLWWKHLU MXGJPHQWV (YDOXDWLRQRI.4,V 6WDELOLVDWLRQRI.4,V )LQDOVHOHFWLRQRIWKHPRVWSURPLVLQJ VHWRI.4,V

6HYHUDODSSURDFKHVFDQEHXVHGIRUWKHLQYROYHPHQWRIH[SHUWV([SHUWVFDQSDUWLFLSDWHLQ IRFXV JURXSV )*V RU VXUYH\V GHSHQGLQJ RQ WKH DPRXQW RI GDWD UHTXLUHG )*V SURYLGH TXDOLWDWLYHGDWDFROOHFWHGLQDZRUNVKRSVW\OHPHWKRGRORJ\XVLQJDQHVWDEOLVKHGQXPEHURI VHOHFWHG LQGLYLGXDOV 7KH PDLQ REMHFWLYH RI )*V LV WR REWDLQ LQIRUPDWLRQ EDVHG RQ WKH RSLQLRQVRIWKHSDUWLFLSDQWVUHJDUGLQJWKHLVVXHEHLQJLQYHVWLJDWHG>@>@6XUYH\VSURYLGH TXDQWLWDWLYHGDWDFROOHFWHGIURPDODUJHSRRORIXVHUVDQGWKHUHDUHPDQ\VXUYH\W\SHVHJ LQWHUFHSWDWKRPHHPDLOZHEDQGVRRQ$OWKRXJK)*VDUHDQLQWHUHVWLQJWRROIRUWKLV SXUSRVHWKHLUXVHLVFRQILQHGWRDYHU\UHVWULFWHGQXPEHURIH[SHUWV7KHUHIRUHZHSURSRVH WKHXVHRILQWHUQDWLRQDOVXUYH\VLQYROYLQJDUDQJHRIH[SHUWV5HJDUGLQJWKHW\SHRIVXUYH\ XVHGZHVXJJHVWWKHZHEVXUYH\DVLWLVDEOHWRHOLFLWLQIRUPDWLRQDWDORZFRVWFDQ UDSLGO\UHDFKH[SHUWVFDQEHFRPSLOHGGLUHFWO\RQOLQHWKXVUHPRYLQJWKHQHHGWRSULQW WKHTXHVWLRQQDLUHDQGUHVXOWVLQGDWDWKDWDUHUHDG\IRUSURFHVVLQJ ,QWKLVPHWKRGRORJ\H[SHUWVQHHGWREHLQYROYHGWZLFHZKHQJDWKHULQJGDWDLQSXWIRUWKH HYDOXDWLRQRIHDFKLQGLFDWRU)LUVWWKHH[SHUWVKHOSLQSULRULWLVLQJWKHOHYHOVRILPSRUWDQFHRI WKHFRPSRQHQWVDQGDWWULEXWHVZLWKWKHUHFRJQLWLRQWKDWWKHVHOHYHOVRILPSRUWDQFHZLOOOLNHO\ YDU\ DPRQJ H[SHUWV 7KXV ZHLJKWV RI LPSRUWDQFH DUH DWWDFKHG WR WKHVH FRPSRQHQWV DQG DWWULEXWHVLHLWHPVWRUHIOHFWWKHYLHZSRLQWVRIH[SHUWV$OWKRXJKZHLJKWVFDQEHGLUHFWO\ DWWDFKHG E\ TXHVWLRQLQJ H[SHUWV DERXW WKHLU SUHIHUHQFHV RQ VLQJOH LWHPV WKLV DSSURDFK LV LPSHUIHFWDVKXPDQVKDYHGLIILFXOW\WUDQVIRUPLQJUHOHYDQWLQIRUPDWLRQDERXWPDQ\LWHPVLQWR UHODWLYHZHLJKWV>@+HQFHLQWKLVVWXG\ZHXVHGWKHDQDO\WLFDOKLHUDUFK\SURFHGXUH$+3 >@>@UDWKHUWKDQDVVLJQLQJZHLJKWVGLUHFWO\ZHLJKWVDUHHVWDEOLVKHGEDVHGRQDSDLUZLVH FRPSDULVRQRILWHPV0RUHSUHFLVHO\WKH$+3HOLFLWVVXEMHFWLYHFRPSDULVRQVE\H[SHUWVRQ SDLUV RI LWHPV DQG WKHQ DJJUHJDWHV WKHVH RXWFRPHV LQWR REMHFWLYH ZHLJKWV 7KH $+3 LV SHUIRUPHGWZLFHRQFHIRUWKHFRPSRQHQWVDQGDJDLQIRUWKHDWWULEXWHV

6HFRQGWKHH[SHUWVPDNHWKHLUHYDOXDWLRQVE\JLYLQJPDUNVIRUHDFKLQGLFDWRUDJDLQVWHDFK DWWULEXWHDFFRUGLQJWRSUHGHILQHGVFDOHVHJQXPHULFDOTXDOLWDWLYHDQGVRRQ

(YDOXDWLRQRI.4,V

,QWKHIRXUWKVWHSZHFRPSXWHWKHWeighted Quality Indicator Performance ScoreWQIPS IRUHDFK.4,E\DJJUHJDWLQJWKHLUSUHYLRXVO\GHULYHGZHLJKWVDQGRXWFRPHPDUNV 0RUHSUHFLVHO\OHW x I EHWKHVHWRILQGLFDWRUV x JEHWKHVHWRIH[SHUWVLQWHUYLHZHG x H EHWKHVHWRIWKHPHWKRGRORJLFDODWWULEXWHV x KEHWKHVHWRIWKHTXDOLW\DWWULEXWHV x ݉തതതEHWKHDYHUDJHZHLJKWRIWKHPHWKRGRORJLFDOFRPSRQHQW x_{ ݍതEHWKHDYHUDJHZHLJKWRIWKHUHOHYDQFHRITXDOLW\FRPSRQHQW} x ݓതതതതEHWKHDYHUDJHZHLJKWRIDWWULEXWHh אH௛ x_{ ݖഥ EHWKHDYHUDJHZHLJKWRIDWWULEXWHkאK௞} x ܸതതതതEHWKHDYHUDJHPDUNRILQGLFDWRUiא ܫIRUDWWULEXWH hאHప௛ x_{ ܸതതതതEHWKHDYHUDJHPDUNRILQGLFDWRUiא ܫIRUDWWULEXWH kאKప௞} )RUHDFKSDUDPHWHUiWKHWQIPSiLVFRPSXWHGDVIROORZV

ܹܳܫܲܵ

ൌ  ݉ഥ ቀσ

ݓ

തതതതܸ

തതതത

ቁ ൅ݍത ቀσ

ݖ

തതതܸ

തതതത

ቁ ׊݅ ൌ ͳ ǥ ȁȁ



6WDELOLVDWLRQRI.4,

7KHILIWKVWHSLVQHFHVVDU\WRHQVXUHWKHVWDELOLW\RIWKHFRPSXWHGWQIPSLVSDUWLFXODUO\ZKHQ WKHQXPEHURIDQVZHUVSURYLGHGE\H[SHUWVLQWKHVXUYH\LVQRWYHU\ODUJHDVRIWHQKDSSHQV LQ WKHVH NLQGV RI VWXGLHV 6WDELOLVDWLRQ RU ³UREXVWLILFDWLRQ´RI WKH HVWLPDWHG WQIPSLLV

SHUIRUPHGXVLQJ0RQWH&DUORVLPXODWLRQPHWKRGVZKLFKDOORZWKHDQDO\VWWRTXDQWLI\WKH HIIHFWRIUDQGRPYDULDWLRQODFNRINQRZOHGJHRUHUURURQWKHVHQVLWLYLW\SHUIRUPDQFHRU UHOLDELOLW\RIWKHV\VWHPEHLQJPRGHOOHG7KHGHWHUPLQLVWLFPRGHORIH[SHUWMXGJHPHQWVLV LWHUDWLYHO\ HYDOXDWHGXVLQJ DVHW RI UDQGRP QXPEHUV DV LQSXWV,Q WKLVZD\ WKH REVHUYHG GHWHUPLQLVWLFPRGHOLVWUDQVIRUPHGLQWRDVWRFKDVWLFPRGHO7\SLFDOO\WKHWKHRUHWLFDOPRGHO XVHGLQWKHGHULYDWLRQRIWQIPSLLVVLPXODWHGWRSURGXFHDUWLILFLDOGDWDWKDWPXVWEHPDWFKHG ZLWKREVHUYHGGDWDLHMXGJHPHQWVPDGHE\H[SHUWV0DWFKLQJLVSHUIRUPHGE\FKRRVLQJD VXLWDEOH SUREDELOLW\ GLVWULEXWLRQ WKDW EHVW UHSUHVHQWV WKH FXUUHQW VWDWH RI NQRZOHGJH

6SHFLILFDOO\WRGHULYHDUREXVWVHWRIWQIPSLVZHDSSO\WKH0RQWH&DUORPHWKRGWZLFHWR UHWUDFHWKHSURFHVVGHVLJQHGLQWKHWKHRUHWLFDOPRGHOZKLFKOHDGVWRWKHFRPSXWDWLRQRIWKH

WQIPSLE\HTQ7KXVWZR0RQWH&DUORH[SHULPHQWVDUHSHUIRUPHGQDPHO\



x ([SHULPHQW:HVLPXODWHWKHGLVWULEXWLRQRIWKHVFRUHVREWDLQHGIRUWKHHQWLUHVHW RI|H|+|K| LWHPVFKDUDFWHULVLQJWKH$+3PDQ\WLPHV BE\JHQHUDWLQJPXOWLQRPLDO GLVWULEXWLRQV ZLWK |H|+|K| FODVVHV 7KH FKRLFH RI PXOWLQRPLDO GLVWULEXWLRQ LV PRWLYDWHGE\WKHFRQVLGHUDWLRQWKDWLQPRVWFDVHV|H|+|K|!$VVXFKDPXOWL FODVVUHIHUHQFHGLVWULEXWLRQLVUHTXLUHG2IFRXUVHIRUWKHFDVH|H|+|K| WKHVKLIW WRDELQRPLDOGLVWULEXWLRQLVVWUDLJKWIRUZDUG7KHVFRUHVREWDLQHGIURPWKHSDLUZLVH FRPSDULVRQVRILWHPVE\WKHH[SHUWVDUHXVHGDVDSULRULSUREDELOLWLHVRIWKH|H|+|K| FODVVHVLQWKHGDWDJHQHUDWLRQSURFHVV'*37KH'*3LVEDVHGRQDQXPEHURI WULDOV VDPSOH VL]H FRUUHVSRQGLQJ WR WKH QXPEHU RI SDUWLFLSDQWV LQYROYHG LQ WKH VXUYH\DQGLVVLPXODWHGB WLPHV7KHILQDOHVWLPDWHGZHLJKWVIRUHDFKFRPSRQHQW DQGDWWULEXWHDUHWKHDYHUDJHVRIWKHLUPHDQYDOXHVREWDLQHGLQHDFKWULDO x ([SHULPHQW:HUHSHDWWKHSURFHVVXVHGLQ([SHULPHQWIRUWKHVHWRILQGLFDWRUV UHODWHGWRWKH_H|+|K| DWWULEXWHV8VXDOO\ILVYHU\ODUJHDQGWKHWRWDOQXPEHURI LWHPVFRQVLGHUHGLQWKHVLPXODWLRQLVS = _+__._× |I|)RUHDFKLQGLFDWRUiאIWKH GLVWULEXWLRQRIVFRUHVREWDLQHGE\WKHH[SHUWVLVVLPXODWHGB WLPHVXVLQJDELQRPLDO GLVWULEXWLRQX GHILQHGLQWKH>@LQWHUYDOVXFKWKDWE(X) = ܸതതതത ȁܬȁప௛Τ RUE(X) = ܸప௞

തതതത ȁܬȁΤ LI LQGLFDWRU iאI FRQFHUQV DQ DWWULEXWH RI D PHWKRGRORJLFDO RU TXDOLW\ FRPSRQHQWUHVSHFWLYHO\0RUHSUHFLVHO\WKHPHDQVFRUHREWDLQHGIRUDQLQGLFDWRU

iאILVUHVFDOHGRQWKH>@LQWHUYDO0RUHRYHUWKLVLVXVHGDVDSDUDPHWHURIWKH

ELQRPLDO GLVWULEXWLRQ VLPXODWHG IRU WKH QXPEHU RI WULDOV WKDW FRUUHVSRQG WR WKH QXPEHU RI LWHPV FKDUDFWHULVLQJ D VSHFLILF LQGLFDWRU $JDLQ WKH DYHUDJH VFRUH REWDLQHGIRUHDFKLQGLFDWRULVGHULYHGE\DYHUDJLQJWKHVFRUHVREWDLQHGLQWKHYDULRXV VLPXODWLRQWULDOV )LQDOVHOHFWLRQRI.4, ,QWKHODVWVWHSRIWKHSURSRVHGPHWKRGRORJ\LQGLFDWRUVDUHUDQNHGLQGHFUHDVLQJRUGHUEDVHG RQWKHLUHVWLPDWHGWQIPSLV. 7KHEHVWLQGLFDWRUiאLVWKHRQHZLWKWKHKLJKHVWHVWLPDWHG WQIPSL0RUHSUHFLVHO\DGRXEOHRUGHULQJRILQGLFDWRUVLVGHILQHG,QWKHILUVWRUGHULQJWKH

WQIPSLV DUH UHFRPSXWHG E\ UHSODFLQJ HPSLULFDO ZHLJKWV ZLWK WKRVH REWDLQHG IURP ([SHULPHQW  1H[W WKH\ DUH VRUWHG LQWR GHFUHDVLQJ RUGHU ,Q WKH VHFRQG RUGHULQJ WKH

([SHULPHQW7KHILQDOUDQNDVVLJQHGWRHDFKLQGLYLGXDOWQIPSLLVWKHDYHUDJHUDQNREWDLQHG IURPWKHGRXEOHRUGHULQJ7KHRXWSXWRIWKLVSURFHVVLVWKH³UREXVW´VHOHFWLRQRIDSRRORIWRS nLQGLFDWRUVWKDWDUHPRVWUHSUHVHQWDWLYHRIWUDQVLWVHUYLFHTXDOLW\ (;3(5,0(17$/5(68/76$1'',6&866,21 :HLPSOHPHQWHGWKLVPHWKRGRORJ\DVVKRZQLQ)LJ 3UHOLPLQDU\VHWXS

7KH (XURSHDQ &RPPLWWHH IRU 6WDQGDUGLVDWLRQ &(17&  LVVXHG WKH (XURSHDQ 1RUP (1 IRU WKH GHILQLWLRQ RI TXDOLW\ RI VHUYLFH LQ SXEOLF SDVVHQJHU WUDQVSRUW $FFRUGLQJWR6HFWLRQZHXVHG&(17&DVWKHPDLQVRXUFHRI.4,V>@IRUWKH IROORZLQJUHDVRQV,WUHSUHVHQWVDSULPDU\WRRORI37&VWRFHUWLI\WKHTXDOLW\RIWKHLU URXWHV,WHPSKDVLVHVXVHURULHQWHGDQGPHDVXUDEOHLQGLFDWRUVIRUPRQLWRULQJTXDOLW\HYHQ WKRXJKLWGRHVQRWVSHFLI\KRZWRFKRRVHDUHOHYDQWVXEVHWRILQGLFDWRUV,WSURYLGHVWKH PRVWUHOHYDQWDQGOHQJWK\OLVWVRIDYDLODEOHVHUYLFHDVSHFWVLQGLFDWRUVDQGVXE LQGLFDWRUV 1RWDEO\ HDFK VHUYLFH DVSHFW UHSUHVHQWV WKH PDLQ FDWHJRU\ RI D VHW RI UHODWHG LQGLFDWRUVZKLFKUHSUHVHQWWKHXQLWVRIPHDVXUH8VXDOO\WKHVHLQGLFDWRUVFDQEHIXUWKHUVSOLW LQWRVXELQGLFDWRUVDQGHYHQVXEVXELQGLFDWRUV)RUWKHVDNHRIV\QWKHVLVZHUHIHUWRWKHP DOODVLQGLFDWRUV 1H[WWKHFRPSRQHQWVDQGDWWULEXWHVLGHQWLILHGLQ6HFWLRQDUHXVHGDVLQSXWHOHPHQWV WRJXLGHLQWKHVHOHFWLRQSURFHVV ,QWHUQDWLRQDOVXUYH\ $VGHVFULEHGLQ6HFWLRQZHFRQGXFWHGDZHEEDVHGLQWHUQDWLRQDOVXUYH\RIH[SHUWVWR REWDLQ WKHLU MXGJHPHQWV :H VHOHFWHG WKHVH H[SHUWV XVLQJ WZR DSSURDFKHV WKH IRUPHU LQYROYHG DFDGHPLF UHVHDUFKHUV DQG WKH ODWWHU SUDFWLWLRQHUV :H VHOHFWHG WKH DFDGHPLF UHVHDUFKHUVEDVHGRQWKHUHVXOWVRIDUHFHQWVXUYH\RIWKHWRSSXEOLFWUDQVSRUWDWLRQVFLHQWLVWV >@DVZHOODVWKHUDQNLQJRIWKHWRSZRUOGZLGHXQLYHUVLWLHVDVFODVVLILHGE\WKH&HQWUH IRU:RUOG8QLYHUVLW\5DQNLQJVDQGWKH$FDGHPLF5DQNLQJRI:RUOG8QLYHUVLWLHV%DVHGRQ WKHVHFULWHULDZHVHOHFWHGHPDLODGGUHVVHVIURPWKHXQLYHUVLW\ZHEVLWHV6LQFHZHIRXQG QRDYDLODEOHZRUOGZLGHUDQNLQJRI37&VZHXVHGDGLIIHUHQWDSSURDFKWRVHOHFWDSDQHORI LQWHUQDWLRQDOSUDFWLWLRQHUV7KH8QLRQ,QWHUQDWLRQDOGHV7UDQVSRUWV3XEOLFV8,73ZDVRXU PDLQVRXUFHEDVHGRQWKHIDFWWKDWLWLVDQRWIRUSURILWLQWHUQDWLRQDORUJDQLVDWLRQZLWK PHPEHUVIURPFRXQWULHVZRUOGZLGH7KH8,73SURYLGHGDOLVWRI37&PHPEHUVIURP ZKLFKZHUDQGRPO\VHOHFWHGSUDFWLWLRQHUVIRUSDUWLFLSDWLRQLQWKHVXUYH\:HFRQGXFWHGWKH VXUYH\LQWZRVWHSV)LUVWZHDVNHGWKHH[SHUWVWRSHUIRUPSDLUZLVHFRPSDULVRQVRIERWK PHWKRGRORJLFDODQGUHOHYDQFHWRTXDOLW\FRPSRQHQWVDQGUHODWHGDWWULEXWHV7KHLUDQVZHUV ZHUHSURYLGHGDFFRUGLQJWRWKHVFKHPHVKRZQLQ)LJ)RUH[DPSOHWKHLQLWLDOTXHVWLRQZDV ³,I \RX ZHUH DVNHG WR FKRRVH D WUDQVLW VHUYLFH TXDOLW\ LQGLFDWRU EDVHG RQ HLWKHU LWV PHWKRGRORJLFDO IHDWXUH RU LWV UHOHYDQFH WR VHUYLFH TXDOLW\ ZKLFK RI WKH WZR FRPSRQHQWV ZRXOG\RXGHHPPRUHLPSRUWDQWWR\RXUVHOHFWLRQDQGKRZVWURQJO\VR"´ ,QWKHVHFRQGVWHSWKHH[SHUWVJUDGHGHDFKLQGLFDWRUDJDLQVWHDFKDWWULEXWHDQGDVVLJQHG WKHPDVSHFLILFRXWFRPHPDUNDFFRUGLQJWRWKHVFKHPHVDVVKRZQLQWKHH[DPSOHLQ)LJ $OWKRXJKZHXVHGDZRUVWWREHVWVFDOHLQDVVLJQLQJRXWFRPHPDUNVWKLVFKRLFHGRHV QRWDIIHFWWKHJHQHUDOLW\RIWKHPHWKRGZKLFKLVHIIHFWLYHXVLQJDQ\VFDOH0RUHRYHUWKH VLPLODUUHOLDELOLW\RIGLIIHUHQWVFDOHVIURPDVWDWLVWLFDOYLHZSRLQWKDVDOUHDG\EHHQHVWDEOLVKHG DOWKRXJKPRUHUHVSRQVHRSWLRQVWHQGWROHDGWRVRPHZKDWORZHUVFRUHV>@

 )LJXUH6WHS±3DLUZLVHFRPSDULVRQVRIFRPSRQHQWVDQGDWWULEXWHV   )LJXUH6WHS±&ODVVLILFDWLRQPDWUL[±$QH[DPSOH 7KHUHVHDUFKGHVLJQUHTXLUHGWKDWWZRVXUYH\ZDYHVEHFRQGXFWHGLQGXULQJ-DQXDU\ DQG-XO\E\HPDLO,QWKLVHPDLOZHRXWOLQHGWKHDLPVDQGVFRSHRIWKHUHVHDUFKSURMHFWDQG SURYLGHGJXLGHOLQHVDQGDOLQNWRWKHZHEVLWHWRDFFHVVWKHVXUYH\7KHILUVWVXUYH\VWHSZDV

FRPSOHWHGE\H[SHUWVDQGWKHFROOHFWHGGDWD\LHOGHGDJOREDOFRQVLVWHQF\UDWLRRI 7KHVHFRQGVXUYH\VWHSZDVFRPSOHWHGE\H[SHUWV7KHUHVXOWVDUHSUHVHQWHGEHORZ :HLJKWVFRULQJDQGVHOHFWLRQRILQGLFDWRUV 7DEOHUHSRUWVWKHGHULYHGFRPSRQHQWZHLJKWVIRUWKHRULJLQDOGDWDEDVHGRQWKHDSSOLFDWLRQ RIWKH$+37KHUHVXOWVVKRZWKDWH[SHUWVSUHIHUWKH³UHOHYDQFHWRWUDQVLWVHUYLFHTXDOLW\´ FRPSRQHQW 7DEOHUHSRUWVWKHGHULYHGDWWULEXWHZHLJKWVhאHDQGk אK, UHVSHFWLYHO\:HREWDLQHG WKHVHRXWFRPHVE\DSSO\LQJWKH$+3WRWKHRULJLQDOGDWD7KHZHLJKWVLQ7DEOHLOOXVWUDWH WKHGLIIHUHQFHVLQWKHRSLQLRQVRIWKHH[SHUWVZLWKUHVSHFWWRWKHPHWKRGRORJLFDOIHDWXUHVDQG UHOHYDQFHWRWUDQVLWVHUYLFHTXDOLW\ 2QRQHKDQGWKHVHRXWFRPHVLQGLFDWHWKDWH[SHUWVLGHQWLI\³,QWHUSUHWDELOLW\´DVWKHPRVW LPSRUWDQWDWWULEXWHWRFRQVLGHUZKHQHYDOXDWLQJWKHPHWKRGRORJLFDOIHDWXUHRIDFRPSRQHQW 2Q WKH RWKHU KDQG ³8VHU2ULHQWDWLRQ´ LV WKH PRVW LPSRUWDQW DWWULEXWH WR FRQVLGHU ZKHQ HYDOXDWLQJ LWV UHOHYDQFH WR WUDQVLW VHUYLFH TXDOLW\ 1H[W DV RXWOLQHG LQ 6HFWLRQ  ZH FRPSXWHGWKHRULJLQDOWQIPSiV:HFRQGXFWHGH[SHULPHQWVDQGDVGHVFULEHGLQ6HFWLRQ ZLWK% DQG6 UHVSHFWLYHO\0RUHRYHUIRUHDFKLQGLFDWRUZHREWDLQHGDQG UDQNHGLWVWQIPSi LQGHFUHDVLQJRUGHUIRUERWKWKHRULJLQDODQGVWDELOLVHGYDOXHVDVGHVFULEHG LQ 6HFWLRQ 7DEOH VKRZV WKHRXWFRPHV LQZKLFK FROXPQV  WR OLVW WKH WKUHHEHVW LQGLFDWRUVRIHDFKRUJDQLVDWLRQDOVHUYLFHDVSHFWDVSURYLGHGE\>@DQGFROXPQVDQG UHSRUWWKHVFRUHVRIWKHVWDELOLVHGDQGRULJLQDOWQIPSiV.

7KHVH UHVXOWV UHYHDO WKDW VWDELOLVLQJ WKH RULJLQDO VFRUHV SDUWLDOO\ FKDQJHG WKHLU H[SHUW HYDOXDWLRQUDQNLQJVRULJLQDOWQIPSi0RUHRYHUVRPHUHOHYDQWFKDQJHVLQWKHVFRUHVRI



7DEOH:HLJKWVRIFRPSRQHQWV

&RPSRQHQW 6\PERO 2ULJLQDOZHLJKW

0HWKRGRORJLFDOIHDWXUHV ݉ഥ 

5HOHYDQFHWRWUDQVLWVHUYLFHTXDOLW\ ݍത 

7DEOH:HLJKWVRIDWWULEXWHV

$WWULEXWH 6\PERO 2ULJLQDOZHLJKW

0HDVXUDELOLW\ ݓതതതതଵ  (DVHRIDYDLODELOLW\ ݓതതതതଶ  6SHHGRIDYDLODELOLW\ ݓതതതതଷ  ,QWHUSUHWDELOLW\ ݓതതതതସ  ,QWHJUDWLRQRIXVHUV±FRPSDQLHV ݖഥ ଵ  8VHURULHQWDWLRQ ݖഥ ଶ  6XEMHFWLYHDQGREMHFWLYHPHDVXUDELOLW\ ݖഥ ଷ  $PRXQWRISDVVHQJHUV ݖഥ ସ 

7DEOH 5HVXOWVRIVWDELOLVHGDQGRULJLQDOWQIPSiVIRUWKHEHVWWKUHHLQGLFDWRUVRIHDFK VHUYLFHDVSHFW  VHYHUDOLQGLFDWRUVDSSHDUZKHQFRPSDULQJWKHRULJLQDODQGVWDELOLVHGVFRUHV:HQRWHWKDW VWDELOLVDWLRQWHQGVWRUHGXFHWKHRULJLQDOVFRUHVLQPRVWFDVHVVHHWKHODVWFROXPQRI7DEOH 7KLVPD\EHGHSHQGHQWRQWKHGLVWULEXWLRQRIVFRUHVREWDLQHGIRUDFHUWDLQLWHPIURPDOOWKH

6HUYLFHDVSHFW ,QGLFDWRU 6XELQGLFDWRU 6WDELOLVHG_:4,36L 2ULJLQDO_:4,36L

$YDLODELOLW\ 'HSHQGDELOLW\2SHUDWLRQ )UHTXHQF\  

2SHUDWLRQ 9HKLFOHORDGIDFWRU   $FFHVVLELOLW\ 7LFNHWLQJ DYDLODELOLW\ $FTXLVLWLRQRQQHWZRUN   7LFNHWLQJ DYDLODELOLW\ 9DOLGDWLRQ   ,QWHUQDOLQWHUIDFH 7UDQVIHUWRRWKHU337_PRGHV    ,QIRUPDWLRQ 7UDYHOLQIRUPDWLRQ RQQRUPDO FRQGLWLRQV $ERXWWLPH   *HQHUDOLQIRUPDWLRQ $ERXWVHFXULW\    7UDYHOLQIRUPDWLRQ QRUPDOFRQGLWLRQV $ERXWURXWH   7LPH $GKHUHQFHWR VFKHGXOH 5HJXODULW\   $GKHUHQFHWR VFKHGXOH 3XQFWXDOLW\   /HQJWKRIWULSWLPH ,QYHKLFOH   &XVWRPHUFDUH 7LFNHWLQJRSWLRQV 3D\PHQWRSWLRQV   $VVLVWDQFH )RUFXVWRPHUV_QHHGLQJKHOS   $VVLVWDQFH _{LQWHUUXSWLRQV} $WVHUYLFH   &RPIRUW 6HDWLQJDQGSHUVRQD VSDFH $WEDSRLQWV   6HDWLQJDQGSHUVRQD VSDFH ,QYHKLFOH   $PELHQWFRQGLWLRQV &OHDQOLQHVV    6HFXULW\ )UHHGRPIURP FULPH 6WDIISROLFHSUHVHQFH   )UHHGRPIURP DFFLGHQW $YRLGDQFHYLVLELOLW\RIKD]DUGV   )UHHGRPIURP FULPH /LJKWLQJ   (QYLURQPHQWDO LPSDFW 3ROOXWLRQ 1RLVH   3ROOXWLRQ 2GRXU   3ROOXWLRQ 'XVWDQG GLUW   

H[SHUWV,QGHHGWKHGLVWULEXWLRQLVUDWKHUKRPRJHQHRXVZLWKDWHQGHQF\WRZDUGFHQWUDOYDOXHV EHWZHHQWKHLQWHUYDOVWRH[FHSWIRUVRPHVSRUDGLFKLJKVFRUHVZKRVHHIIHFWRQWKHPHDQ VFRUHLVPLWLJDWHGE\WKH0RQWH&DUORVLPXODWLRQV0RUHRYHUIRULQGLFDWRUVPDUNHGZLWK³ ´ VWDELOLVDWLRQFRQVLVWHQWO\LPSURYHVWKHLUUDQNLQJV,QGHHGWKHVHLQGLFDWRUVDUHLQWKHWRSWKUHH ZLWKUHVSHFWWRWKHVWDELOLVHGWQIPSiVHYHQLIWKH\KDGRFFXSLHGWKHORZHVWSRVLWLRQLQWKH RULJLQDOUDQNLQJ)RUWKHVHFDVHVLWLVOLNHO\WKDWWKHRULJLQDOVFRUHVDUHPRUHKRPRJHQHRXV ZKLFK \LHOGV DQ LQFUHDVH LQ WKH VWDELOLVHG VFRUHV )XUWKHUPRUH WKH VWDELOLVDWLRQ SURYLGHV HYLGHQFHWKDWWKHWKUHHEHVWLQGLFDWRUVDUHWKRVHUHODWHGWRLQIRUPDWLRQDERXWWLPHVHFXULW\DQG URXWH UDWKHU WKDQ WKH IUHTXHQF\ UHJXODULW\ DQG LQIRUPDWLRQ DERXW WLPH GHULYHG IURP WKH RULJLQDOWQIPSiV. 7KHVHVWDELOLVHGRXWFRPHVDUHXQH[SHFWHG7KH\GLIIHUIURP>@ZKRVH DXWKRUVUHSRUWHGWKDWVHUYLFHUHOLDELOLW\ZKLFKLQFOXGHVWLPHLQGLFDWRUVLVWZLFHDVLPSRUWDQW DV IUHTXHQF\ DQG DOPRVW VHYHQ WLPHV PRUH LPSRUWDQW WKDQ LQIRUPDWLRQ IRU SDVVHQJHUV ,Q DGGLWLRQ WKH\ DUH GHULYHG IURP D 8. SUDFWLFDO JXLGHOLQH ZKLFK UHSRUWV SDVVHQJHUV¶ SHUFHSWLRQVRIORFDOEXVVHUYLFHVZKLFKLVXVHGLQUDQNLQJVRILPSRUWDQFH )LQDOO\ZHFRQVLGHUWKHVWDELOLVHGDYHUDJHVFRUHVREWDLQHGIRUHDFKVHUYLFHDVSHFW)LJ VKRZVSORWVRIWKHGLVWULEXWLRQVRIWKHVHDYHUDJHVFRUHVIRUWKHDQDO\VLVRIWKHGLVWULEXWLRQRI HDFKLQGLYLGXDOVHWRIVFRUHVDQGIRURXUHYDOXDWLRQRIWKHPRVWDQGOHDVWLPSRUWDQWVFRUHV:H VRUWHGWKHER[SORWVLQDWRSGRZQPDQQHUWRSURYLGHHYLGHQFHUHJDUGLQJWKHPRVWLPSRUWDQW VHUYLFHDVSHFWVRQDYHUDJH³7LPH´LVWKHVHUYLFHDVSHFWWKDW\LHOGVWKHKLJKHVWWQIPSiRQ DYHUDJHZKHUHDVHQYLURQPHQWDOLPSDFWLVUDQNHGDVOHVVLPSRUWDQW0RUHRYHU)LJVKRZV WKHYDULDELOLW\RIWKHDYHUDJHVFRUHVZLWKLQHDFKVHUYLFHDVSHFW,QWKLVUHVSHFWVHFXULW\LVWKH PRVWKRPRJHQHRXVVHUYLFHDVSHFWDVWKHWQIPSiVRIWKHDVVRFLDWHGLQGLFDWRUVDUHTXLWHFORVH WRWKHDYHUDJHVFRUHV&RQYHUVHO\WKHVHUYLFHDVSHFW³(QYLURQPHQWDO,PSDFW´VKRZVWKHPRVW GLVSHUVHGWQIPSiV &21&/86,216$1'5(6($5&+3(563(&7,9(6

5HFHQW DGYDQFHPHQWV LQ WUDQVLW VHUYLFHV KDYH FDSWXUHG WKH LQWHUHVW RI H[SHUWV UHJDUGLQJ PHWKRGV IRU PRQLWRULQJ WKH TXDOLW\ RI SXEOLF WUDQVSRUW VHUYLFHV 0DQ\ VWXGLHV KDYH EHHQ SURSRVHGIRUPRQLWRULQJVHUYLFHTXDOLW\DQGDSSURSULDWHUHFRPPHQGDWLRQVPDGH0RUHRYHU   )LJXUH'LVWULEXWLRQRIDYHUDJHVWDELOLVHGWQIPSVIRUHDFKVHUYLFHDVSHFW>@ (, $F F & 6 & & $Y * , 7P        7P 7LPH *, *HQHUDO,QIRUPDWLRQ $Y $YDLODELOLW\ && &XVWRPHU&DUH 6 6HFXULW\ & &RPIRUW $FF $FFHVVLELOLW\ (, (QYLURQPHQWDO,PSDFW

WRWKHEHVWRIRXUNQRZOHGJHSXEOLFWUDQVSRUWFRPSDQLHVPRQLWRUVHUYLFHTXDOLW\EDVHGRQDG KRFLQGLFDWRUVUDWKHUWKDQRQWKRVHGHULYHGXVLQJDFOHDUDQGZHOORUJDQLVHGPHWKRGRORJ\ +RZHYHUWKHFKRLFHRIVSHFLILF.4,VUHSUHVHQWVDFUXFLDOHOHPHQW7KLVSDSHUFRQWULEXWHV WR WUDQVLW VHUYLFH UHVHDUFK E\ SURSRVLQJ D UREXVW PHWKRGRORJ\ IRUGHULYLQJ WRS.4,V IRU GHVFULELQJWUDQVLWVHUYLFHTXDOLW\7KHPHWKRGRORJ\FRPSULVHVVL[VWHSV)LUVWDOHQJWK\OLVW RILQGLFDWRUVLVLGHQWLILHG1H[WWKHJRDODVZHOODVWKHFRPSRQHQWVDQGDWWULEXWHVDUHGHILQHG IRUHDFKLQGLFDWRUDQGOHDGLQJH[SHUWVDUHLQYROYHGWRHOLFLWMXGJHPHQWIRUHDFKLQGLFDWRU 1H[W HDFK LQGLFDWRU LV HYDOXDWHG DQG WKH PRVW SURPLVLQJ VHW LV GHWHUPLQHG :H GHPRQVWUDWHGWKHUREXVWQHVVRIWKLVPHWKRGWZLFH)LUVWZHJDWKHUHGGDWDLQSXWFRPSULVLQJ LQGLFDWRUZHLJKWVDQGPDUNVLQDZHEEDVHGLQWHUQDWLRQDOVXUYH\6HFRQGO\ZHVWDELOLVHGWKH RXWFRPHVXVLQJ0RQWH&DUORVLPXODWLRQPHWKRGV

7KLVPHWKRGLVDQWLFLSDWHGWRDGYDQFHWKHVWDWHRIWKHDUWERWKWKHRUHWLFDOO\DQGLQSUDFWLFH ,W ILOOV D JDS LQ WKH FKDUDFWHULVDWLRQ RI WUDQVLW TXDOLW\ LH WKH VHOHFWLRQ RI LQGLFDWRUV 0RUHRYHULWSURYLGHVRXWFRPHVZKLFKFDQEHUHDGLO\DSSOLHGE\HYHU\37&WRPRQLWRULWV VHUYLFHTXDOLW\ 7RWKHEHVWRIRXUNQRZOHGJHWKLVLVWKHILUVWSDSHUSURYLGLQJDVHWRI.4,WKDWGHVFULEH VHUYLFHTXDOLW\EDVHGRQDUREXVWPHWKRGRORJ\ 7RFRQFOXGHZHVXJJHVWWZRIXWXUHUHVHDUFKGLUHFWLRQV)LUVWWKLVPHWKRGFDQEHHDVLO\ DSSOLHGWRRWKHUWUDQVSRUWDWLRQPRGHVHJ DLUPDULWLPHDQGVRRQDVZHOODVRWKHUJHQHUDO VHUYLFHV6HFRQGDPRUHFKDOOHQJLQJVWXG\EDVHGRQVRFLDOZHEDQGDGYDQFHPHQWVLQZHE VHPDQWLFVFRXOGLQYROYHXVHUVUDWKHUWKDQH[SHUWVWRFRQVWUXFWDOLVWRILQGLFDWRUV7KLVLVPDGH SRVVLEOH E\ WKH XVH RI IHHGEDFN REWDLQHG IURP XVHUV RQ VRFLDO PHGLD UHJDUGLQJ WKHLU H[SHULHQFHVZLWKWUDQVSRUWDWLRQFRPSDQLHVHJ)DFHERRN7ZLWWHU,QVWDJUDP,QWKLVZD\ QHZLQGLFDWRUVFDQEHGHULYHGRUROGHURQHVFRQILUPHG $&.12:/('*(0(17 7KLVZRUNKDVEHHQSDUWLDOO\VXSSRUWHGE\WKH,WDOLDQ0LQLVWU\RI8QLYHUVLW\DQG5HVHDUFK 0,85ZLWKLQWKH6PDUW&LW\IUDPHZRUNSURMHFW321DB³&$*/,$5,´ 5()(5(1&(6 >@ /H&DOOHW30|OOHU6 3HUNLV$HGV4XDOLQHW:KLWH3DSHURQ'HILQLWLRQVRI 4XDOLW\ RI ([SHULHQFH (XURSHDQ QHWZRUN RQ TXDOLW\ RI H[SHULHQFH LQ PXOWLPHGLD V\VWHPVDQGVHUYLFHV&267$FWLRQ,&/DXVDQQH6ZLW]HUODQG9HUVLRQ 0DUFK

>@ 7UDQVSRUWDWLRQ±/RJLVWLFVDQGVHUYLFHV(XURSHDQ6WDQGDUG(1

>@ 'H2xD- 'H2xD54XDOLW\RIVHUYLFHLQSXEOLFWUDQVSRUWEDVHGRQFXVWRPHU VDWLVIDFWLRQ VXUYH\V $ UHYLHZ DQG DVVHVVPHQW RI PHWKRGRORJLFDO DSSURDFKHV

Transportation ScienceSS±'2,WUVF

>@ GH2xD-'H2xD5(EROL/ 0D]]XOOD*,QGH[QXPEHUVIRUPRQLWRULQJWUDQVLW VHUYLFHTXDOLW\Transportation Research Part A: Policy and PracticeSS±

'2,MWUD >@ (EROL/ 0D]]XOOD*$PHWKRGRORJ\IRUHYDOXDWLQJWUDQVLWVHUYLFHTXDOLW\EDVHG RQVXEMHFWLYHDQGREMHFWLYHPHDVXUHVIURPWKHSDVVHQJHU¶VSRLQWRIYLHZTransport PolicySS±'2,MWUDQSRO >@ (EROL/ 0D]]XOOD*3HUIRUPDQFHLQGLFDWRUVIRUDQREMHFWLYHPHDVXUHRISXEOLF WUDQVSRUWVHUYLFHTXDOLW\European TransportSS± >@ 1DWKDQDLO(0HDVXULQJWKHTXDOLW\RIVHUYLFHIRUSDVVHQJHUVRQWKH+HOOHQLFUDLOZD\V

Transportation Research Part A: Policy and PracticeSS±'2,

>@ 7\ULQRSRXORV< $QWRQLRX&$FRPSOHWHPHWKRGRORJ\IRUWKHTXDOLW\FRQWURORI SDVVHQJHUVHUYLFHVLQWKHSXEOLFWUDQVSRUWEXVLQHVVEuropean TransportSS±  >@ 7\ULQRSRXORV< $QWRQLRX&3XEOLFWUDQVLWXVHUVDWLVIDFWLRQYDULDELOLW\DQGSROLF\ LPSOLFDWLRQVTransport PolicySS± '2,MWUDQSRO >@ GHOO¶2OLR/,EHDV$ &HFLQ37KHTXDOLW\RIVHUYLFHGHVLUHGE\SXEOLFWUDQVSRUW XVHUVTransport PolicySS± '2,MWUDQSRO

>@ +DVVDQ 01 +DZDV -(  $KPHG . $ PXOWLGLPHQVLRQDO IUDPHZRUN IRU HYDOXDWLQJWKHWUDQVLWVHUYLFHSHUIRUPDQFHTransportation Research Part A: Policy

and PracticeSS±'2,MWUD

>@ 6WUDGOLQJ 6 $QDEOH -  &DUUHQR 0 3HUIRUPDQFH LPSRUWDQFH DQG XVHU GLVJUXQWOHPHQW $ VL[ PHWKRG IRU PHDVXULQJ VDWLVIDFWLRQ ZLWK WUDYHO PRGHV

Transportation Research Part A: Policy and PracticeSS±'2,

MWUD

>@ &KHQ;<X/=KDQJ< *RX-$QDO\]LQJXUEDQEXVVHUYLFHUHOLDELOLW\DWWKH VWRSURXWHDQGQHWZRUNOHYHOVTransportation Research Part A: Policy and Practice

SS±'2,MWUD

>@ /LQ-:DQJ0/ %DUQXP3'$TXDOLW\FRQWUROIUDPHZRUNIRUEXVVFKHGXOH UHOLDELOLW\ Transportation Research Part E: Logistics and Transportation Review

SS±'2,MWUH

>@ %DUDELQR% 'L)UDQFHVFR0&KDUDFWHUL]LQJPHDVXULQJDQGPDQDJLQJWUDQVLW VHUYLFHTXDOLW\Journal of Advanced TransportationSS±'2,

DWU

>@ 0DKPRXG00+LQH- .DVK\DS$%XVWUDQVLWVHUYLFHTXDOLW\PRQLWRULQJLQ 8.$PHWKRGRORJLFDOIUDPHZRUNProceedings of the ITRN 

>@ &DVWLOOR+ 3LWILHOG'((/$67,&²$PHWKRGRORJLFDOIUDPHZRUNIRULGHQWLI\LQJ DQG VHOHFWLQJ VXVWDLQDEOH WUDQVSRUW LQGLFDWRUV Transportation Research Part D:

Transport and EnvironmentSS±

'2,MWUG

>@ ,EHDV $ GHOO¶2OLR /  0RQWHTXtQ 5% &LWL]HQ LQYROYHPHQW LQ SURPRWLQJ VXVWDLQDEOH PRELOLW\ Journal of Transport Geography, SS ± 

'2,MMWUDQJHR

>@ =HODQ\0$FRQFHSWRIFRPSURPLVHVROXWLRQVDQGWKHPHWKRGRIWKHGLVSODFHGLGHDO

Computers &Operations Research±SS±'2,



>@ 6DDW\7/The Analytic Hierarchy Process,0F*UDZ+LOO1HZ<RUN

>@ 6DDW\ 5: 7KH DQDO\WLF KLHUDUFK\ SURFHVV²ZKDW LW LV DQG KRZ LW LV XVHG

Mathematical modelling±SS± >@ +HLOLJ/ 9R‰6$VFLHQWRPHWULFDQDO\VLVRISXEOLFWUDQVSRUWUHVHDUFKJournal of Public TransportationSS±'2, >@ 'DZHV-'RGDWDFKDUDFWHULVWLFVFKDQJHDFFRUGLQJWRWKHQXPEHURIVFDOHSRLQWVXVHG" $Q([SHULPHQW8VLQJ3RLQW3RLQWDQG3RLQW6FDOHVInternational Journal of Market ResearchSS±'2, >@ %DOFRPH5HWDO7KHGHPDQGIRUSXEOLFWUDQVSRUWDSUDFWLFDOJXLGH75/5HSRUW 75/75//LPLWHG