Validity and reliability of a test used to assess university students' academic language proficiency

Peer review

Submitted 2019-01-10 Accepted 2019-02-09 Published 2019-06-07 Open access

© 2019 | cb Creative Commons Attribution 4.0 International Public License EL.LE

Vol. 7 – Num. 3 – Novembre 2018

Validity and Reliability

of a Test Used to Assess

University Students’ Academic

Language Proficiency

Marco Mezzadri

Università degli Studi di Parma, Italia

Flora Sisti

Università degli Studi di Urbino “Carlo Bo”, Italia

Abstract This article presents a test model developed to assess university students’ academic language proficiency. The context is that of Italian universities, which are re-quired by law to assess first year students’ initial preparation. Drawing on the literature regarding test construct, it explores the validity and reliability of the test through the analysis of the data collected during an experimental implementation phase carried out at the Universities of Genova, Parma and Urbino.

Keywords Educational linguistics. Testing academic language proficiency. Assessing university students’ initial preparation. Test validity. Test reliability.

Summary 1 Introduction. – 2 Context, Methods and Participants. – 2.1 Aims and Struc-ture of the VPI Test. – 2.2 Online and OMR Paper Versions. – 3 Results.– 4 Discussion and Conclusions.

1

Introduction

,WDOLDQXQLYHUVLWLHVDUHUHTXLUHGWRDVVHVVೳUVW\HDUVWXGHQWVވLQLWLDO SUHSDUDWLRQ93,LQDOOWKHGHJUHHFRXUVHVZKLFKGRQRWKDYHDG-PLVVLRQWHVWV 7HVWLQJZDVPDGHFRPSXOVRU\E\ODZLQDQGVLQFHWKHQWKHUH KDYHKDVEHHQDYDULHW\RILQWHUSUHWDWLRQVRQKRZWRDVVHVVVWXGHQWVވ LQLWLDOSUHSDUDWLRQDQGKRZWRIROORZXSZLWKUHPHGLDOZRUNDLPHGWR IXOೳOOWKHUHTXLUHPHQWVVHWE\HDFKFRXUVHRIVWXG\ $JURXSRIUHVHDUFKHUVLQHGXFDWLRQDOOLQJXLVWLFVIURPWKUHH,WDO-LDQXQLYHUVLWLHV*HQRYD3DUPDDQG8UELQRLVFRQGXFWLQJDVWXG\ ZLWKWKHDLPRIFUHDWLQJDFURVVFXUULFXODUDVVHVVPHQWSURJUDPPH IRUODQJXDJHFRPSHWHQFHVDQGFRPPXQLFDWLRQVNLOOVLQSDUWLFXODU UHDGLQJDQGOLVWHQLQJFRPSUHKHQVLRQDGGUHVVHGWRೳUVW\HDUVWX-GHQWV 7KHIUDPHZRUNDGRSWHGLVEDVHGRQODQJXDJHHGXFDWLRQSULQFL-SOHVDQGDLPVWRKLJKOLJKWWKHVWUDWHJLFDQGFURVVFXUULFXODUUROHRI VWXGHQWVވODQJXDJHFRPSHWHQFHDQGFRPPXQLFDWLRQVNLOOVZKHQHQ-WHULQJXQLYHUVLW\ 7KHVWXG\1UHOLHVRQWKHUHVHDUFKHUVވGHFDGHORQJH[SHULHQFHLQ WKHೳHOGVRIWHDFKLQJOHDUQLQJDQGDVVHVVLQJ,WDOLDQODQJXDJHFRP-SHWHQFHIRUDFDGHPLFSXUSRVHV&RQWHQWDQG/DQJXDJH,QWHJUDWHG /HDUQLQJ&/,/PHWKRGRORJ\DQGWHDFKLQJDQGOHDUQLQJWKURXJK ,QIRUPDWLRQDQG&RPPXQLFDWLRQ7HFKQRORJ\,&7 ,QWKHDFDGHPLF\HDUDSLORWYHUVLRQRIDWHVWZDVFUHDWHG ZKLFKೳUVW\HDUVWXGHQWVIURPGLೲHUHQWFRXUVHVLQ3DUPDDQG8UEL-QRZHUHUHTXLUHGWRWDNHLQHLWKHUDQRQOLQHSDSHURURSWLFDOPDUN UHFRJQLWLRQ205YHUVLRQDIWHUSUDFWLFLQJWKHWHVWIRUPDWWKURXJK DQRQOLQHPRFNWHVW,QWKHIROORZLQJDFDGHPLF\HDUDUHYLVHGYHU-VLRQRIWKHWHVWZDVDGPLQLVWHUHGWRVWXGHQWVIURPERWKXQLYHUVLWLHV 7KHGDWDFROOHFWHGDUHDQDO\VHGLQWKHSUHVHQWDUWLFOHDQGUHVXOWV RIWKHRQJRLQJSURMHFWDUHLOOXVWUDWHG7KHDLPLVERWKWRYHULI\WKH YDOLGLW\DQGUHOLDELOLW\RIWKHWHVWFRQVWUXFWZLWKWKHKHOSRIVWDWLVWLFDO WRROVDQGWRUH೴HFWXSRQODQJXDJHFRPSHWHQFHLVVXHV7KHTXDQWLWD-WLYHDQDO\VLVSURSRVHGLVDFFRPSDQLHGE\DUH೴HFWLRQRQWKHDVSHFWV HQWDLOHGLQDQDSSURDFKIRFXVHGRQWKHDFDGHPLFODQJXDJHSURೳFLHQ-F\RIXQLYHUVLW\VWXGHQWVLQGLೲHUHQWGLVFLSOLQDU\ೳHOGV 1ݲ0DUFR0H]]DGULFRQWULEXWHGWRSODQWKHWHVWWRFROOHFWDQGDQDO\VHGDWDDQGWR ZULWHWKHPDQXVFULSWLQSDUWLFXODUii)ORUD6LVWLFRQWULEXWHGLQ SODQQLQJWKHWHVWFROOHFWLQJGDWDDQGZULWLQJWKHPDQXVFULSWLQSDUWLFXODUii (WKLFVVWDWHPHQWVWKHUHVHDUFKZDVEDVHGRQDUHWURVSHFWLYHDQDO\VLVRISUHYLRXV-O\FROOHFWHGDQGDQRQ\PL]HGGDWDDQGWKHUHIRUHDQHWKLFVDSSURYDOIRUWKLVUHVHDUFK ZDVQRWUHTXLUHGDVSHURXU,QVWLWXWLRQDO5HYLHZ%RDUGވVJXLGHOLQHVDQGUHJXODWLRQV $VDFRQVHTXHQFHRIWKHODFNRIGLUHFWFRQWDFWZLWKKXPDQVXEMHFWVLQIRUPHGFRQVHQW SURFHGXUHVZHUHQRWDSSOLFDEOH

7KHFRQFHSWRIYDOLGLW\LVFHQWUDOWRUHVHDUFKDQGDSSOLFDWLRQLQ ODQJXDJHWHVWLQJ2YHUWKH\HDUVUHVHDUFKHUVKDYHWDNHQGLೲHUHQW SHUVSHFWLYHVQRWRQO\DVUHJDUGVYDOLGLW\EXWDOVRYDOLGDWLRQ,QRQHRI KHUFRQWULEXWLRQV&KDSHOOHSURYLGHVDKLVWRULFDORYHU- YLHZRIWKHFRQFHSWRIYDOLGLW\DQGWKHSURFHVVRIYDOLGDWLRQWRJHWK- HUZLWKDGLVFXVVLRQRIWRGD\ވVFULWLFDOLVVXHVLQWKLVGRPDLQ6KHDU-JXHVWKDWGLೲHUHQWDSSURDFKHVKDYHOHGDXWKRUVWRDWWULEXWHGLೲHUHQW PHDQLQJVWRWKHFRQFHSWRIYDOLGLW\DQGWKDWWKHNH\TXHVWLRQVUH-PDLQXQFKDQJHGKRZGRZHLQWHUSUHWWKHPHDQLQJRIWHVWVFRUHVDQG KRZFDQWKH\EHXVHG" 7KHDPRXQWRIUHVHDUFKFDUULHGRXWEDFNHGE\DQHYHUJURZLQJ QHHGIRUODQJXDJHWHVWVDQGFHUWLೳFDWLRQLVVRYDVWWKDWH[WHQGHGUH-YLHZVRIFRQWHPSRUDU\WKLQNLQJKDYHEHHQSURYLGHGDWPDQ\GLೲHUHQW SRLQWVLQWLPH/HWXVMXVWFLWHVRPHRIWKHPRVWSURGXFWLYHUHVHDUFK-HUVLQWKHೳHOGDQGWKHLUNH\ZRUNV$OGHUVRQ$OGHUVRQ DQG%DQHUMHH%DFKPDQ%DFKPDQDQG3DOPHU &KDSHOOH+DPS/\RQVDQG/\QFK.DQH .XQQDQ6RPHUHYLHZVKDYHOLQNHGGLೲHUHQWW\SHVRIYD-OLGLW\WRTXDQWLWDWLYHPHWKRGV&XPPLQJ%DFKPDQDQG(LJQRU .XQQDQ &KDSHOOHDOVRIRFXVHVRQFXUUHQWGHYHORSPHQWVVXFKDVWKRVHRXW-OLQHGE\.DQHZLWKHPSKDVLVRQLQWHUSUHWLYHDUJXPHQWVDQG VWDWHVWKDWދYDOLGLW\LQPDLQVWUHDPODQJXDJHDVVHVVPHQWPD\EHPRY-LQJIRUZDUGLQKDUPRQ\ZLWKHGXFDWLRQDOPHDVXUHPHQWތ 7KLVSHUVSHFWLYHLVFRQVLGHUHGSDUWLFXODUO\UHOHYDQWIRURXUWHVWDO- WKRXJKZHVKDUH&KDSHOOHވVRSLQLRQWKDWދZLWKLQHGXFDWLRQDOPHDV- XUHPHQWWKHFRQFHSWLRQRIYDOLGLW\LVQRWDVHWWOHGPDWWHUEXWUDWK-HUDVRXUFHRIFRQWLQXLQJGLVFXVVLRQތ,WLVZLWKWKLVLQPLQGWKDW WKHLVVXHRIYDOLGLW\LVDGGUHVVHGWKURXJKTXDQWLWDWLYHDQDO\VLVLQ WKLVDUWLFOH $V0F1DPDUDDQG'RXJODVSRLQWRXWWHVWVGHDOLQJ ZLWKODQJXDJHIRUVSHFLೳFSXUSRVHVFDQQRWDYRLGORRNLQJLQWRWKH FRPSOH[LW\RIFRPPXQLFDWLYHFRPSHWHQFHWKXVLQFOXGLQJVWUDWHJLHV NQRZOHGJHDQGWKHFRQWH[WWKURXJKZKLFKLWLVGLVSOD\HG$FFRUGLQJ WR&KDSHOOHދ>W@KHFRQVWUXFWGHೳQLWLRQKDVWRLQFOXGHWKH GRPDLQRIODQJXDJHXVHDVZHOODVWKHDELOLW\WRPDNHDSSURSULDWH OLQJXLVWLFFKRLFHVDQGLQWHUSUHWDWLRQVLQRUGHUWRPDNHPHDQLQJތ 7KLVDSSURDFKUHTXLUHVDUHYLVLRQRIWUDGLWLRQDOYLHZVRIYDOLGLW\ ZKLFKFDQEHVXPPDULVHGLQ/DGRވVDVVXPSWLRQދ'RHVD WHVWPHDVXUHZKDWLWLVVXSSRVHGWRPHDVXUH",ILWGRHVLWLVYDOLGތ :HLUVXJJHVWVDPRGLೳFDWLRQFODLPLQJWKDWދYDOLGLW\LVPXOWLIDFHWHG DQGGLೲHUHQWW\SHVRIHYLGHQFHDUHQHHGHGWRVXSSRUWDQ\FODLPVIRU WKHYDOLGLW\RIVFRUHVRQDWHVWތDQGWKDW >Y@DOLGLW\LVSHUKDSVEHWWHUGHೳQHGDVWKHH[WHQWWRZKLFKDWHVW

FDQEHVKRZQWRSURGXFHGDWDLHWHVWVFRUHVZKLFKDUHDQDFFX-UDWHUHSUHVHQWDWLRQRIDFDQGLGDWHވVOHYHORIODQJXDJHNQRZOHGJH RUVNLOOV,QWKLVUHYLVLRQYDOLGLW\UHVLGHVLQWKHVFRUHVRQDSDU-WLFXODUDGPLQLVWUDWLRQRIDWHVWUDWKHUWKDQLQWKHWHVWper se 5HVHDUFKHUVKDYHDOVRGHYHORSHGGLೲHUHQWIUDPHZRUNVWRPDNHWKHLU FRQFHSWLRQRIYDOLGLW\H[SOLFLWDQGDSSOLFDEOHWRSUDFWLFDOFRQWH[WV 2QHRIWKHVHLV:HLUވVIUDPHZRUNZKLFKDSSHDUVWREHH[KDXV- WLYHDQGFRKHUHQWZLWKWKHDLPVRIRXUWHVWDQGDJRRGZD\WRFODUL-I\LWVGHVLJQDQGLWLVDGRSWHGLQWKLVVWXG\ 5HOLDELOLW\LVWKHRWKHUPRVWLPSRUWDQWIHDWXUHRIDWHVWVHH-RQHV $SSOLHGWRWHVWLQJWKHWHUPދUHOLDELOLW\ތGRHVQRWUHIHUWRWUXVW-ZRUWKLQHVVDVLWGRHVLQHYHU\GD\(QJOLVK7KHFORVHVWPHDQLQJWKDW FDQEHDVVRFLDWHGZLWKދUHOLDEOHތLQWKHೳHOGRIWHVWLQJLVދFRQVLVW-HQWތ-RQHVFODLPVWKDWދ>D@UHOLDEOHWHVWLVFRQVLVWHQWLQWKDWLW SURGXFHVWKHVDPHRUVLPLODUUHVXOWRQUHSHDWHGXVHWKDWLVLWZRXOG UDQNRUGHUDJURXSRIWHVWWDNHUVLQQHDUO\WKHVDPHZD\ތ 1HYHUWKHOHVVUHOLDELOLW\DORQHLVQRWVXೲLFLHQWVLQFHHYHQLIDWHVW LVUHOLDEOHLWGRHVQRWQHFHVVDULO\PHDQWKDWLWLVDOVRDJRRGWHVWLH DV:LHUSXWVLWLWPD\QRWEHDFFXUDWHLQFRQYH\LQJDFRU-UHFWUHSUHVHQWDWLRQRIDWHVWWDNHUވVOHYHORIODQJXDJHNQRZOHGJHRU VNLOOV,QWKHVDPHZD\YDOLGLW\DORQHLVQRWVXೲLFLHQWIRUDJRRGWHVW ,QWKLVDUWLFOHWKHGDWDFROOHFWHGIURPWKHYHUVLRQRIRXU93, WHVWDUHDQDO\VHGWKURXJKDTXDQWLWDWLYHSHUVSHFWLYHWRೳQGHYLGHQFH RIWKHYDOLGLW\DQGUHOLDELOLW\RIWKHWHVW

2

Context, Methods and Participants

6LQFHWKH8QLYHUVLW\RI3DUPDKDVEHHQDFWLYHLQDUHVHDUFK SURJUDPPHGHGLFDWHGWRDFDGHPLFODQJXDJHWHDFKLQJDQGWHVWLQJ ,QSDUWLFXODUKLJKVFKRROVWXGHQWVވFRPSHWHQFHLQ,WDOLDQDVDQ/ IRUVWXG\SXUSRVHVKDYHEHHQLQYHVWLJDWHGVHH0H]]DGUL DE&RQFXUUHQWO\WZRRWKHUFORVHO\UHODWHG OLQHVRIUHVHDUFKKDYHEHHQGHYHORSHGWKHIRUPHUGLUHFWHGWRXQL- YHUVLW\VWXGHQWVZLWKDQDWLYHODQJXDJHGLೲHUHQWIURP,WDOLDQ0H]-]DGUL DQGWKHODWWHULQYROYLQJೳUVW\HDUXQLYHUVLW\VWXGHQWV UHJDUGOHVVRIWKHLUPRWKHUWRQJXH7KLVKDVEHHQGRQHZLWKWKHSXU-SRVHRIDVVHVVLQJWKHLULQLWLDOSUHSDUDWLRQDVUHTXLUHGE\,WDOLDQODZ 7KURXJKWKLVH[SHULHQFHDWHVWFDOOHGItalstudio ZDVFUHDWHG7KLV WHVWDYDLODEOHWRERWK/VFKRRODQGXQLYHUVLW\VWXGHQWVVHUYHGDV WKHIRXQGDWLRQIRUWKHWHVWGHVLJQHGIRUೳUVW\HDUXQLYHUVLW\VWXGHQWV 5HVHDUFKFRQGXFWHGLQ/HGXFDWLRQDOFRQWH[WVKDVDOORZHGXVWR UH೴HFWXSRQPHWKRGRORJLFDORSWLRQVLQWHDFKLQJDQGOHDUQLQJWKHODQ- JXDJHQHFHVVDU\IRUVWXG\SXUSRVHV7RGRVRWKHQDWXUHDQGWKHVSH-FLೳFIHDWXUHVRIDODQJXDJHXVHGIRUDFDGHPLFSXUSRVHVKDYHEHHQ VWXGLHGOHDGLQJWRSRVVLEOHPHWKRGRORJLFDOVROXWLRQVERWKWRDVVHVV

ODQJXDJHFRPSHWHQFHOHYHOVWKURXJKWKHItalstudioWHVWDQGWRGHVLJQ DQGPDQDJHVSHFLೳFFRXUVHV 7KLVOLQHRIUHVHDUFKKDVPDGHLWSRVVLEOHWRGHೳQHWKHFRPSHWHQFH ZLWKLQDQLQWHUQDWLRQDOIUDPHZRUNVXFKDVWKDWRIWKHVFDOHVRIWKH &RPPRQ(XURSHDQ)UDPHZRUNRI5HIHUHQFHIRU/DQJXDJHV&()5 PDNLQJWKHQHFHVVDU\DGMXVWPHQWVIRUDGLೲHUHQWFRQWH[WWKDWLV ODQJXDJHIRUVWXG\SXUSRVHV$QRWKHUDFKLHYHPHQWKDVEHHQWKHLVR-ODWLRQRIFHUWDLQHOHPHQWVXVHIXOLQUHGHVLJQLQJV\OODEXVHVVXFKDV WKHJUDPPDWLFDODQGV\QWDFWLFDOV\OODEXVHVDSSOLHGWRLQWHUQDWLRQDO JHQHUDOODQJXDJHWHVWVIRU,WDOLDQDVDIRUHLJQODQJXDJH0RUHRYHU DQHZV\OODEXVGHYRWHGWRVWXG\VNLOOVZLWKDORQJLWXGLQDOGHYHORS-PHQWEDVHGRQWKHOHYHOGHVFULSWRUVRIWKH&()5KDVEHHQFUHDWHG 7KHDQDO\VLVRIWKHQDWXUHRIWKHODQJXDJHIRUVWXG\SXUSRVHVKDV DOORZHGUHVHDUFKHUVWRGHೳQHWKHGLೲHUHQFHVEHWZHHQDODQJXDJH XVHGIRUJHQHUDODFDGHPLFSXUSRVHVDQGIRUVSHFLೳFDFDGHPLFSXU-SRVHVVHH0H]]DGUL 7KLVGLVWLQFWLRQLVQRWFHQWUDOWRWKH93, SURJUDPPHDVLWIRFXVHVPDLQO\RQJHQHUDODFDGHPLFODQJXDJHFRP-SHWHQFH,WLVDKLJKO\GHPDQGLQJFRQWH[WIURPDFRJQLWLYHSRLQWRI YLHZUHO\LQJPDLQO\RQFURVVFXUULFXODUVWXG\VNLOOV)RULQVWDQFH UHJDUGOHVVRIWKHGLVFLSOLQHVWXGLHGOLVWHQLQJVNLOOVPXVWEHGHYHO-RSHGWREHDEOHWRIROORZDOHFWXUHDVZHOODVWHFKQLTXHVWRWDNHDQG SURFHVVQRWHVRUWRPDQDJHSDUDWH[WXDOLQIRUPDWLRQ7KLVH[DPSOH KLJKOLJKWVWKHPRVWUHOHYDQWFURVVFXUULFXODUDFWLYLW\LQDQ,WDOLDQ DFDGHPLFFRQWH[WDVPRVWFRQWHQWLQIRUPDWLRQLVFRQYH\HGRUDOO\ WKURXJKOHFWXUHVHVSHFLDOO\LQWKHKXPDQLWLHV+RZZULWWHQWH[WV DUHPDQDJHGFDQDOVREHREVHUYHG5HDGLQJWHFKQLTXHVDUHFRPPRQ WRDOOGLVFLSOLQHVDOWKRXJKWHDFKHUVDQGVWXGHQWVVKRXOGEHDEOHWR FKRRVHDPRQJWKHPDFFRUGLQJWRWKHVFLHQWLೳFDUHDRIVWXG\RUWKH WHDFKHUVވPHWKRGRORJLFDOSUHIHUHQFHV$VUHJDUGVZULWWHQSURGXF-WLRQWKHDFWLYLWLHVLQYROYHGDUHPDLQO\ZULWLQJHVVD\VDQGUHSRUWV WDNLQJDQGSURFHVVLQJQRWHVZULWLQJVXPPDULHVFUHDWLQJFRQFHSW PDSVDQGYDULRXVPDWHULDOVWRDFFRPSDQ\RUDOSUHVHQWDWLRQV$OO RIWKHVHDUHFURVVFXUULFXODUDFWLYLWLHVWKDWPXVWQRWEHOLPLWHGWR DVLQJOHGLVFLSOLQDU\DUHD%OXH'XGOH\(YDQV6W-RKQ 7KHLVVXHGHVHUYHVDGHHSHUUH೴HFWLRQWKHH[WHQWRIZKLFKFDQQRW EHDGGUHVVHGLQWKLVVWXG\1RQHWKHOHVVWKHEULHIGHVFULSWLRQDERYH PD\KHOSWRXQGHUVWDQGKRZRXUUHVHDUFKJURXSKDVRSHUDWHGLQVWXG- \LQJWKHFRPPRQJURXQGRIDVHFRQGODQJXDJHIRUDFDGHPLFSXUSRV-HVLQWKLVFDVH,WDOLDQDQG93, 7KHUHVHDUFKJURXSZDVRೲLFLDOO\FUHDWHGLQDQGLVFRPSRVHG RIUHVHDUFKHUVIURPWKUHH,WDOLDQXQLYHUVLWLHV*HQRYD3DUPDDQG 8UELQR,WVDLPLVWRLQYHVWLJDWHLVVXHVUHODWHGWRWKHDFTXLVLWLRQRI WKH,WDOLDQODQJXDJHDVDPHDQVWRDFTXLUHNQRZOHGJHLQGLVFLSOLQDU\ ೳHOGVGLೲHUHQWIURPWKRVHUHODWHGWROLQJXLVWLFVDQGIRUHLJQODQJXDJH VWXGLHV7KLVDSSOLHVWRYDULRXVHGXFDWLRQDOFRQWH[WVDQGWHDFKLQJ RSWLRQVIURPWUDGLWLRQDOWRHOHDUQLQJRUEOHQGHGPRGHV

,Q2FWREHUWKHYHUVLRQRIWKHWHVWZDVDGPLQLVWHUHGWR VWXGHQWVDWWHQGLQJWKHೳUVW\HDURIWKHGHJUHHFRXUVHVLQ)RUHLJQ /DQJXDJHVDQGLQ&RPPXQLFDWLRQDQG&RQWHPSRUDU\0HGLDIRU &UHDWLYH,QGXVWULHVDWWKH8QLYHUVLW\RI3DUPD7KHVWXGHQWV VDWIRUWKHWHVWLQDWUDGLWLRQDOPDQQHUUHFHLYLQJWKHLUWHVWRQSDSHU 7KHWHVWVZHUHPDUNHGE\WKHWHDFKLQJVWDೲLQYROYHGXQGHUWKHVX-SHUYLVLRQRIUHVHDUFKHUVIURPWKH93,UHVHDUFKJURXS $WWKH8QLYHUVLW\RI8UELQRއ&DUOR%RވVWXGHQWVHQUROOHGLQWKH ೳUVW\HDURIHLJKWGLೲHUHQWGHJUHHFRXUVHVWRRNWKHWHVWLQ1RYHP-EHULQWKH205YHUVLRQ/DZ/DZIRU/DERXU&RQVXOWDQF\ DQG6DIHW\DW:RUN3ROLWLFDO6FLHQFHV(FRQRPLFVDQG*RYHUQ-PHQW6RFLRORJ\DQG6RFLDO6HUYLFHV)RUHLJQ/DQJXDJHVDQG &XOWXUHV&RPPXQLFDWLRQ6FLHQFHV+XPDQLWLHV&XOWXU-DO+HULWDJH6WXGLHVDQG3KLORVRSK\(GXFDWLRQDO6FLHQFHV 7KHVHVWXGHQWVKDGSUHYLRXVO\UHFHLYHGDQHPDLOPHVVDJHFRQWDLQ-LQJWKHLUDFFHVVFRGHVDQGLQIRUPDWLRQRQORJJLQJSURFHGXUHV$W ERWKXQLYHUVLWLHVVWXGHQWVKDGKDGWKHRSSRUWXQLW\WRWDNHDPRFN WHVWRQOLQHWREHFRPHDFTXDLQWHGZLWKWKHW\SHRIWHVWDQGWHVWWDVNV 7KH8QLYHUVLW\RI8UELQRWHVWLQJSURFHGXUHZDVVXSSRUWHGE\DVSH-FLDOLVHGFRPSDQ\LQWKHDGPLQLVWUDWLRQRIWKH205VFRULQJV\VWHP WHVWEHFDXVHRIWKHODUJHQXPEHURIVWXGHQWVLQYROYHG7KHFRPSD-Q\ZDVLQFKDUJHRIDFFUHGLWLQJWKHWHVWWDNHUVFROOHFWLQJGDWDDQG PDUNLQJWKHWHVWV ,QQHLWKHURIWKHFDVHVZHUHVWXGHQWVވQDWLYHODQJXDJHVRURWK- HULQIRUPDWLRQWDNHQLQWRDFFRXQWVLQFHWKHDLPRIWKHWHVWLVWRSUR-YLGHDQRYHUYLHZRIWKHVWXGHQWSRSXODWLRQRQDQLQGLYLGXDOEDVLV DQGDVDZKROH

2.1

Aims and Structure of the VPI Test

7KHPDLQDLPLVWRFUHDWHDWHVWEDVHGRQWKHOLQHRIUHVHDUFKGHYHO- RSHGWRPHHWWKHQHHGVRIWKHWKUHHXQLYHUVLWLHVLQYROYHG,QIDFWVFL-HQWLೳFDVSHFWVRIWKHWDVNDUHFRPELQHGZLWKWKHQHHGIRUWRROVWKDW DUHLPPHGLDWHO\DSSOLFDEOHLQDQDFDGHPLFFRQWH[W,QWUXWKWKHODFN RIJXLGHOLQHVIURPWKH0LQLVWU\RI5HVHDUFKDQG8QLYHUVLW\PDNHVLW UDWKHUGLೲLFXOWWRLPSOHPHQWDFWLRQVWKDWDUHYDOLGUHOLDEOHDQGDW WKHVDPHWLPHHFRQRPLFDOO\VXVWDLQDEOH 7KHೳHOGRIDSSOLFDWLRQRIWKHWHVWUHJDUGVFRPPXQLFDWLYHFRP-SHWHQFHLQ,WDOLDQIRUDFDGHPLFSXUSRVHV7KHUDWLRQDOHEHKLQGWKLV FKRLFHLVWKDWQRPDWWHUZKLFKGHJUHHFRXUVHVWXGHQWVDUHHQUROOHG LQWKHLUFRPPXQLFDWLYHFRPSHWHQFHPXVWEHHYDOXDWHGDQGLIWRR ZHDNVWUHQJWKHQHGWKURXJKDGGLWLRQDOOHDUQLQJRSSRUWXQLWLHVDQG WKURXJKVSHFLೳFUHPHGLDOZRUN7HDFKLQJVWDೲDWWKHGLೲHUHQWIDF- XOWLHVPXVWDOVREHSHUVXDGHGWRDFFHSWWKHWHVWLQJPHWKRGRORJ\EH-FDXVHWKHLUIUDPHRIPLQGDQGSURIHVVLRQDOVNLOOVPD\GLೲHUJUHDWO\

IURPWKDWRIWKHHGXFDWLRQDOOLQJXLVWVZKRFUHDWHGWKH93,WHVW7KH H[SHFWHGUHVXOWLVDKLJKHUOHYHORIDZDUHQHVVRIWKHNH\UROHSOD\HGE\ FRPPXQLFDWLRQVNLOOVIRUDOOVWXGHQWVUHJDUGOHVVRIWKHLUೳHOGRIVWXG\ 6WXGHQWVވRXWFRPHVPD\EHDWULVNLIWKHLUFRPPXQLFDWLYHFRPSH-WHQFHDUHQRWSURSHUO\GHYHORSHGDQGVXSSRUWHG 7KHSRVVLELOLW\RISURYLGLQJVWDWLVWLFDOGDWDWRKHOSLGHQWLI\WKH VNLOOVLQZKLFKDVWXGHQWQHHGVVXSSRUWKDVEHHQDGYDQWDJHRXVLQGH- YHORSLQJFORVHUFROODERUDWLRQEHWZHHQWKHUHVHDUFKJURXSDQGRWK-HUWHDFKLQJVWDೲ0RUHRYHUWKHTXDOLW\PDQDJHPHQWV\VWHPVRIDQ\ XQLYHUVLW\FDQHDVLO\ೳQGWKLVDSSURDFKFRQVLVWHQWZLWKWKHLUJRDOV 7KHWHVWLVGLYLGHGLQWRVHFWLRQVDVIROORZV • RUDOFRPSUHKHQVLRQPLQXWHV • ZULWWHQFRPSUHKHQVLRQPLQXWHV • XVHRIODQJXDJHOH[LFDODQGPRUSKRV\QWDFWLFDOFRPSHWHQFH GLVFRXUVHPDUNHUVSXQFWXDWLRQDFDGHPLFFRPPXQLFDWLRQUHJ-LVWHUVPLQXWHV 6WXGHQWVDUHJLYHQPLQXWHVWRFRPSOHWHWKHZKROHWHVW2UDOFRP- SUHKHQVLRQLVWHVWHGXVLQJDUHFRUGLQJSOD\HGWZLFHJLYLQJWHVWWDN-HUVWKHRSSRUWXQLW\WRGRZQVL]HWKHOLVWHQLQJWDVNV7KLVGHFLVLRQ ZDVPDGHWRSURYLGHDEHWWHUFKDQFHRIIROORZLQJWKHUDWKHUFRPSOH[ PHFKDQLVPLPSRVHGE\DVWUXFWXUHGOLVWHQLQJDFWLYLW\7KHW\SHRI DFWLYLW\LVQRWXVXDOLQHYHU\GD\DFDGHPLFOLVWHQLQJFRQWH[WVPDLQ-O\EDVHGRQQRWHWDNLQJGXULQJOHFWXUHV7KHWHVWSURYLGHVDFWLYLWLHV DQGTXHVWLRQVEDVHGRQERWKLQIHUHQWLDODQGQRQLQIHUHQWLDOLQIRUPD- WLRQ%HIRUHOLVWHQLQJWRWKHUHFRUGLQJWHVWWDNHUVDUHDVNHGWRDQD-O\VHDVHWRISLFWXUHVDLPHGWRDFWLYDWHWKHLUEDFNJURXQGNQRZOHGJH DQGJXLGHWKHFRPSUHKHQVLRQSURFHVV7KHUHDUHWKUHHWDVNVUHODWHG WRWKHOLVWHQLQJWH[W7KHೳUVWVWLPXODWHVJOREDOFRPSUHKHQVLRQZKLOH WKHVHFRQGIRFXVHVRQGHWDLOHGFRPSUHKHQVLRQ7KHODVWWDVNFDOOV IRUDV\QWKHWLFWH[WUHFRQVWUXFWLRQEDVHGRQDFRQFHSWPDS$OOWKH WHVWLWHPVDUHREMHFWLYH7KHOLVWHQLQJWH[WLVDOHFWXUHODVWLQJDERXW PLQXWHV,WGHDOVZLWKWRSLFVWKDWVWXGHQWVFDQKDQGOHUHJDUGOHVV RIWKHGHJUHHFRXUVHWKH\DUHHQUROOHGLQ 7KHUHDGLQJFRPSUHKHQVLRQLVEDVHGRQWZRGLೲHUHQWWH[WV7KH ೳUVWWH[WLVSURFHVVHGLQLWLDOO\WKURXJKDWDVNDLPHGDWDVVHVVLQJJORE-DOFRPSUHKHQVLRQHJFKRRVLQJWKHULJKWWLWOHIRUHDFKSDUDJUDSK WKHQWHVWWDNHUVGRDWDVNLQYROYLQJGHWDLOHGFRPSUHKHQVLRQZKLFK UHTXLUHVVWXG\VNLOOVVXFKDVPDQDJLQJDFRQFHSWPDS$IWHUWKHVH WZRFRPSUHKHQVLRQDFWLYLWLHVWHVWWDNHUVDUHUHTXLUHGWRDQVZHUD VHULHVRIPXOWLSOHFKRLFHLWHPVZLWKWKHDLPRIDVVHVVLQJFRPSUHKHQ-VLRQRIFRQFHSWVVRDVWRVWUHQJWKHQDQGEURDGHQZKDWKDVDOUHDG\ EHHQWHVWHGWKURXJKDFRQFHSWPDS)URPDFRJQLWLYHSRLQWRIYLHZ WKHLWHPVLQFOXGHGLQWKHWKLUGWDVNUHTXLUHPRUHFRPSOH[DQVZHUV DQGDUHPDLQO\LQIHUHQWLDO7KHVHFRQGSDUWRIWKHUHDGLQJFRPSUH-KHQVLRQVHFWLRQLVEDVHGRQDFOR]HWHVWLQLWVFODVVLFDOIRUPXVXDOO\

ZLWKDEODQNHYHU\VHYHQZRUGVWKHGHOHWHGLWHPVDUHUDQGRPL]HG LQDER[DWWKHHQGRIWKHWH[W 7KHWKLUGVHFWLRQLVGHGLFDWHGWRWKHXVHRIODQJXDJHDQGLQYROYHV GLೲHUHQWFRPPXQLFDWLYHFRPSHWHQFHVPRUSKRV\QWDFWLFOH[LFDOWH[- WXDODQGWKRVHUHODWHGWRSXQFWXDWLRQDQGWKHUHJLVWHUVXVHGLQDF-DGHPLFFRPPXQLFDWLRQ7KHೳQDOSDUWRIWKHWH[WXVHGIRUWKHFOR]H DFWLYLW\LVHPSOR\HGWRIDFHDWDVNWKDWDVVHVVHVOH[LFDOFRPSHWHQFH 7KLVLQYROYHVQRWRQO\WKHNQRZOHGJHRIWHUPVEXWDOVRWKHDELOLW\ WRKDQGOHZRUGVDFFRUGLQJWRGHULYDWLRQDQGWRLGHQWLI\DVVRFLDWLRQV V\QRQ\PVRSSRVLWHVHWFDPRQJZRUGVLQFOXGLQJVSHFLೳFVFLHQWLI-LFDUHDVRUKLJKUHJLVWHUDQGORZIUHTXHQF\WHUPV$IWHUWKLVSKDVH WHVWWDNHUVDUHUHTXLUHGWRೳOOLQWHQLWHPVZKHUHVSHFLೳFPRUSKR V\QWDFWLFVWUXFWXUHVKDYHEHHQGHOHWHG7KHJRDOVLVWRUHDFKERWK PRUSKRV\QWDFWLFDFFXUDF\DQGFRPPXQLFDWLYHHೲLFDF\,QWKLVVHF- WLRQWH[WXDOFRPSHWHQFHLVDVVHVVHGWKURXJKWKHDELOLW\WRXVHGLV-FRXUVHPDUNHUV7KHFRPSHWHQFHUHODWHGWRWKHXVHRISXQFWXDWLRQ LVDVVHVVHGIURPDORJLFDOUDWKHUWKDQDVW\OLVWLFSHUVSHFWLYHWRDV- VXUHFRKHUHQFHDQGFRKHVLRQLQWKHWH[W7KHODVWDFWLYLW\LQWKLVVHF-WLRQLVGHGLFDWHGWRDFDGHPLFFRPPXQLFDWLRQ7KHJRDOLVWRDVVHVV WHVWWDNHUVވFRPSHWHQFHLQPDQDJLQJUHJLVWHUVWKDWDUHDSSURSULDWH WRDFDGHPLFFRPPXQLFDWLYHFRQWH[WV

2.2

Online and OMR Paper Versions

7KHVWUXFWXUHRIWKHRULJLQDOSDSHUEDVHGWHVWIRUFODVVURRPXVHGH-VFULEHGVRIDUZDVODWHUPRGLೳHGWRFUHDWHWZRGLೲHUHQWYHUVLRQVRQH WHVWLQGLJLWDOIRUPDWWREHDGPLQLVWHUHGRQOLQHDQGRQHSHQWRSD-SHUWREHVFRUHGXVLQJRSWLFDOPDUNUHFRJQLWLRQ2059DULDWLRQV ZHUHPLQLPDOLQWKHೳUVWFDVHDQGRQO\LQYROYHGWKHVWUXFWXUHRIVRPH RIWKHTXHVWLRQIRUPVZKLOHLQWKHVHFRQGTXHVWLRQVEHFDPHPRUH RUOHVVGLೲLFXOWUHTXLULQJGLೲHUHQWVROXWLRQVWUDWHJLHV7KHWLPHDO-ORZHGUHPDLQHGWKHVDPH 6LQFHWKHೳUVWYHUVLRQRIWKHWHVWZDVDGPLQLVWHUHGLQDGLVWDQFH OHDUQLQJHQYLURQPHQW3ROLWLFDO(FRQRPLFDQG*RYHUQPHQW6FLHQFH LVDQRQOLQHGHJUHHDQGEHFDXVHDVDPSOHWHVWKDGWREHSURYLGHGRQ WKH8QLYHUVLW\ZHEVLWHLWZDVQHFHVVDU\WRPRGLI\VRPHRILWVVHF-WLRQVWRPDNHWKHPVXLWDEOHIRUWKHFRPSXWHULVHGIRUPDW2 _The col- RXUIXOOD\RXWRIWKHRULJLQDOYHUVLRQZDVSUHVHUYHGDVPXFKDVSRV-VLEOHLQIDFWLQWKHGLJLWDOYHUVLRQLOOXVWUDWLRQVDQGFRQFHSWPDSV ZHUHNHSW0RUHRYHUWRPDNHLWPRUHXVHUIULHQGO\WKHQDYLJDWLRQ PHQXDQGUHPDLQLQJWLPHZHUHDOZD\VYLVLEOH7KHOLVWHQLQJSDUW 2ݲ7KHGLJLWDOYHUVLRQZDVHQWLUHO\HODERUDWHGE\6LPRQH7RUVDQLRIWKH8QLYHUVLW\RI *HQRD

ZDVDFWLYDWHGZLWKDFOLFNDQGPXOWLSOHFKRLFHTXHVWLRQVRUWUXH IDOVHTXHVWLRQVZHUHOHIWLQWKHLURULJLQDOIRUP7KHFOR]HWHVWLQ ZKLFKWKLUW\PLVVLQJWHUPVPXVWEHFRUUHFWO\LQVHUWHGLQWRDWH[W ZDVWUDQVIRUPHGLQWRGUDJDQGGURSIRUPDW7KHVHFWLRQUHJDUGLQJ PRUSKRV\QWD[UHTXLULQJVHQWHQFHFRPSOHWLRQFRXOGEHFRPSOHWHG E\W\SLQJWKHUHVSRQVHGLUHFWO\LQWRWKHVSDFHSURYLGHG)RUWHFKQL-FDOUHDVRQVWKHSDUWRIWKHRQOLQHWHVWGHDOLQJZLWKSXQFWXDWLRQZDV WKHPRVWVXEVWDQWLDOO\PRGLೳHGLQWKHQHZGLJLWDOIRUPDWEHFDXVHLW ZDVVLPSOLೳHGWRVRPHH[WHQW7HVWWDNHUVZHUHQRORQJHUDVNHGWR LGHQWLI\ೳYHHUURUVLQWKHZKROHWH[WVLQFHSUHFLVHSRLQWVZHUHLQGL-FDWHGLQWKHWH[WZKHUHSXQFWXDWLRQZDVWREHFRUUHFWHGLIQHFHVVDU\ 7KLVGHFLVLRQZDVPDGHLQRUGHUWRDYRLGWKHLQVHUWLRQRIWH[WXDOGD- WDೳHOGV7KHSDSHUYHUVLRQRIWKHWHVWZLWKWKHDXWRPDWLF205VFRU-LQJV\VWHPZDVVXEVWDQWLDOO\WUDQVIRUPHGEHFDXVHWKHHQWLUHWHVWKDG WREHPXOWLSOHFKRLFH3,QWKHWZRYHUVLRQVRSWLRQ$DQGRSWLRQ%LQ ZKLFKSRVVLEOHUHVSRQVHVDUHUDQGRPL]HGGLೲHUHQWO\IRUHDFKTXHV-WLRQWKHWZRFRQFHSWPDSVRQHIRUOLVWHQLQJFRPSUHKHQVLRQDQGRQH IRUUHDGLQJFRPSUHKHQVLRQZHUHPRGLೳHGWRVXLWWKHPXOWLSOHFKRLFH IRUPDWRIWKHWHVW,QWKHFDVHRIOLVWHQLQJWKHVHPRGLೳFDWLRQVRQO\ LQYROYHGDORVVRIWKHFRQFHSWPDSVWUXFWXUHEXWSUHVHUYHGWKHVDPH QXPEHURIUHVSRQVHRSWLRQVLQWKHUHDGLQJWKHWHVWWDNHULVDVNHGWR FKRRVHDPRQJWKUHHUHVSRQVHVUDWKHUWKDQWKHRULJLQDOೳIWHHQ7KLV PRGLೳFDWLRQXQGRXEWHGO\PDNHVWKHH[HUFLVHHDVLHULQERWKFDVHV WKHVXPPDUL]LQJIXQFWLRQRIWKHOD\RXWLVORVW 7KHWDVNLQYROYLQJWKHLGHQWLೳFDWLRQRISDUDJUDSKWLWOHVLVPRUH ODERXUVRPHDQGWLPHFRQVXPLQJEHFDXVHWKHRSWLRQVDUHQRORQJHU SURYLGHGDWWKHERWWRPRIWKHSDJHEXWUHSURSRVHGLQOLVWIRUPIRU HDFKTXHVWLRQ7KHVDPHDGMXVWPHQWZDVDSSOLHGWRWKHFOR]HWHVWLQ ZKLFKWKHGHOHWHGLWHPVDUHQRWJLYHQDWWKHHQGRIWKHWH[WEXWLQ PXOWLSOHFKRLFHIRUPDWWKUHHRSWLRQV 1RWHWKDWPRUSKRV\QWD[DQGSXQFWXDWLRQDUHWKHWZRDUHDVRIWKH WHVWWKDWZHUHPRGLೳHGPRVWVXEVWDQWLDOO\0RUSKRV\QWDFWLFLWHPV ZKLFKRULJLQDOO\UHTXLUHGWKHWHVWWDNHUWRೳOOLQDEODQNZLWKQR SURPSWLQJDUHQRZLQPXOWLSOHFKRLFHIRUPDWWKUHHFKRLFHVPDN-LQJWKHWDVNHDVLHU 2QWKHFRQWUDU\WKHSXQFWXDWLRQVHFWLRQZKLFKRULJLQDOO\UHTXLUHG LGHQWLೳFDWLRQDQGFRUUHFWLRQRIPLVSXQFWXDWLRQLQWKHHQWLUHWH[WQRZ LQYROYHVWKHVHOHFWLRQRIFRUUHFWO\SXQFWXDWHGSKUDVHVDPRQJWKUHH RSWLRQVH[WUDFWHGIURPDJLYHQWH[W7KHWDVNEHFRPHVQRWDEO\PRUH EXUGHQVRPHDQGWLPHFRQVXPLQJLQFUHDVLQJFRJQLWLYHORDG 3ݲ7KHYHUVLRQZLWKWKHDXWRPDWLF205VFRULQJV\VWHPZDVHODERUDWHGE\*LRYDQQD &DUORQLDQG)ORUD6LVWLRIWKH8QLYHUVLW\RI8UELQR

3

Results

,QWKHIROORZLQJSDJHVZHSUHVHQWWKHUHVXOWVRIDVWDWLVWLFDODQDO\-VLVFRQGXFWHGWRVWXG\WKH93,WHVWOHYHOVRIUHOLDELOLW\DQGYDOLGLW\ VHH0H]]DGUL 'XHWRWKHGLೲHUHQFHVLQWKHZD\WKHWHVWZDVDGPLQLVWHUHGDW WKHWZRXQLYHUVLWLHVDGHFLVLRQZDVPDGHWRFDUU\RXWWZRVHSDUDWH DQDO\VHVRQWKHVDPHIHDWXUHV 7KHDQDO\VHVZHUHFRQGXFWHGRQೳYHLWHPVOLVWHQLQJUHDGLQJ FOR]HXVHRIODQJXDJHDQGRYHUDOOVFRUH7KHೳUVWIRXULWHPVZHUHQRU-PDOLVHGWRDQGWKHODVWWR7KHUHVXOWVFRQFHUQLQJWKHGLೲHUHQW DFWLYLWLHVLQWKHOLVWHQLQJVHFWLRQDQGLQWKHXVHRIODQJXDJHVHFWLRQ ZHUHJURXSHGWRFUHDWHWZRRQHLWHPFRPSRQHQWV7KHFOR]HDFWLYLW\ ZDVNHSWVHSDUDWHIURPWKHWKUHHUHDGLQJFRPSUHKHQVLRQWDVNVWL-WOHPDWFKLQJFRQFHSWPDSDQGIXUWKHUTXHVWLRQVIRUWKHೳUVWWH[W 7KLVZDVGRQHEHFDXVHRISUHYLRXVUHVXOWVVHH0H]]DGUL DQGWKDWFOHDUO\VKRZHGDVWURQJFRUUHODWLRQEHWZHHQWKHFOR]H WHVWDQGRWKHUFRPSRQHQWVVXFKDVWKHXVHRIODQJXDJHRUWKHZULW- WHQSURGXFWLRQDQGQRFRUUHODWLRQZLWKWKHOLVWHQLQJDQGWKHUHDG-LQJVHFWLRQRIWKH,WDOVWXGLRWHVWLQYHVWLJDWHGDWWKHWLPH$VVXPLQJ WKDWVLPLODUUHVXOWVFRXOGDSSO\WRWKH93,WHVWWKHೳUVWDQDO\VLVZDV FRQGXFWHGLQWKHVDPHZD\OHDYLQJRWKHUSRVVLEOHVROXWLRQVWRDODWHU VWDJHLIQHHGHG7KHDQDO\VLVZDVWKHQFDUULHGRXWRQMXVWIRXULWHPV SOXVWKHRYHUDOOVFRUHLWHPHYHQLIWKLVPHDQWWDNLQJDULVNRISD\LQJ DFRVWLQWHUPVRIUHOLDELOLW\PHDVXUHGWKURXJK&URQEDFKވVDOSKDDV GLVFXVVHGEHORZGXHWRWKHUHGXFHGQXPEHURILWHPV

7KHDQDO\VLV[tables 1-2]VKRZVWKDWWKHWHVWVWUXFWXUHSUHVHQWVDJRRG

GHJUHHRIFRUUHODWLRQ$OOWKHYDULDEOHVDUHFORVHO\FRUUHODWHGZLWKWKH RYHUDOOVFRUHVDQGZLWKHDFKRWKHU7KLVDSSHDUVWRFRQೳUPWKDWLWZDV VRXQGGHFLVLRQWRJURXSWKHWZROLVWHQLQJLWHPVLQWRDVLQJOHLWHPDQG WKHWZRUHDGLQJLWHPVLQWRDQRWKHUVLQJOHLWHP7KHGHJUHHRIVLJQLೳ- FDQFHRIWKHFRUUHODWLRQZLWKRYHUDOOVFRUHVUHPDLQVIDLUO\KLJKUDQJ-LQJIURPXVHRIODQJXDJHWRUHDGLQJIRU3DUPDUHVXOWV7KH GDWDIURP8UELQRUDQJHIURPOLVWHQLQJWRXVHRIODQJXDJH 7KLVVHHPVWRWHVWLI\WRJRRGFRKHUHQFHRIWKHWHVWFRQVWUXFWDVDZKROH 7KHZHDNHVWFRUUHODWLRQVRFFXUEHWZHHQUHDGLQJDQGFOR]H DQGOLVWHQLQJDQGXVHRIODQJXDJHZLWKWKHVDPHUHVXOWLQ3DU-PDDQGEHWZHHQUHDGLQJDQGXVHRIODQJXDJHDQGOLVWHQLQJDQG FOR]HLQ8UELQR,IZHORRNDWWKHVHFRQGZHDNHVWFRUUHODWLRQV LQERWKJURXSVZHQRWLFHWRWDOV\PPHWU\VHHGDWDLQEROGLQWDEOH DQG,WLVZRUWKZKLOHQRWLFLQJWKHV\PPHWU\EHWZHHQWKHUHVXOWV REWDLQHGWKDWFRUUHODWHOLVWHQLQJZLWKUHDGLQJDQGFOR]HZLWK XVHRIODQJXDJHLQ3DUPDDQGLQ8UELQRDQGUHVSHF-WLYHO\7KLVVXJJHVWVLPSRUWDQWIHDWXUHVWKDWZLOOEHRXWOLQHGEHORZ ZKHQWKHGLೲHUHQWGHJUHHVRIFRPSOH[LW\LQWKHODQJXDJHFRPSHWHQF-HVLQYROYHGLQWKHWHVWDUHGLVFXVVHG

Table 1 Correlations – Parma

Listening Reading Use of language Cloze Overall scores Listening 1 ,499** _,356** _,420** _,728** Reading ,499** _{1 ,380}** _,356** _,787** Use of language ,356** _,380** _{1 ,509}** _,718** Cloze ,420** _,356** _,509** _{1 ,763}** Overall scores ,728** _,787** _,718** _,763** ₁

**. Correlation is significant at the 0.01 level (2-tailed).

Table 2 Correlations – Urbino

Listening Reading Use of language Cloze Overall scores Listening 1 ,419** _,286** _,265** _,638** Reading ,419** _{1 ,267}** _,278** _,732** Use of language ,286** _,267** _{1 ,492}** _,754** Cloze ,265** _,278** _,492** _{1 ,691}** Overall scores ,638** _,732** _,754** _,691** ₁

**. Correlation is significant at the 0.01 level (2-tailed).

$IDFWRUDQDO\VLV[tables 3-4]

ZDVFRQGXFWHGWRDOORZWKHLQWHUQDOVWUXF-WXUHRIWKHVHWRIYDULDEOHVWRHPHUJHUHGXFLQJLWWRWZRIDFWRUV7KLV PDNHVLWSRVVLEOHWRFKHFNZKHWKHUDQGWRZKDWH[WHQWRXUWHVWVXF-FHHGVLQPHDVXULQJWKHGLೲHUHQWOLQJXLVWLFFRPSHWHQFHV3DUPDވV VHWRIGDWDVKRZVWKDWZLWK(LJHQYDOXHVH[WUDFWLRQ(LJHQYDOXHV! WKHWRWDOYDULDQFHH[SODLQHGLV,IDVHFRQGIDFWRULVDGGHG WKHWRWDOYDULDQFHH[SODLQHGUHDFKHV,Q8UELQRZLWK(LJHQ-YDOXHVH[WUDFWLRQ(LJHQYDOXHV!WKHWRWDOYDULDQFHH[SODLQHGLV %XWLIDVHFRQGIDFWRULVDGGHGWKHWRWDOYDULDQFHH[SODLQHG UHDFKHV

Table 3 Total variance explained- Parma

Component Initial Eigenvalues Extraction Sums of Squared Loadings

Total % of Variance Cumulative % Total % of Variance Cumulative %

1 2,260 56,505 56,505 2,260 56,505 56,505

2 ,749 18,721 75,227 ,749 18,721 75,227

3 ,540 13,497 88,724

4 ,451 11,276 100,000

Table 4 Total variance explained – Urbino

Component Initial Eigenvalues Extraction Sums of Squared Loadings

Total % of Variance Cumulative % Total % of Variance Cumulative %

1 2,005 50,133 50,133 2,005 50,133 50,133

2 ,906 22,655 72,788 ,906 22,655 72,788

3 ,584 14,604 87,392

4 ,504 12,608 100,000

Extraction Method: Principal Component Analysis.

$IWHURUWKRJRQDOURWDWLRQ9DULPD[[table 5]

WKHWZRIDFWRUVDUHFRQ-ೳUPHGWREHIRUPHGDVIROORZVWKHೳUVWE\WKHUHDGLQJDQGOLVWHQLQJ YDULDEOHVDQGWKHVHFRQGE\WKHFOR]HDQGXVHRIODQJXDJHYDULDEOHV Table 5 Component Matrix – Parma and Urbino

Parma Component Urbino Component

1 2 1 2

Use of language ,852 ,200 Use of language ,845 ,171

Cloze ,824 ,253 Cloze ,850 ,160

Reading ,199 ,850 Reading ,157 ,829

Listening ,253 ,821 Listening ,166 ,825

Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. $QDQDO\VLVRIWKHGHJUHHRIUHOLDELOLW\RIWKHWHVWZDVVXEVHTXHQWO\ FRQGXFWHGPHDVXULQJ&URQEDFKވV$OSKD7DEOHVDQGVKRZ3DUPD UHVXOWV7KHYDOXHREWDLQHGRQWKH3DUPDWHVWVLVZKLFKLV UDWKHUKLJKIRUDLWHPFRQVWUXFW,WLVZRUWKQRWLFLQJWKDWLIDQ\RI WKHLWHPVLVGHOHWHGWKHYDOXHGHFUHDVHVWHVWLI\LQJWRDKLJKOHYHORI LQWHUQDOFRKHUHQFHRIWKHWHVW

Table 6 Reliability Statistics – Parma

Cronbach’s Alpha No. of Items

,727 4

Table 7 Item-Total Statistics – Parma

Scale mean if item deleted Scale variance if item deleted Corrected item-total correlation Cronbach’s alpha if item deleted Listening 21,6359 16,607 ,322 ,658 Reading 23,1878 12,630 ,302 ,688 Cloze 20,5849 14,189 ,330 ,659 Use of language 23,0510 16,477 ,310 ,666

7KHVDPHDQDO\VLVRIWKHGHJUHHRIUHOLDELOLW\RIWKHRQOLQHYHUVLRQRI WKHWHVWFRQGXFWHGRQWKHVHWRIGDWDFROOHFWHGYDOLGVXEMHFWVDW WKH8QLYHUVLW\RI8UELQRVKRZVDUHODWLYHO\ORZHUOHYHORIUHOLDELOLW\ DV&URQEDFKވV$OSKDLV,WLVZRUWKUHPHPEHULQJWKDWWKHLWHPV ZHUHRQO\IRXU[Tables 6 to 9]

Table 8 Reliability Statistics - Urbino OTTO

Cronbach’s Alpha No. of items

,647 4

Table 9 Item-Total Statistics – Urbino NOVE

Scale mean if item deleted Scale variance if item deleted Corrected item-total correlation Cronbach’s alpha if item deleted Listening 22,8820 16,090 ,441 ,589 Reading 23,8288 12,148 ,406 ,608 Cloze 21,7646 14,808 ,479 ,556 Use of language 23,3684 11,888 ,454 ,565 $VIRUWKH3DUPDVHWRIGDWDLIDQ\RIWKHLWHPVLVGHOHWHGWKHYDO-XHGHFUHDVHV $IWHUDQDO\VLQJWKHGLೲHUHQFHVLQUHVXOWVEHWZHHQ3DUPDDQG8UEL-QRVHWVRIGDWDDIXUWKHULQYHVWLJDWLRQZDVFRQGXFWHGWRWU\WRೳQGRXW ZKHWKHUWKHUHZDVDQ\HYLGHQFHWKDWFRXOGWHVWLI\WRVXEVWDQWLDOGLೲHU-HQFHVLQWKHFRQVWUXFWRULQWKHIRUPDWRIWKHWZRYHUVLRQVRIWKHWHVWV :HVHSDUDWHGWKHGLೲHUHQWGHJUHHSURJUDPPHVSURYLGLQJWKH 8UELQRVHWRIGDWDWRFUHDWHDJURXSPDGHXSRIWZRGHJUHHSUR-JUDPPHVVLPLODUWRWKRVHRIWKH3DUPDVHWVSHFLೳFDOO\FRQVLVWLQJ RI)RUHLJQ/DQJXDJHVDQG&XOWXUHVVXEMHFWVDQG&RPPXQLFD- WLRQ6FLHQFHVVXEMHFWV$OWKRXJKZHPHDVXUHGWKHVDPHSDUDP- HWHUVQRWKLQJZDVDEOHWRDOLJQWKHUHVXOWVRIWKHVHWVRIWKHWZRXQL-YHUVLWLHV2QWKHFRQWUDU\IRUH[DPSOH&URQEDFKވV$OSKDZDV FRPSDUHGWRYHULೳHGLQWKHDQDO\VLVFRQGXFWHGRQDOOWKHGHJUHH SURJUDPPHVWRJHWKHU:HWKHQFUHDWHGDJURXSWKDWLQFOXGHGDOOWKH GHJUHHSURJUDPPHVH[FHSWIRU)RUHLJQ/DQJXDJHVDQG&XOWXUHV &URQEDFKވV$OSKDLQFUHDVHGWRLQOLQHZLWKWKHUHVXOWVFRUHG ZKHQPHDVXULQJDOOWKHGHJUHHSURJUDPPHVWRJHWKHU )XUWKHUPRUHLQDOOFDVHVDIWHURUWKRJRQDOURWDWLRQ9DULPD[WKH WZRIDFWRUVZHUHFRQೳUPHGWREHIRUPHGE\WKHUHDGLQJDQGOLVWHQLQJ YDULDEOHVDQGE\WKHFOR]HDQGXVHRIODQJXDJHYDULDEOHV )LQDOO\ZHGHWHUPLQHGWKDWLQRUGHUWREHWWHULQWHUSUHWWKHGD-WDDYDLODEOHPRUHYDULDEOHVVKRXOGEHWDNHQLQWRDFFRXQWVXFKDV WLPHGLVWULEXWLRQLQWKHWZRGLೲHUHQWYHUVLRQVDQ\VOLJKWGLೲHUHQFH LQWKHZD\WHVWLQJDFWLYLWLHVDUHSUHVHQWHGGLೲHUHQFHVLQTXDOLW\RI

FRPPXQLFDWLRQFKDQQHOVPRUHSUHFLVHLQIRUPDWLRQRQWKHWHVWWDN-HUVLQVWLWXWLRQDOSUHVVXUHRQWKH93,WHVWWKDWPD\GLೲHUDWWKHWZR XQLYHUVLWLHVRYHUDOOSHUIRUPDQFHRIVWXGHQWVEHIRUHHQWHULQJXQLYHU-VLW\JHQGHULVVXHVHWF $PRUHH[WHQVLYHDQDO\VLVDQGIXUWKHUUH೴HFWLRQDUHOHIWWRDODWHU VWDJHRIWKLVUHVHDUFKEHLQJEH\RQGWKHDLPRIWKLVDUWLFOH

4

Discussion and Conclusions

7KHVWDWLVWLFDODQDO\VLVRIWKHYHUVLRQRIWKH93, WHVWVKRZVD VLJQLೳFDQWGHJUHHRIUHOLDELOLW\:HH[SHFWHGWKHVFRUHVWRSURYH WKHYDOLGLW\RIWKHWHVWFRQVWUXFWGHYHORSHGRQWKHEDVLVRIH[WHQG- HGPHWKRGRORJLFDOUH೴HFWLRQVDQGRSWLRQVHJZKLFKW\SHVRIWHVW-LQJWHFKQLTXHVDQGWH[WW\SRORJ\WRLQFOXGH7KHVWDWLVWLFDODQDO\VLV VKRZVWKDWWKHWHVWLVLQIDFWFRQVLVWHQWDVWKHYDULDEOHVWKDWZHFRQ-VLGHUHGVKRXOGEHORQJWRWKHVDPHJURXSDUHLQIDFWFRQVLVWHQWZLWKD FRPPRQODWHQWIDFWRU:HFODLPWKDWWKLVUHVXOWFRQೳUPVDQLQWHUQDO FRKHUHQFHEHWZHHQWKHZD\WKHRU\KDVEHHQDSSOLHGWRRSHUDWLRQDO-L]HWKHFRQVWUXFWLQYDULDEOHVDQGWKHDFWXDOUHODWLRQVKLSVEHWZHHQ WKHWHVWYDULDEOHV7KHVHೳQGLQJVDUHRISDUDPRXQWLPSRUWDQFHLI ZHVKDUH%DFKPDQDQG3DOPHUވVRSLQLRQWKDWދXQOHVVWHVW VFRUHVDUHUHODWLYHO\FRQVLVWHQWWKH\FDQQRWSURYLGHXVZLWKDQ\LQ-IRUPDWLRQDWDOODERXWWKHDELOLW\ZHZDQWWRPHDVXUHތ 7KLVTXDQWLWDWLYHDQDO\VLVLVDFFRPSDQLHGE\UH೴HFWLRQVRQWKH FRPPXQLFDWLYHVNLOOVVSHFLೳFDOO\WKRVHUHTXLUHGIRUDFDGHPLFSXU-SRVHVZKLFKVWXGHQWVVKRXOGSRVVHVVZKHQWKH\HQWHUXQLYHUVLW\ +LJKHUHGXFDWLRQLQVWLWXWLRQVDFURVV(XURSHKDYHUHFHQWO\EHFRPH LQYROYHGLQDSURFHVVRIUHGHVLJQLQJWKHLUFXUULFXODWKURXJKWKHDQDO-\VLVDQGUHLQWHUSUHWDWLRQRIWKHGHPDQGVRIWRGD\ވVMREPDUNHW7KH PRVWSUHVVLQJLVVXHVHHPVWREHWKHSURPRWLRQRI6RIW6NLOOVWREH XVHGLQZRUNFRQWH[WV,QRecommendation of the European Parlia-ment and of Council of 18 December 2016 on Key Competence for life-long learning (2006/962/EC4 FRPPXQLFDWLRQLVWKHೳUVWNH\FRPSH-WHQFHLQWKH5HIHUHQFH)UDPHZRUNVHWދ,QWKHFRQWH[WRI(XURSHވV PXOWLFXOWXUDODQGPXOWLOLQJXDOVRFLHWLHVLWLVUHFRJQLVHGWKDW>ޕ@DELO- LW\WRFRPPXQLFDWHLQDQRೲLFLDOODQJXDJHLVDSUHFRQGLWLRQIRUHQ-VXULQJIXOOSDUWLFLSDWLRQRIWKHLQGLYLGXDOLQVRFLHW\ތ)RUVHYHUDO \HDUVWKH(XURSHDQ8QLRQKDVVXSSRUWHGXQLYHUVLW\FRPPLWPHQWLQ LGHQWLI\LQJWKHEHVWSUDFWLFHVIRUWKHSURPRWLRQRI6RIW6NLOOVDVLWLV FRPPRQO\UHFRJQLVHGWKDWWKH\DUHIXQGDPHQWDOERWKIRUDFDGHPLF VXFFHVVDQGIXWXUHHPSOR\DELOLW\RIJUDGXDWHV 4ݲ,Q http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex:32006H0962 

2XUWHVWDLPVWRGHWHUPLQHZKHWKHUೳUVW\HDUXQLYHUVLW\VWXGHQWV DOUHDG\SRVVHVVHVVHQWLDOFRPPXQLFDWLYHVNLOOVLQ,WDOLDQIRUXVHLQ DFDGHPLFFRQWH[WVDQGLIVRWRZKDWH[WHQW$FFRUGLQJWRWKHGHೳQL-WLRQJLYHQE\WKH(XURSHDQ5HFRPPHQGDWLRQ &RPPXQLFDWLRQLQWKHPRWKHUWRQJXHLVWKHDELOLW\WRH[SUHVVDQG LQWHUSUHWFRQFHSWVWKRXJKWVIHHOLQJVIDFWVDQGRSLQLRQVLQERWK RUDODQGZULWWHQIRUPOLVWHQLQJVSHDNLQJUHDGLQJDQGZULWLQJ DQGWRLQWHUDFWOLQJXLVWLFDOO\LQDQDSSURSULDWHDQGFUHDWLYHZD\ LQDIXOOUDQJHRIVRFLHWDODQGFXOWXUDOFRQWH[WVLQHGXFDWLRQDQG WUDLQLQJZRUNKRPHDQGOHLVXUH 0RUHRYHU &RPPXQLFDWLRQLQWKHPRWKHUWRQJXHUHTXLUHVDQLQGLYLGXDOWR KDYHNQRZOHGJHRIYRFDEXODU\IXQFWLRQDOJUDPPDUDQGWKHIXQF-WLRQVRIODQJXDJH,WLQFOXGHVDQDZDUHQHVVRIWKHPDLQW\SHVRI YHUEDOLQWHUDFWLRQDUDQJHRIOLWHUDU\DQGQRQOLWHUDU\WH[WVWKH PDLQIHDWXUHVRIGLೲHUHQWVW\OHVDQGUHJLVWHUVRIODQJXDJHDQGWKH YDULDELOLW\RIODQJXDJHDQGFRPPXQLFDWLRQLQGLೲHUHQWFRQWH[WV 7KHWKUHHVHFWLRQVRIRXUWHVWRUDOFRPSUHKHQVLRQZULWWHQFRPSUH-KHQVLRQDQGXVHRIODQJXDJHZHUHFRQFHLYHGWRFROOHFWGDWDRQWKH VXEFRPSHWHQFHVPHQWLRQHGDERYH7KHWHVWKDVVRIDUEHHQDGPLQ- LVWHUHGWRVWXGHQWVDWWHQGLQJGHJUHHFRXUVHVLQWKHKXPDQLWLHV(G-XFDWLRQ3KLORVRSK\DQG+XPDQLWLHV)RUHLJQ/DQJXDJHV3ROLWLFDO (FRQRPLFDQG*RYHUQPHQW6FLHQFH&RPPXQLFDWLRQDQG0HGLDEXW LWFRXOGEHH[WHQGHGWRLQFOXGHDOOGHJUHHSURJUDPPHVFRQVLGHULQJ WKHVWUDWHJLFDQGFURVVFXUULFXODUUROHRIFRPPXQLFDWLYHVNLOOVDWWKH WLPHRIXQLYHUVLW\HQUROPHQW $JUDGXDOLPSOHPHQWDWLRQRIWKHWHVWLQGLೲHUHQWDUHDVRI+LJK- HU(GXFDWLRQFRXOGIDFLOLWDWHDPRUHFRKHUHQWVWUDWHJ\IRUWKHDV- VHVVPHQWRIVWXGHQWVވLQLWLDOSUHSDUDWLRQ$WWKHVDPHWLPHUHPHGL-DOVWUDWHJLHVWRHQVXUHVXSSRUWDQGDVVLVWDQFHDIWHUWDNLQJWKHWHVW DQGKDYLQJHYDOXDWHGSRVVLEOHZHDNQHVVHVFRXOGSURPSWDVHFRQG OLQHRIUHVHDUFKWKDWLVLQGHHGDOUHDG\EHLQJH[SORUHGDQGGHYHORSHG ERWKDWWKH8QLYHUVLW\RI3DUPDDQG8UELQR

Bibliography

Alderson, Charles J. (1991). Language Testing in the 1990s. London: Modern English Publication.

Alderson, Charles J. (1994). “The State of Language Testing in the 1990s”. Huhta, Ari; Sajavaara, Kari; Takala, Sauli (eds), Language Testing: New

Open-ings. Jyväskyla: University of Jyväskyla, 1-19.

Alderson, Charles J.; Banerjee, Jayanti (2001). “Language Testing and Assess-ment” Language Teaching, 34, pt. 1, 213-36.

Alderson, Charles J.; Banerjee, Jayanti (2002). “Language testing and assess-ment”. Language Teaching, 35, pt. 2, 79-113.

Bachman, Lyle F. (2000). “Modern Testing at the Turn of the Century: Assuring that We Count Counts”. Language Testing, 17(1), 1-42.

Bachman, Lyle F.; Eignor, Daniel R. (1997). “Recent advances in quantitative test analysis”. Clapham, Caroline; Corson, David (eds), Language Testing and

As-sessment. Vol. 7 of Encyclopedia of Language and Education. Dordrecht:

Klu-wer Academic Publishers, 227-42.

Bachman, Lyle F.; Palmer, Adrian S. (2010). Language Assessment in Practice. Oxford: Oxford University Press.

Barni, Monica (2005). “Etica e valutazione delle competenze in L2”. Vedovel-li, Massimo (ed.), Manuale della certificazione dell’italiano L2. Roma: Ca-rocci, 329-40.

Biber, Douglas et al. (2004). Representing Language Use in the University:

Analy-sis of the TOEFL 2000 Spoken and Written Academic Language Corpus.

Prince-tonJ: Educational Testing Service.

Bloor, Meriel (1998). “Variations in the Methods Sections of Research Articles Across Disciplines: The Case of Fast and Slow Text”. Thompson, Paul (ed.),

Issues in EAP Writing, Research and Instruction. Reading: CALS, The

Univer-sity of Reading, 84-106.

Blue, George M. (1993). Language, Learning and Success: Studying through

Eng-lish. Developments in ELT. London: Macmillan, Modern English Teacher and

British Council.

Buck, Gary (2001). Assessing listening. Cambridge: Cambridge University Press. Chalhoub-Deville, Micheline; Deville, Craig (2005). “A Look Back and Forward

to What Language Testers Measure”. Hinkel, Eli (ed.), Handbook of Research

in Second Language Teaching and Learning. Mahwah (NJ): Erlbaum, 815-32.

Chalhoub-Deville, Micheline; Deville, Craig (2008). “Utilizing Psychometric Methods in Assessment”. Shohamy Elana; Hornberger Nancy H. (eds), Vol-ume 7: Language testing and assessment. Vol. 7 of Encyclopedia of language

and education. 2nd ed. New York: Springer Science-Business Media LLC.

Chapelle, Carol A. (1999). “Validity in Language Assessment”. Annual Review of

Applied Linguistics, 19(1), 254-72.

Chapelle, Carol A. (2012). “Conceptions of validity”. Fulcher, Glenn; Davidson Fred (eds), The Routledge Handbook of Language Testing. London: Rout-ledge, 21-33.

Chapelle, Carol A.; Enright, Mary K.; Jamieson, Joan (eds) (2008). Building a

Validity Argument for the Test of English as a Foreign Language. London:

Routledge.

Chini, Marina et al. (2003). “Aspetti della testualità”. Giacalone Ramat, Anna (a cura di), Verso l’italiano. Roma: Carocci, 179-219.

Coonan, Carmel M. (2012). La lingua straniera veicolare. Torino: UTET.

Cronbach, Lee J. (1971). “Test validation”. Thorndike, Robert L. (ed.),

Educa-tional measurement. 2nd ed. Washington, DC: American Council on

Edu-cation, 443-507.

Cronbach, Lee. J. (1988). “Five Perspectives on Validity Argument”. Wainer, Howard; Braun, Henry (eds), Test validity. Hillsdale (NJ): Erlbaum, 3-17. Cronbach, Lee J.; Meehl, Paul E. (1955). “Construct validity in psychological

tests”. Psychological Bulletin, 52, 281-302.

Cumming, Alister (1996). “Introduction: The Concept of Validation in Language Testing”. Cumming, Alister; Berwick, Richard (eds), Validation in Language

Testing. Clevedon: Multilingual Matters, 1-14.

Davies, Alan; Elder, Cathie (2005). “Validity and Validation in Language Test-ing”. Hinkel, Eli (ed.), Handbook of Research in Second Language Teaching

and Learning. Mahwah (NJ): Erlbaum, 795-845.

De Beni, Rossana; Pazzaglia Francesca (1995). La comprensione del testo.

Mo-delli teorici e programmi di intervento. Torino: UTET.

De Mauro, Tullio; Ferreri, Silvana (2005). “Linguistica educativa e insegnamen-to delle lingue: questioni scientifiche e questioni didattiche”. Voghera, Mi-riam; Basile, Grazia; Guerriero, Anna R. (a cura di), E.LI.CA., educazione

lin-guistica per l’accesso. Perugia: Guerra, 15-28.

Douglas, Dan (2000). Assessing Languages for Specific Purposes. Cambridge: Cambridge University Press.

Dudley-Evans, Tony; St John, Maggie J. (1998). Developments in English for

Spe-cific Purposes. Cambridge: Cambridge University Press.

Fahmy, Jackson; Bilton Linda (1991). “Listening and Note-taking in Higher Edu-cation”. Anivan, Sarinee (ed.), Language Teaching Methodology for the

Nine-ties. Singapore: SEAMO Regional Language Centre, 106-26.

Faraco, Martine; Barbier, Marie-Laure; Piolat, Annie. (2002). “A Comparison Be-tween L1 and L2 Note-taking by Undergraduate Students”. Ransdell, Sa-rah; Barbier, Marie-Laure (eds), New Directions in Research on L2 Writing. Dordrecht: Kluwer Academic Publishers, 145-68.

Ferne, Tracy; Rupp, Andre A. (2007). “A Synthesis of 15 Years of Research on DIF in Language Testing: Methodological Advances, Challenges, and Recom-mendations”. Language Assessment Quarterly, 4(2), 113-48.

Fulcher, Glenn; Davidson, Fred (eds) (2012). The Routledge Handbook of

Lan-guage Testing. London: Routledge.

Giacalone Ramat, Anna (ed.) (2004). Verso l’italiano. Roma: Carocci.

Green, Alison (1998). Verbal Protocol Analysis in Language Testing Research: A

Handbook. Cambridge: Cambridge University Press.

Hamp-Lyons, Liz (2016). “Purpose of assessment”. Tsagari, Dina; Banerjee, Jay-anti (eds), Handbook of Second Language Assessment. Boston; Berlin: de Gruyter Mouton, 13-28.

Hamp-Lyons, Liz (2011). “English for Academic Purposes”. Hinkel, Eli (ed.),

Handbook of Research in Second Language Teaching and Learning.

Abing-don: Routledge, 89-105.

Hamp-Lyons, Liz; Lynch, Brian K. (1998). “Perspectives on Validity: a Histori-cal Analysis of Language Testing Conference Abstracts”. Kunnan, Antho-ny J. (ed.), Validation in Language Assessment: Selected Papers from the

17th Language Testing Research Colloquium, Long Beach. Mahwah (NJ):

Hatch, Evelyn; Lazaraton, Anne (1991). The Research Manual: Design and

Statis-tics for Applied LinguisStatis-tics. New York: Newbury House Publishers.

Henning, Grant (1987). A Guide to Language Testing: Development, Evaluation,

Research. London: Newbury House.

Hughes, Arthur (1989). Testing for Language Teachers. Cambridge: Cambridge University Press.

Hyland, Ken (2006). English for Academic Purposes. New York: Routledge. Jones, Neil (2012). “Reliability and dependability”. Fulcher, Glenn; Davidson

Fred (eds), The Routledge Handbook of Language Testing. London: Rout-ledge, 350-63.

Kane, Michael T. (1992). “An Argument-based Approach to Validity”.

Psycho-logical Review, 112, 527-35.

Kane, Michael T. (2001). “Current Concerns in Validity Theory”. Journal of

Edu-cational Measurement, 38, 319-42.

Kane, Michael T. (2006). “Validation”. Brennen, Robert L. (ed.), Educational

Measurement. 4th ed. Westport (CT): Greenwood Publishing, 17-64.

Kunnan, Anthony. J. (ed.) (1998). Validation in language assessment: Selected

papers from the 17th Language Testing Research Colloquium, Long Beach.

Mahwah (NJ): Erlbaum.

Kunnan, Anthony J. (1999). “Recent Developments in Language Testing”.

An-nual Review of Applied Linguistics, 19(1), 235-53.

Lado, Robert (1961). Language Testing: The Construction and Use of Foreign

Lan-guage Tests. New York: McGraw Hill.

Lo Duca, Maria G. (2004). Lingua italiana ed educazione linguistica. Tra storia,

ricerca e didattica. Roma: Carocci.

Lumley, Tom; Brown, Annie (2005). “Research Methods in Language Testing”. Hinkel, Eli (ed.), Handbook of Research in Second Language Teaching and

Learning. Mahwah (NJ): Erlbaum, 833-56.

Lynch, Brian. K.; Davidson, Fred (1997). “Criterion Referenced Testing”. Clapham, Caroline; Corson, David (eds), Language Testing and Assessment. Vol. 7 of Encyclopedia of Language and Education. Dordrecht: Kluwer Aca-demic Publishers, 263-73.

Lynch, Brian K.; McNamara, Timothy F. (1998). “Using G-Theory and Many-fac-et Rasch Measurement in the Development of Performance Assessments”.

Modern Language Journal, 15(2), 158-80.

McNamara, Timothy F. (1996). Measuring Second Language Performance. Lon-don and New York: Longman.

McNamara, Timothy F. (2006). Language Testing. Oxford: Oxford University Press.

Mezzadri, Marco (2008). Italiano L2: progetti per il territorio. Perugia: Guerra. Mezzadri, Marco (2010). “Italiano L2 e integrazione scolastica: una ricerca

sul-le competenze linguistiche degli studenti stranieri a Parma e Reggio Emi-lia”. Mezzadri, Marco (ed.), Le lingue dell’educazione in un mondo senza

fron-tiere. Perugia: Guerra, 37-50.

Mezzadri, Marco (2011). Studiare in italiano. Milano: Mondadori.

Mezzadri, Marco (2013a). “Si può osare? Studio sull’accessibilità della forma passiva e del passato remoto per apprendenti non italofoni in contesto scolastico”. EL.LE, 2(2), 375-426. DOI http://doi.org/10.14277/2280-6792/61p.

Mezzadri, Marco (2013b). “Sviluppare, valutare e certificare l’italiano per lo stu-dio”. Rassegna Italiana di Linguistica Applicata, R.I.L.A., Anno XLIV, 151-63.

Mezzadri, Marco (2016). Studiare in italiano all’università. Torino: Bonacci. Mezzadri, Marco (2017). Testing Academic Language Proficiency. Newcastle

upon Tyne: Cambridge Scholars Publishing.

Moss, Pamela A. (2003). “Reconceptualizing Validity for Classroom Assess-ment”. Educational Researcher, 25(1), 13-25.

Oller, John (1979). Language tests at school. London: Longman.

Porcelli, Gianfranco (1975). II ‘language testing’. Bergamo: Minerva Italica. Porcelli, Gianfranco (1992). Educazione linguistica e valutazione. Padova:

Li-viana.

Purpura, James E. (2009). “The Impact of Large-scale and Classroom-based Language Assessments on the Individual”. Taylor, Lynda; Weir Cyril J. (eds), Language Testing Matters: Investigating the Wider Social and Educational Im-pact of Assessment = Proceedings of the ALTE Cambridge Conference (April 2008). Cambridge: Cambridge University Press, 301-25.

Purpura, James E. (2011). “Quantitative Research Methods in Assessment and Testing”. Hinkel, Eli (ed.), Handbook of research in second language teach-ing and learnteach-ing, vol. 2. New York: Routledge, 731-51.

Sawaki, Yasuyo (2009). “Application of three cognitive diagnosis models to ESL reading and listening assessments”. Language Assessment Quarter-ly, 6(3), 239-63.

Scaglioso, Anna M. (2005). “La valutazione delle abilità di produzione scrit-ta e di produzione orale”. Vedovelli, Massimo, Manuale della certificazione dell’italiano L2. Roma: Carocci, 217-88.

Sisti, Flora; Torrisi, Giovanni (2016). “Il puzzle dell’innovazione didattica all’U-niversità di Urbino: l’esperienza del CISDEL (Centro Integrato Servizi Didat-tici ed E-learning)”, in “Scuola Democratica 3”, special issue, Innovazioni di-dattiche nelle riforme universitarie. Bologna: il Mulino, 625-44.

Taylor, Lynda; Weir, Cyril J. (2009). Language Testing Matters: Investigating the Wider Social and Educational Impact of Assessment = Proceedings of the ALTE Cambridge Conference (April 2008). Cambridge: Cambridge University Press. Vedovelli, Massimo (2005). Manuale della certificazione dell’italiano L2. Roma:

Carocci.

Weigle, Sara C.; Lynch, Brian. K. (1996). “Hypothesis Testing in Construct Valida-tion”. Cumming, Alister (ed.), Selected Papers from the 1992 Language Test-ing Research Colloquium. Clevedon: MultilTest-ingual Matters, 58-71.

Weir, Cyril. J. (2005). Language Testing and Validation. Basingstoke: Palgrave Macmillan.

Xi, Xiaoming (2008). “Methods of Test Validation”. Shohamy, Elana; Hornberg-er Nancy H. (eds), Language Testing and Assessment. Vol. 7 of Encyclopedia of Language and Education. 2nd ed. New York: Springer Science-Business Media LLC, 177-96.