CAD: How it works, how to use it, performance

(1)

ContentslistsavailableatSciVerseScienceDirect

European

Journal

of

Radiology

j ou rn a l h om ep a g e :w w w . e l s e v i e r . c o m / l o c a t e / e j r a d

CAD:

How

it

works,

how

to

use

it,

performance

Daniele

Regge

a,∗

_,

_Steve

_Halligan

b,1

a_Institute_for_Cancer_Research_and_Treatment,_{Candiolo-Torino,}_Italy b_University_College_Hospital_London,_London,_UK

a

r

t

i

c

l

e

i

n

f

o

Articlehistory: Received17April2012 Accepted21April2012 Keywords:

ComputerAidedDiagnosis(CAD) CTcolonography(CTC) Colorectalcancer

a

b

s

t

r

a

c

t

Computer-aideddiagnosis(CAD)systemsaresoftwarealgorithmsdesignedtoassistradiologists(or otherpractitioners)insolvingadiagnosticproblembyusingavisualprompt(or“CADmark”)todirect theobservertowardspotentialpathology.CTcolonographyisarecentarrivaltoCAD,butcouldrepresent oneofitsmostfruitfulapplicationsinthefuture.Incontrasttootherorgans,whereavarietyof differ-entpathologiesareequallyrepresented,significantcolorectalpathologiesotherthanpolypsandcancer arerelativelyuncommon.Asweshallsee,thissimplifiesthediagnostictaskforartificialintelligence developersandalsoforradiologistswho,ultimately,mustmakethefinaldecision.

Thisreviewaimstopresentthecurrentstate-of-the-artforCADappliedtoCTcolonography.Abrief overviewofthetechnicalessentialsandofthediagnosticperformanceofCADinisolation,isfollowedby anexplanationofhowCADisusedinday-to-daypractice.Thelastsectionwilldealwiththemost con-troversialissuesaffectingCADperformanceinclinicalpractice,withafocusontheinteractionbetween humanandartiﬁcialintelligence.

1. IntroductionandrationaleofCADassistedCT

colonography

Computer-aideddiagnosis (CAD)systems are software algo-rithms designed to assist radiologists (or other practitioners) in solving a diagnostic problem by using a visual prompt (or “CADmark”)todirecttheobservertowardspotentialpathology. Such algorithmsmaybeused tohelppinpointareas ofdisease (computer-aided-detection, CADe), or help interpreters decide whetherabnormalﬁndingsarebenignor malignant (computer-aided diagnosis, CADx). Regardless of their task, the human interpreter must alwaysmake the ﬁnal diagnostic decision, by decidingwhetherornotto“believe”theCADmarks.

CADsystemsarenotnewtoradiology.Digitalplatformswere developedinthe1980swiththeaimofdetectingand characteriz-ingbreastnodulesinmammograms.Morerecently,CADsoftware hasbeendevelopedtosearchforlungnodulesinCTscansandhelp deﬁnetheirnature,andtodetectprostateandbreastcanceratMRI. DespiteahugeresearcheffortonthepartofAcademicInstitutions

∗ Correspondingauthorat:InstituteforCancerResearchandTreatment,Str. Provincialen◦_142,_Km_3,95,_{Candiolo-Torino,}_Italy._Tel.:₊₃₉_0119933022.

E-mailaddresses:daniele.regge@ircc.it(D.Regge),s.halligan@ucl.ac.uk (S.Halligan).

1 _University _College _Hospital _London, _Department _of _Specialist _Radiology, Podium,Level2,UniversityCollegeHospital,235EustonRoad,London,UK.

andIndustry,fewCADalgorithmsareusedroutinelyinday-to-day clinicalpractice.Severalreasonsunderpinthisfailuretoimplement CADindailypracticeandsomearewellunderstoodwhileothers arelessobvious.Mostobviously,suchalgorithmsmaybeexpensive topurchaseandservice.Furthermore,dependingontheCountryin question,over-restrictivelicensingstipulationsmayrestrict avail-ability.OneintuitiveobjectiontotheuseofCADsystemsisthat theygeneratevisualmarksthattheradiologistmusttaketimeto interpret.Interpretationisrequiredbecauseeachmarkiseither true-positiveforpathologyorfalse-negative,andtheradiologist mustdecidewhichitis.Alargenumberofmarkswilllikely pro-longexaminterpretationtime.Also,mostmarksarefalse-positive andthereisconcernthatifadisproportionatenumberoftheseare misinterpretedbytheradiologistastrue-positive,thenthe num-berofindividualsundergoingunnecessarysecondlevel,follow-up testswillincrease.Suchadditionalexams,whichareusuallymore invasivethanﬁrstleveltests,willincreaseprocedurerelatedcosts, likely increase the rate of adverse events, and also precipitate patientanxiety.Itisalsopossiblethatradiologistsmaybe reluc-tanttorelyonthehelpofartiﬁcialintelligence,whichtheymight distrust.

CTcolonographyisarecentarrivaltoCAD,butcouldrepresent oneofitsmostfruitfulapplicationsinthefuture.Thisisbecausethe colorectalcancermodelisrelativelysimpleandcanbeepitomized as a trafﬁc light weregreen is a negative test, yellowa polyp, andredacancer.Incontrasttootherorgans,whereavarietyof differentpathologiesareequallyrepresented,signiﬁcantcolorectal

(2)

1172 D.Regge,S.Halligan/EuropeanJournalofRadiology82 (2013) 1171–1176

pathologiesotherthanpolypsand cancerarerelatively uncom-mon.Asweshallsee,thissimplifiesthediagnostictaskforartificial intelligencedevelopersandalsoforradiologistswho,ultimately, mustmakethefinaldecision.Thiscompellingrationaleprobably explainswhythedevelopmentofCADsolutionsforCT colonog-raphyhasbeensofast;todate,over20companiesareproducing andcommercializingdedicatedCADsoftwareworldwide.

Likedetectionalgorithmsinotherorgans,themainobjective ofCADassistedCTcolonographyis toidentifylesionsthathave beenmissed bythereader,so called“operatorerrors”or “per-ceptive errors”. Such errors account for approximately half to three-quartersofallmissedbutdetectablelesions[1].Perceptive errorhasbeenevaluatedextensivelyinbreastimaging,whereitis amajorcauseforconcern.Indeed,incontrasttothecolon,were thetargetlesionismainlybenignneoplasia,thelargemajorityof breastlesionsaremalignant.Perceptualerrorrateisinfluencedby readerexperienceandintrinsicdiagnosticthreshold,working envi-ronment,examquality,andlesioncharacteristicsandmorphology. CADassistedCTcolonographyshouldimprovepolypdetectionby influencingsomeofthesefactors.Intuitively,pinpointinglesions thathave beenoverlookedbecauseofan unfavourablelocation and/oratypicalmorphologyiswheremostdiagnosticimprovement byusingCADistobeanticipated.Byimplication,suchgainsshould bedisproportionatelybeneficialforlessexperiencedreaders.

Thisreview aims topresent thecurrent state-of-the-art for CADappliedtoCTcolonography.Abriefoverviewofthetechnical essentialsandofthediagnosticperformanceofCADinisolation,is followedbyanexplanationofhowCADisusedinday-to-day prac-tice.Thelastsectionwilldealwiththemostcontroversialissues affectingCADperformanceinclinicalpractice,withafocusonthe interactionbetweenhumanandartiﬁcialintelligence.

2. Insidetheblackbox

RadiologistsdonotneedtoknowhowCADalgorithmsworkin ordertoemployasystemintheirreporting.However,asuperﬁcial understandingofwhathappensinsidethe“blackbox”doesprovide someinsightastowhythingsoccasionallydon’thappenaccording toplan(e.g.toomanyCADcandidates,excessiveprocessingtimes, incompletesegmentation)andastothetypeoflesionthatmight bemissedbyaspeciﬁcCADplatform.

Typically, CAD development schemes require three sequen-tialsteps.Thefirststepiscolonicsegmentation,wherebyvoxels locatedattheinterfacebetweentheluminalcontentandmucosa areisolatedandextractedfromsurroundingCTdata(Fig.1aand b).Segmentationis usuallyperformedbydefining localdensity thresholds,aprocessthatdiscriminatesvoxelsoriginatinginthe colonwallfromthoseofsurroundingtissues[2].While segmenta-tionisusuallyanautomaticprocess,someCADsystemshavetools availabletoremove structuresthat havebeenerroneously seg-mentedwiththecolon(mostfrequentlysmallbowel,stomachand lungbases),ortoreconnectcolonicsegmentsseparatedbyspasm, etc.,sothatevaluationoftheentirecoloncanbeperformed(Fig.1c). Sinceitisimportanttodetectlesionsthataresubmergedbyfluid, segmentationshouldincludeelectroniccleansingoftaggedfluid [3],i.e.residualfluidiseliminatedbythesoftware.

ThesecondstepinthedevelopmentofaCADalgorithmisthe identificationofpolyps,or“candidates”.Thisprocessisbasedonthe analysisofspecificmorphologicalfeaturesoftheextractedsurface, inparticularthosethatindicateapolypoidstructureprotruding fromtheluminalsurface.Thereareseveralwaystoapproachthis; themostefficientistouseshapeandcurvatureindexes.Theshape indexexaminesagroupofcontiguousvoxelsandassignstoeach ascorefrom−1to+1,accordingtotheirmorphologyata spe-cificpoint.Dome-shapedstructures,suchaspolyps,areassigned

Fig.1.Exampleofcolonsegmentation.(a)Thesoftwareplacesseedsautomatically withinthecolonlumen.Theseedsexpanduntiltheyreachthebowelwall.Allextra dataisremovedaftertheinterfacebetweenlumenandbowelwallhasbeen identi-ﬁed.(b)CTimagesarethenstackedandthesurfacebetweenslicesisgeneratedby interpolation.Thecolonisthensmoothed,artiﬁciallycolouredandlightisprojected onitssurfacetoimproverendering.(c)Theimageincludestwosmallbowelloops (i.e.theyellowandbluetransparentstructuresontheupperleftside)thathave beensegmentedwiththecolon,andcanberemovedmanually.

(3)

Fig.2. CADsystemuserinterface.IntheexampleayellowframecontourstheCADcandidateinthe2Dimage.IntheendoluminalviewCADmarkingsarehighlightedwith adifferentcolorfromthatofthesurroundingnormalmucosa.AlistofindividualCADcandidatesisusuallyavailableinaddition,separatelyfortheproneandsupinescans.

thehighestvalues. Thecurvatureindexdescribesthedegreeof localsurface“slope”.Lowvaluesareassignedtoﬂatsurfacesand, alternatively,highvaluesareassignedtosteepslopes.Polyps usu-allyhaveintermediatecurvaturevalues.Acombinationofthetwo indexesallowsdiscriminationbetweenpolypsandfoldsordipsin theendoluminalsurface.

Thisextractionphase generatesa largenumber ofcandidate polyps,typicallybetween50and100foreachCTseries,mostof whicharethickenedhaustralfoldsorfecalresidues.Eventhough mostcandidatesareeasytodismissbyanexperiencedradiologist, reviewofsuchalargenumberisalaboriousandtime-consuming task.So,thethirdstepinCADdevelopmentistoreducethenumber ofpolypcandidatespresentedtothereaderbyselectingonlythose thathavethehighestprobabilityofbeingtruepositivepolyps.This

processisperformedbyassessingmultipledifferentcharacteristics foreachcandidate(e.g.volumetrictextureanalysis,CT attenua-tion,randomorthogonalshapesection,opticalflow,etc.).Adiscrete valueisthenassignedtoeachcandidatebasedonthecalculated featuresandastatisticalclassifierissubsequentlyappliedtoselect CADcandidatesthathavethehighestprobabilityofbeingtrue pos-itives.Ultimately,polypcandidatesselectedbytheclassifierare highlightedtothevieweroftheCADinterfacewithamarker,or adifferentcolorfromthatofthesurroundingnormalmucosa.A listofindividualCADcandidatesisusuallyavailableinaddition,for boththeproneandsupinescansseparately(Fig.2).

Whiletheprocessdescribedabovecanbeinitiatedbysoftware code alone, the best CADsystems are “trained” by using large numbersofCTcolonography datasetscontainingendoscopically

(4)

validated polyps.Essentially, by using such databases software developerscanfine-tunetheCADalgorithmbyexposingittoreal polyps.Alloftheprocessesdescribed thusfarcanbe conceptu-alizeda CAD“development”.However,when theCADsoftware appearstoreachadequatediagnosticperformance,itisthentested using a CT colonography database composed of cases that the softwarehasnot“seen”before.The“stand-alone”performanceof aCADalgorithmexcludestheradiologistinteraction,andmerely reportstheabilityofthesoftwaretodetecttruepolyps(sensitivity) and reports the number of false-positivedetections per series, whichisequivalentto1-specificity.Since2002,severalresearchers havereportedthestand-alonesensitivityandspecificityofvarious academicand commercialCAD systems [4,5]. AsCAD schemes evolve,performancesimproveintandem.Recently,Lawrence[6] reported per-polyp CAD sensitivities in isolation, regardless of histology,of90%and96%at6-and10-mm-diameterthresholds, respectively,ata meanfalse-positiverateof4.7 perseries.The diagnosticperformanceofCADsystemscanalsoberepresentedby Free-responseReceiver Operating Characteristics (FROC) curves [7].FROCcurvescombinesensitivityandspecificityintoasingle metric which is then plotted at a range of differentdiagnostic thresholds.Specifically,sensitivityisplottedagainst1-specificity (i.e. the false-positive rate) as the classifier’s threshold value for a true positive findingis lowered progressively(i.e. as the systembecomesprogressivelymoresensitive).FROCcurvesallow differentCADsystemstobecomparedandareusedbyresearchers to select the most appropriate single operating point for the software,whichis inevitablyacompromise betweensensitivity and specificity. Although the stand-alone performance of an algorithm provides some information regarding the diagnostic capabilities of the software, this alone cannot determine the diagnosticperformanceofCADinclinicalpractice.Thisisbecause itisnecessarytoincorporatethehumanreaderandunderstand howtheirinteractionwithartificialintelligenceaffectsdiagnostic performance.

3. HowtouseCAD

ACADreadingparadigmisadescriptionofhowinterpretinga CTCexamwithCADisachieved.Themaindifferencebetween dif-ferentparadigmsisinthetimingwithwhichCADcandidatesare presentedtothereader.Theperformanceofdifferentparadigms andtheirinteractionwiththehumanreaderforCTChavebeen assessed recentlyby researchers, and ﬁndings are summarized below.

1)“Firstreader”CAD.Inthefirstreaderparadigm,theCAD algo-rithmis activatedimmediatelywhen thereaderinterprets a case.AlistofCADmarksarepresentedtotheinterpreterfor review. The reader reviews all CAD candidates sequentially andreportsthosethathe/shebelievesaretruelesions,ifany. Interpretation is restricted totheCAD marksand because a full,unassistedreviewoftheentireendoluminalsurfaceisnot required,readingtimeisreducedgreatly[8].Shortening inter-pretationtimemaybeimportantiflargenumberofcasesmust bereadsequentially,which couldbethecaseinthefutureif CTCscreeningisimplementedwidely.However,intheprocess, sensitivityandpositivepredictivevalueshouldnotbeaffected negatively, so it is important that thestand-alone detection characteristicsofthesystemareoutstanding.Preliminarydata regardinguseofthefirstreadparadigminmassscreening pro-grammesdemonstratethataccuracyofCTCissustainedwhen compared to unassisted reading, but reading time isshorter [9].Inasymptomaticenvironment,wherediseaseprevalence ishigher,thefirst-readparadigmcouldpossiblybeapproved

ifthestand-alonesensitivitywas100%.Ifper-polypsensitivity fallsmuchbelowthisforlargerpolyps,thefirstreadparadigm shouldprobablynotberecommended.Thisisespeciallythecase becauseCADschemesinisolationarelessefficientwhen iden-tifyinglesionsthatdonotprotrudeintothecolonlumen,such asinfiltratingflatcancers.Thereforeclinicallyrelevantlesions couldbemissed ifradiologistsdo notreview theentireCTC dataset[10].

2)“Second-reader”CAD.Insecond-readmode,theradiologistfirst performsaninitialunassisted readoftheentireCTCdataset, usingaprimary2Dor3Dread,accordingtoindividual prefer-enceandexactlymimickingday-to-daypracticewithoutCAD. Afterthisunassistedread,CADisactivatedandanycandidates arecheckedforadditionalfindingsmissedontheinitial inspec-tion.Asforthefirst-readparadigm,readersmustdecidewhether toacceptorrejecteachCADcandidate;however,theyshould not reverse decisions taken during theirunassisted reading. Intuitively,second-readCADshouldbemostbeneficialwhen adopted by less experienced readers; radiologists relatively unfamiliarwithCTChavelowerdetectionratesandtheirgain whenusingCADshouldbelargerthanforexperiencedreaders [11].Bakeretal.[12]haveshownthat,forpolyps6mmorlarger, second-readCADimprovesdetectionofreaderswithsomeor noexperienceinCTCreportingfrom81%to91%.Theseauthors alsofoundthatCADsecond-readincreasesdetectionof inter-mediatesizepolyps.Recently,Halliganetal.[13]demonstrated thatsecond-readCADalsobenefitstheexperiencedCTCreader, findingthatmeanper-polypsensitivityforpolyps6mmorlarger increasedonaverageby9%.Asexpected,additional interpreta-tionofCADcandidatesextendsoverallCTCreadingtimebya fewminutes[12,14].

3)“Concurrent” CAD. In the concurrent-reader paradigm, CAD is applied immediately prior to review, like the first-read paradigm.However,thereaderdoesnotrestricthimselftothe CADmarksbutalsointerpretsthewholedataset.Thisparadigm thus combines elements of both the first- and second-read paradigms;CTCinterpretationisperformedonce,withtheCAD candidates on the screen from the beginning. Readers may choosetoturnCADmarkingonand offanytime theywish. Halliganetal.[11]haveshownthatconcurrentreadincreases theper-polypdetectionofinexperiencedradiologistson aver-ageby9.1%(i.e.from17%to26%)whilereducingreadingtime byslightlyover 2min.Thebenefittheyobservedwasmainly duetoincreaseddetectionofmediumandsmall-sizedpolyps. However,themagnitudeofbenefitwasnotaspronouncedas forsecond-readCAD,whenthetwoparadigmswerecompared directlyonthe same datasetsand readers [13]. Thereasons underpinningthelowerdiagnosticperformanceofconcurrent reading are not yet well understood. It is possible that, by focusingtheir attentiononCAD marks, readersreduce their assessmentoftheremainingportionsofcolorectalmucosaas opposedto whathappens duringunassisted reading [14]. In summary,accordingtoavailableevidence,concurrentreading islesseffectivethanthesecond-readparadigmanditsroutine useisnotrecommendedinclinicalpracticeatthepresenttime. Thesamecommentappliestofirst-readCAD.

4. LimitsofknowledgeofCADperformanceand

controversies

CADimprovesdetectionofcolorectallesionsbydirectingthe readertosuspiciousareasofcolorectalmucosa.Forboth inexpe-riencedreadersandexperts,thediagnosticperformanceofCADis

(5)

Fig.3.Fifty-eightyearoldmaleperformingCTcolonographyafterapositivefecaloccultbloodtest.(a)TheaxialCTscanshowsa6mmpolyp-likestructureofthetransverse colon.(b)ThelesionwaspromptedbytheCADsystembutwassubsequentlyrejectedbytheradiologistreviewingthecase.(c)Thepeduncolatedpolypwasconﬁrmedat endoscopy.(d)ThecorrectlyannotatedlesionwasprobablyrejectedbecauseofthelargenumberoffalsepositiveCADannotationsthatwerepresentinthesameexam(areas highlightedwithadifferentcolor).

bestwhenusedasthesecond-reader,eventhoughoverallreporting timeisincreased.

However,becauseCADschemeshavebeentestedmostlyin ret-rospectivecohortsandinalaboratoryenvironment,littleisknown regardingitsperformanceinday-to-dayclinicalpracticewhenused byrepresentativeradiologists.AnItaliancross-sectional multicen-trestudyhasrecentlyshedsomenewlightontheutilityofCAD inclinicalpractice.Thestudy,recruiting18experiencedreaders from10Academiccentres,concludedthatsecond-readCAD sig-nificantlyimprovesdetectionofpolypsbetween6and9mm,but observednobenefit forlargerlesions[15].In ordertoestablish firmlyifthereisaroleforCADinclinicalpractice,andfor screen-ing,furtherevidencewillberequiredfromlarge-scale,preferably randomizedtrials,retracingthepathofrecentlypublishedstudies evaluatingtheutilityofbreastmammographyCADschemes[16].

Little is known regarding how inexperienced readers – i.e. “beginners”andreaderswithlimitedexperiencewithCT colonog-raphy–interactwithCADinclinicalpractice.Whencomparedtoan

unassistedread,second-readCADincreasesthefalse-positiverate ofnon-expertsin43%ofcases(i.e.3of7readers)[12].However,it hasbeenshownthattrainingreducesthenumberoffalse-positive diagnosesbyreaders[17].Therefore,thereiscompellingevidence thatthelevelofreaderexperienceaffectstheirultimate perfor-mancewhenusingCAD.Exam-relatedfactors,suchasthenumber ofCADprompts[18]andthetypeofbowelpreparation[10],may alsoinﬂuencereaders’false-positiverates.Onthispoint,resultsare somewhatcontroversial.Forexample,astudybyTayloretal.[19] foundthatarelativelyhighnumberoffalse-positivesdoesnot sig-niﬁcantlydistractreadersfromtheirtrue-positivedetections,but doesincreaseinterpretationtime.Certainly,sincereadingtimeis proportionaltothenumberofCADmarks,interpretationtimewill beprolongedwhenthenumberofmarksishigh.

Sinceresearchﬁndingsremaininconsistentandcontroversial, itisgenerallyagreedthatradiologistsshouldadoptCADonlyafter theyhave beenadequately trained in unassisted CT colonogra-phyinterpretation.However,sincethedeﬁnitionof“adequately

(6)

trained”isitselfstillamatterofdebate,thesuggestedtimingfor CADadoption is uncertain. Atthe time of writing, theauthors believethatintermsofclinicalutility,thereissufﬁcientcurrent evidencetosuggest thattheadditionalpolypsdetectedbyCAD outweighanyadditionalfalse-positivediagnoses.CADislesslikely tobeclinicallyusefulinsituationswhere thereare overwhelm-ingnumbersoffalsepositiveprompts,suchasoccursinapoorly preparedcolonforexample.

Finally,readersshouldbeawarethatitispossibletoerroneously rejecttruepositiveCADprompts.Inmoststudies,thesensitivityof CADinisolationhasbeensuperiortothatachievedbyradiologists assistedbythesameCADalgorithm[5].Itisclearthatdiminished sensitivityisduetothefactthatreadersignorea percentageof true-positiveCADmarks[11,20].Thereasonsunderpinning this apparentlyillogicalbehaviourareuncertain.Tayloretal. investi-gatedthisphenomenonbycomparingtwogroupsofpolypsthat werecorrectlydetectedbyCAD;onegroupwhereCADdetection resultedinanadditionalcorrectdetectionbyreaders,andasecond groupforwhichtheCADmarkdidnotimprovesensitivity[20]. Theyfoundthatlesionsmisclassifiedasfalse-negativesbyreaders weresignificantlylargerandirregular.Theauthorsconcludedthat difficultywithcharacterization–i.e.duetoanirregularand/orflat morphology–wasthemaindeterminantwhyradiologistsrejected truepositiveCADprompts.Otherreasonsthatmaycausereaders torejectcorrectlyannotatedpolypsare:polypsnearoronfolds, lesionscoatedorsubmergedbytaggedfluid,ahighnumberofCAD annotations(Fig.3),andalocationnearotherpolyps.

5. Concludingremarks

CADforCTcolonographyhasprogressedalongwayoverthe lastdecade,andsystemswithexcellentdetectioncharacteristics havebeendeveloped,validated,andarecommerciallyavailable. Whilesuchsystemsdonotobviatetheneedforspecifictrainingin interpretationofCTcolonography,theydoprovidegainsin sensi-tivitywhensearchingforpolypswithoutadisproportionatedrop inspecificity.Thesegainsarelargelydependentontheexpertise ofthereader,butadiagnosticbenefitofvaryingdegreesseems tobeavailabletoreadersofallabilitieswhenCADisusedas stipu-latedbyregulatoryauthorities.Thenextdecadewillrevealwhether thesesystems are adopted by users and so become integrated intoday-to-daypractice,orwhethertheyremainconfinedtothe laboratory.

References

[1]DoshiT,RusinakD,HalvorsenRA,etal.CTcolonography:false-negative inter-pretations.Radiology2007;244(1):165–73.

[2]BidgoliJ,AhmadianA,AkhlaghporS,etal.Anefﬁcientcolonsegmentation methodfororalcontrast-enhancedCTcolonography.In:Conf.Proc.IEEEEng. Med.Biol.Soc.,vol.4.2005.p.3429–32.

[3]MangT,BogoniL,SalganicoffM,Wolf M,RaykarV,MacariM,Pickhardt PJ, IafrateF,LaghiA,WeberM,BakerME,RinglH,HeroldCJ, GraserA. Computer-aided detection of colorectalpolyps in CTcolonography with andwithoutfecaltagging:astand-aloneevaluation.InvestigativeRadiology 2011.

[4]Summers RM, Yao J, Pickhardt PJ, et al. Computedtomographic virtual colonoscopycomputer-aidedpolypdetectioninascreeningpopulation. Gas-troenterology2005;129(6):1832–44.

[5]TaylorSA,HalliganS,BurlingD,RoddieME,HoneyﬁeldL,McQuillanJ,Amin H,DehmeshkiJ.Computer-assistedreadersoftwareversusexpertreviewers forpolypdetectiononCTcolonography.AmericanJournalofRoentgenology 2006;186(3):696–702.

[6]LawrenceEM,PickhardtPJ,KimDH,RobbinsJB.Colorectalpolyps:stand-alone performanceofcomputer-aideddetectioninalargeasymptomaticscreening population.Radiology2010;256(3):791–8.

[7]BornefalkH.EstimationandcomparisonofCADsystemperformanceinclinical settings.AcademicRadiology2005;12(6):687–94.

[8]ManiA,NapelS,PaikDS,etal.Computedtomographycolonography: feasibil-ityofcomputer-aidedpolypdetectionina“ﬁrstreader”paradigm.Journalof ComputerAssistedTomography2004;28(3):318–26.

[9]IussichG,CorrealeL,IafrateF,etal.Computer-aideddetectioninCT colonog-raphy(CTC):whichCADparadigmisbestinascreeningpopulation?ESGAR. 2011.CodeSS10.08.

[10]Robinson C, Halligan S, Iinuma G, et al. CT colonography: computer-assisted detection of colorectal cancer. British Journal of Radiology 2011;84(1001):435–40.

[11]HalliganS,AltmanDG,MallettS,etal.Computedtomographic colonogra-phy:assessmentofradiologistperformancewithandwithoutcomputer-aided detection.Gastroenterology2006;131(6):1690–9.

[12]Baker ME,BogoniL, ObuchowskiNA,et al.Computer-aided detectionof colorectalpolyps:canitimprovesensitivityofless-experiencedreaders? Pre-liminaryﬁndings.Radiology2007;245(1):140–9.

[13]HalliganS,MallettS,AltmanDG,etal.Incrementalbeneﬁtofcomputer-aided detectionwhenusedasasecondandconcurrentreaderofCTcolonographic data:multiobserverstudy.Radiology2011;258(2):469–76.

[14]TaylorSA,CharmanSC,LefereP,McFarlandEG,PaulsonEK,YeeJ,AslamR, Bar-lowJM,GuptaA,KimDH,MillerCM,HalliganS.CTcolonography:investigation oftheoptimumreaderparadigmbyusingcomputer-aideddetectionsoftware. Radiology2008;246(2):463–71.

[15]ReggeD,DellamonicaP,CorrealeL,etal.TheCAD-IMPACTstudy: computer-aideddetectionassecondreaderinaprospectivemulticentersetting.In:RSNA. 2009.CodeSSE08-04.

[16]GilbertFJ,AstleySM,GillanMG,etal.Singlereadingwithcomputer-aided detection forscreeningmammography.NewEnglandJournalofMedicine 2008;359(16):1675–84.

[17]TaylorSA,BurlingD,RoddieM,etal.Computer-aideddetectionforCT colonog-raphy:incrementalbeneﬁtofobservertraining.BritishJournalofRadiology 2008;81(963):180–6.

[18]Taylor SA, Brittenden J, Lenton J, et al. Inﬂuence of computer-aided detectionfalse-positivesonreaderperformanceanddiagnosticconﬁdence for CT colonography. American Journal of Roentgenology 2009;192(6): 1682–9.

[19]TaylorSA,GreenhalghR,IlangovanR,etal.CTcolonographyand computer-aided detection: effectof false-positiveresultson readerspeciﬁcity and reading efﬁciency in a low-prevalence screening population. Radiology 2008;247(1):133–40.

[20]TaylorSA,RobinsonC,BooneD,etal.Polypcharacteristicscorrectlyannotated bycomputer-aideddetectionsoftwarebutignoredbyreportingradiologists duringCTcolonography.Radiology2009;253(3):715–23.