U
VERS
TÀD
ADVA FACTÀD

GEGER
A

INTERFACCIA GRAFICA E APTICA

PER UN'APPLICAZIONE DI BRONCOSCOPIA VIRTUALE

Relatore: Prof. AlfredoRuggeri

Correlatore: Prof. JanRosell Grata òs

Laureando: PaoloCabras

Corso diLaurea Spe ialisti ain Ingegneriadell'Automazione

7 Di embre 2010

Abstra t

Virtualbron hos opyisa omputer-generated,three-dimensionalre onstru tionte hniquethat

sim-ulatesabron hos opist'sviewoftheairways, reatingatrainingenvironment wherethedo tor an

be assistedwithindi ation of the shortest path onne ting thetra hea to thelung'sdiseased zone

whi h hewants to analyze.

The ideais to make the lungexploration asmu h real aspossible,asifthe do tor wasreally

han-dling a bron hos ope, even giving thesensation of tou hingand intera ting withthe real physi al

bron hial tube.Withthis aim,inthis work,theuserwill arryout theexplorationtaskbyusing a

hapti devi e, whi h isa devi ethatallows for efeedba k that an beexploited to give tou h and

fri tion sensationand also to provide guidan ealong the al ulated path.

The development of theproje trequires: a) Theimplementation of thegraphi alrepresentation of

the lungs both asa general view and asseen from the bron hos ope tip; b) The determination of

themovement ofthe hapti devi ethatallowsto ontrolthemotionsofthevirtualbron hos opein

a realisti way; )Thehapti renderingofthe onta tfor es;d) Validation withqualiedpersonal.

The software will be developed using H3DAPI for images representation and Hapti interfa e, Qt

for theGraphi UserInterfa e andCMake fordeningtheproje t: dueto this hoi estheprogram

Contents Abstra t 3 Prefa e 9 1 Introdu tion 11 2 State of Art 13 2.1 Bron hos opy . . . 13 2.2 Virtual Bron hos opy . . . 15 2.3 SimulationTraining . . . 17 3 Motivation 21 4 Obje tives 23

5 Involved Hardware: Hapti Devi e 25

5.1 Hapti Devi es . . . 25

5.2 PHANTOM Omni . . . 27

5.3 Hapti Renderers . . . 28

6.2 Graphi Rendering:H3DAPI . . . 32

6.3 Hapti s Rendering Engine: HAPI . . . 33

6.4 Framework for softwaredevelopment: Qt . . . 34

7 Hapti s Interfa e 37 7.1 Bron hos ope Modeling . . . 40

7.2 Navigation System . . . 41

7.2.1 Moving dire tlythe amera . . . 43

7.2.2 Moving the amera fromthe baseof thetip . . . 51

7.2.3 Going Ba kward . . . 59

7.3 Collisions andHapti Renderer Chosen . . . 61

7.3.1 C++ implementation . . . 65

8 Graphi s Interfa e 69 8.1 Qt - H3Dintegration: TheQTWindow Class . . . 69

8.2 MainAppli ation . . . 71

8.2.1 Graphi design . . . 71

8.2.2 Fun tioning . . . 74

8.3 Two dierent views . . . 77

9 Environmental Analysis 81

10Costs Analysis 83

11Results 85

A.1 Example ofnavigation . . . 96

Bibliography 99

Bibliographi Referen es . . . 99

Prefa e

Lately,engineeringandmedi ineare ollaborating moreandmoretondnewsolutionstomakethe

do tortaskeasierandin rease the omfortforthepatient.Thetenden yistomake theoperations

the leastinvasive possible and to intensify theuse of virtual reality for training or diagnosis.This

te hniques have been improving their reliability and they are in reasingly assuming a

fundamen-tal role on the medi ine s ene. Despite this big steps, many invasive medi al pro edure are still

unavoidable, an example is bron hos opy. It still representsthe unique wayto obtain samples of

thepossibletumoral tissuefor furtheranalysisor to remove foreignobje ts fromtheairways.This

pro edureisstillverybotheringforthepatientandinsome asesittakesalongtime(bron hos opy

an last between 20and 30 minutes dependingon thepro edureto perform).

Also inthis eldthe do tor an re eive animportant assistan e fromte hnology.The most spread

and armed tools used are Virtual Bron hos opy and training platform. They take are of two

dierent but both important sides of the bron hos opy eld. Training platforms provide medi al

sta with a very real environment where they an approa h to the bron hos opy in a safe way

and experiment some kind of ompli ations that an omeout during therealpro edure.

Virtual Bron hos opy helps do tors to make preliminary studies on the onformation of the

pa-tient airwayand hoosethe better solution pro edure. Thein reasing diusion of this tool an be

onsidered a natural expression of the urrent tenden y of the personalization of the health are

whosebasi ideaistousethepatient'sinternalstru turetoplanthebetterpro edure.Furthermore,

nowadays the learness of the information is another important point and with 3D re onstru tion

providedbyvirtualbron hos opy,patientanddo tor anhavealmostthesamelevelof

understand-ing.

Thiswork wants to pla e itself exa tlyin thejun tion of these two elds (training and virtual

re onstru tion to previous planning and diagnosis) reatinga new tool that an personalize also

Chapter 1

Introdu tion

Bron hos opy is an interventional medi al pro edure used to analyze the bron hial tube, to see

abnormalities of the airway, obtain samples from a spe i lung site identied on preoperative

images (CTI or MRI) 1

,evaluate a person who has bleeding in the lungs, possible lung an er, a

hroni ough, or a ollapsed lung, remove foreignobje ts lodgedinthe airway or open thespa es

of ablo ked airway.

A riti al point for this te hnique isto nd theright path to getto thetarget: do tors usually

analyze the DICOM images 2

and try to plan the path on these 2-D images, whi h are very

dif-ferent fromthe real s ene they will fa ewhile exe uting bron hos opy.Finding theway insidethe

tra heobron hial tree,then, omesto be verydi ultandittakesalongtimeto thedo torto nd

theright pathor to make anoverallanalysisof theairways,witha onsequent greatbotherforthe

patient.Con erning thislast point,the urrent tenden yinmedi alpro edures isto make themas

mu h no-invasive as possible so that the patient would be less ae tedand would feel more

om-fortable. Considering this fundamental question, Virtual Bron hos opy (VB) ould be onsidered

a useful tool. VB isa omputer-generated, three-dimensional re onstru tion te hnique that allows

medi alstatoexplorethetra heobron hialtree.Thiste hniqueisalreadyusedtomakeaprevious

analysis oflungs,help individuating thediseasedzone andto see ifitis ne essaryto pro eedwith

real bron hos opy.

Thisproje t intends to improve Virtual Bron hos opyso thatto make thelung explorationas

mu hrealaspossible,asifthedo torwasreallyhandlingabron hos ope,evengivingthesensation

of tou hing and intera ting with the real physi al tra heobron hial tube. With this aim, in this

work, the user will arry out theexploration task byusing a hapti devi e, whi h isa devi e that

allows for efeedba kthat an be exploited to give tou h and fri tionsensation.

1

ComputedTomographyandMagneti Resonan eImages

Chapter 2

State of Art

2.1 Bron hos opy

Bron hos opy isa pro edure inwhi h a thin tube isthreaded through thenose or mouth into the

windpipe and lungs. This allows the lini ian to look inside a patient's airway for abnormalities

like blo kages, bleeding,tumors,or inammation.The lini ianoftentakessamples frominsidethe

lungs: biopsies, uid, or endobron hial brushing. The lini ian may useeither a rigid or a exible

bron hos ope.

Types

There aretwo ways to perform a bron hos opy, depending on whi h bron hos ope is used: a rigid

bron hos ope or a exible beropti bron hos ope. We will enter on exible bron hos ope, sin e,

today, themajorityof bron hos opies are performed using the exible beropti s ope be ause of

the improved patient omfortand redu ed use of anesthesia.For thesame reason, inour study,it

hasbeen de idedto modelthemovements ofthis kind ofbron hos ope.

Flexible beropti bron hos opy usesa beropti amera on aexible mounting to lookinsidethe

airway.Thiste hnique ismore omfortablefor thepatient and doesnotrequire general anesthesia,

although, in most ases, ons ious sedation (twilight sleep) is utilized. The bron hos ope an be

onsidered asadevi e with3DOF, asitis showninthepi ture2.1:

•

insertion/extra tion of thetube(A).

•

rotation alongthe axisofthe tube (D).

•

rotation (

±90

o

PSfragrepla ements A B C D D E Opti Fiber

Figure 2.1:Degrees of freedom of a bron hos ope. 1) translational motion forward and ba kward along the

D-axis generatedby pushing andpullingthe bron hos ope (B); 2)rotational motion around the

C-axisobtainedbyturningwheelA(the amera entralaxis,E, oin ides withDwhenthe wheel

is atitshome position); 3)rotational motion aroundthe D-axis produ edby the rotation of the

wholebron hos ope handle.

Thebron hos opeisinsertedthroughthenose(ormouth),downthethroat,andintotheairways

(see g.2.2). The bron hos ope an also be inserted into the airway through a breathing tube. A

exiblebron hos opeisalong,thintubeofopti albersthattransmitlight.Beforethepro edure,

thenoseandthroatarenumbedbyasprayofmedi inesquirtedintothoseareas.Thishelpsprevent

oughing and gagging when the bron hos ope is inserted. The vo al ords, windpipe and airways

arevisualized by alight and mini- amera situatedon thetip of thebron hos ope.Pre-medi ation

isgivenbeforethepro edure to relaxthepatient but not ause lossof ons iousness.

Asmentionedbefore,thenavigationinthetra heobron hialtreeand, onsequently,thediagnosis

anbeverydi ult:be auseoftheir dimension andlowexibility,standard bron hos opes annot

rea h all the zone of the lungs (above all the upperand boarder zones), so the do tor isnot able

to see dire tlyall the part of the lungs themselves. In the exploration withthe bron hos ope, the

do torhave notmanyreferen es thathelpshim tond therightpath towards thezoneof interest,

sothenavigation isverydi ult,sometimes.Newadvan edmethods have been introdu ed tohelp

navigatethebron hos opeandimprovediagnosti ability,su hasVirtualBron hos opy(whi hwill

be illustratedinthenext se tion),ele tromagneti navigation and endobron hial ultrasound.

ele tromag-Figure2.2: S hemeofexiblebron hos opy

Endobron hialultrasound anbe ombinedwithele tromagneti navigationbron hos opytoa hieve

a great te hnique for tumor dete tion: a study found that the ombination of the two te hniques

was able to su essfully diagnose lesions in

88%

of patients, whereas ele tromagneti navigation bron hos opyalone su essfullydiagnosed lesions in59% ofpatients[1℄.

2.2 Virtual Bron hos opy

Advan esin omputer te hnology have permitted development of virtual realityimages of the

tra- heobron hial tree using data sets derived from heli al CT of the hest. One method is virtual

bron hos opy (VB),whi h isa formof omputed tomography(CT).

Virtual bron hos opy is used for thes reening of airways inendoluminal neoplasias diagnosis,

for the evaluation of bron hial stenosisand to obtain a sort of map for the prospe tive beropti

bron hos opy.Thiste hnique doesnot need theinsertion of anyinstrument intheairways.It uses

spe ialx-rayequipmenttotake lear,detailedpi turesoftheinsideofthelungs.Its3D

re onstru -tionallowsabetterobservationoftherelationbetweentheairwaysandalltheotherorgansorother

stru tures. Ithasimproved the abilityof bron hos opyto dete t small, an erous lunglesionsthat

are ina essible by onventional methods, above all in peripheral zones, but it annot be used to

sampleneither ytologi nor histologi material.VBdoesnotrepresentanalternative tober-opti

physi ian'spra ti e, andisespe iallyusefulinevaluatingtheairwaybeforeinitialorsequential real

bron hos opies where an interventional pro edure might be onsidered. These images redu e the

han esofsigni antsurpriseso urringduringanypro edure,andallowthepatientandfamily to

be asfullyinformedofthe re ordsasthe physi ian is.

Complex Virtual Bron hos opy

Virtualbron hos opy an beusedtoa hieve dierentresultsanditisbe omingthe urrenthelping

tooltoimprovediagnosisandtofa ilitatethetaskofthebron hos opy.We anbasi allydistinguish

three omplex appli ation ofthe VB idea.

Therstofthese omplex virtualbron hos opy appli ationsusesthedata ontainedwithinthe

three-dimensional image for pro edures of lo alization within the mediastinum 1

and hilar

stru -tures 2

.

A simple example is understanding where a mediastinal lymph node is in relation to the

tra- heobron hial tree.In traditionalbron hos opy,the do tor an only seethe airway lumen and not

on the other side of the nontransparent bron hial epithelium, where the lymph node (that is the

targetforatransbron hialneedleaspirationor orebiopsy)maybe.Usingthevirtualbron hos opy

information obtainedfrom thethree-dimensional MDCT(MultiDete tor Computed Tomography)

dataset, it an be shown pre isely the same image as the real bron hos ope does, withgray-s ale

rendering.Withthesetwo imagessidebyside(thevirtual bron hos opi visualization andthereal

bron hos ope visualization of the same region), it is possible to make the virtual bron hial wall

transparent,sothat the stru turesthroughthebron hial wall an be visualized.

On e the lymph node is made visible on the virtual bron hos ope images, omparing the virtual

and real s ene,thebron hos opy operator an,with a great degree of onden e, know where the

targetlymphnode is.Histask anbe madeeasierusinga omputers riptfor theimage pro essing

to transfer the virtual image with the lymph node in question to overlay the real bron hos opi

image.

Earlyresults suggest a greater than 90% su ess rate for mediastinal and hilar lymph node

biop-sies.Clearly,theseearly su essesopen thepossibility ofsampling mediastinal stru turespre isely

1

Themediastinum isa non-delineatedgroupofstru tures inthe thorax, surroundedbyloose onne tivetissue.

Itisthe entral ompartmentofthethora i avity.It ontainstheheart,thegreatvesselsoftheheart,esophagus,

tra hea,phreni nerve, ardia nerve,thora i du t,thymus,andlymphnodesofthe entral hest.

2

Aboveandbehindthe ardia impressionisatriangulardepressionnamedthehilum,wherethestru tureswhi h

formtherootofthelung enterandleavethevis us.Thesein lude thepulmonaryartery,thesuperiorand inferior

pulmonaryveins,lymphati vesselsandthebron hus,withbron hialvesselssurroundingit.Apleuralsleeveis reated

aroundthesestru tures,wherethepleuraree ts, hangingfromvis eraltoparietal.Thesestru turespassthrough

generations of airways an now be automati ally extra ted and evaluated. A spe i software an

interrogate thethree-dimensional image dataset andprovide apathwaythatlinksthetra heawith

the lesion in the lungs peripheral region. This path, then, an be easily annulated and on e this

tube ispla ed,severalprobes an be pla edeitherto brushor biopsyoropti ally orbyultrasound

sample thelesionofinterest.

Using these sortsof approa hes, inearly studies,

80%

of peripheral lung lesions an be easily and satisfa torily sampled. These pulmonary pathnding appli ations are in lini al studies and are

beingdeveloped bya numberof ompaniesfor medi al appli ation.

The third advan ed virtual bron hos opy appli ation involves the targeting of the peripheral

lung for endobron hial valve-type pro edures in the management of pulmonary emphysema,

so- alled endobron hial lung volume redu tion surgery [3℄. Here, the information that is required is

the state of the lung paren hyma and the extent of emphysema inea h segmental region together

withtheanatomi ongurationand size ofthesubtendingairwaysegments. Inthis ase, itallows

operatinginterventionalpulmonologisttoplanmanythingsbeforethepro edure,su hasvalvesize,

howmanysegmentsadevi emightrequireto bepla ed,whetherthesegment lengthsareadequate

for thevalvepla ement.

In summary, it an be armed that VB is getting a more and more useful tool to be used

with a tual bron hos opy. Appli ation of advan ed virtual bron hos opy te hniques is an obvious

solution to shorten pro edure times, to improve a ura y of devi e pla ement, to redu e medi al

error, andto edu ate the patient and families.

2.3 Simulation Training

One important tool for the training of the bron hos opy operator is the bron hos opy simulator

(2.3). It's a virtual reality simulator with a model exible bron hos ope that looks and feels like

the real obje t. It intera ts with a small ben h top sensor whi h registers the s ope's movements

and translatesthis to athree-dimensional virtual airwaydisplayed ona omputers reen.The user

an progress throughvirtual lini als enarios re eiving instant feedba kfrom thesimulator whi h

measures parti ipant performan e. In ertainmodels there onstru tion of medi al environment is

Figure 2.3:The VR bron hos opy skill station in ludes a art, omputer, display, printer, proxy exible

bron hos ope, andkeyboard.Whilethe operator performsFlexibleFiberBron hos opeviatheleft

nares, realisti resistan eisfeltduring manipulationof the proxyexiblebron hos ope.

A bron hos opysimulator allows the do tors to repeatthe pro edural experien es at their own

pa e and pra ti e medi al pro edures by a urately dupli ating the look and the feel of real-life

situations. This improves patient safety by allowing the do tor to be ome better trained without

puttingpatientsatrisk.Thestudyin[4 ℄ omparedthebron hos opyskillsand ognitiveknowledge

of22 fellows whore eived standard bron hos opy training with22 fellowswho re eived additional

bron hos opytraining,in ludingsimulationbron hos opyandanonline urri ulum.Resultsshowed

thatfellows who re eived additional simulation training signi antly improved their bron hos opy

skillsanda elerated the a quisition ofskills ompared withthosewho re eived standard training.

Simulatorswillallowthisinitialtrainingtoo urinatime-e ientand ost-ee tivemanner.In

averyshortperiodoftime,fellows anbeexposedtoabroadrangeof asesthatree tvariationsin

patient anatomy,pathology,and physiology.Theseexer isesor pro edureswould otherwiserequire

numerousreal-life en ounters and ostly hoursof supervision.

Simulation Pro edure

Tobeginthebron hos opy, theuserinsertsthebron hos ope intotheroboti devi e.The

bron ho-s opefeelsanda tslike ana tualexibleberopti bron hos ope.Thedevi etra ksthemotionsof

theexible bron hos opeand reprodu esthe for es feltduring ana tual bron hos opi pro edure.

The proximal end of the interfa e devi e an be shaped like a human fa e with a port to insert

theexible bron hos ope through one of the nasal passages. The exible bron hos ope tra ks the

instru-Inadditiontobeinganatomi ally orre t,thevirtualpatientalsobehavesinarealisti manner.

The patient breathes, oughs, bleeds, and exhibits hanges in vital signs. Compli ations are

pro-grammedinsu haslido ainetoxi ity ausingthe patientto seizeordevelopa ardia arrhythmia.

Thesimulation omputersoftwarere ordsallthea tionsoftheuserandstoresthisinformation

inadatabase.Compli ationssu hashemorrhage,pneumothorax and ardiorespiratorydistress an

be programmed to o ur during a simulated ase. The trainee must then respond in a timely and

Chapter 3

Motivation

In omparison with real bron hos opy, Virtual Bron hos opy has some advantages. It is a

non-invasive pro edure that an visualizeareas ina essible to theexible bron hos ope. Virtual

bron- hos opyisabletoevaluate bron hialstenosisandobstru tion ausedbyboth endoluminal

pathol-ogy(tumor,mu us,foreignbodies)andexternal ompression(anatomi alstru tures,tumor,lymph

nodes), an be helpful in the preoperative planning of stent pla ement and an be used to

evalu-atesurgi al suturesafterlungtransplantations, lobe tomyor pneume tomy.Virtualbron hos opy,

then, is a very useful toolbut it provides only an exploration made by a amera passing through

points, whi h areestablished previously.Sin e all the ommands aregiven by keyboard or mouse,

this pro eduredoesnotallowanysortofintera tion withthetissuenor a orresponden ewiththe

real movements the do tor would do with the bron hos ope. Furthermore, sin e no intera tion is

provided, the virtual amera an goout of the tra heobron hial tree,whileexploring within.

Thebron hos opysimulator,whi h,ononeside,providesaveryrealisti environment

( onsider-ingfor efeeling,pro edureandinstruments),isnot verypra ti al,ontheotherhand,for aspe i

analysis on a parti ular patient: it provides only a range of ases that ree t variations in patient

anatomy,pathology,and physiology.Moreover, itdoes osta lot.

Therefore,there is theneed ofa VBsystemable to be navigated in a more realisti way.This

an be a hieved exploiting the hara teristi of hapti s devi esthat an provide thedo tor witha

ompleteplatformtoseeand,somehow,feelasifheweredoingarealbron hos opyonthatspe i

patient.

The natural target of this software will be bron hos opy experts, who will exploit the useful

hara teristi s ofthe VBjoined withamore ergonomi wayofdoing it.

Byusing VBwith hapti feedba k, medi al stais expe ted to arryout their tasks ina more

ur-thermore,thisfa tisalsoexpe tedto improvethepatient rea tiontothisoperation,sin eitwould

take less time and redu e the number of forward-ba kward movements of the bron hos ope in

bron his, whi hareveryannoying for the patient.

The present work is fo used on the development of a system able to allow people to navigate

insidethetra heobron hialtreeusinganhapti devi e.Sin eitispartofabiggerproje t,thiswork

also responds to the need of having a software that ould integrate all the tools developed inthe

wholeproje t:loadthe 3D-lungre onstru tionfromCT imagesandtheshortestpath al ulated to

getto thediseasedlungzoneand allowpeople to navigate the 3Ds ene usinga hapti devi e.

Thiswork intends to represent another littlestep to thepersonalization of health are, whi h

meanstousethepatient'sinternalstru tureandfun tionforthetrainingorpreplanningof omplex

diagnosti or therapeuti pro edures.Thisseemstobeafundamentally important hara teristi for

Chapter 4

Obje tives

Themain obje tive ofthiswork istomakea basi appli ation whi h,exploitingvirtualrealityand

hapti sdevi epotentialities, anbeusedto exe utevirtual bron hos opyofthatparti ular patient

in a more pra ti al and realisti way, using the same movements to move the amera within the

airways andre reating,somehow, thefeelingsofa realbron hos opy.Itis nottheaim ofthiswork

to build a omplex simulator.

To a hieve this obje tive, thiswork isdivided into two modules:

•

Navigation Module

Therstproblemthathastoberesolvedisthedeterminationofthehapti sdevi emovements

thatallowto ontrolthemotionsofthevirtualbron hos opeinthemostrealisti way,inother

wordsthe most a urate orresponden ebetween ameramotion ontrolledbyhapti sdevi e

andthe ameramotion ontrolledbybron os opehasto befound,disabling,ifitisne essary,

those devi eDOFnot present intherealbron os ope.

Thismodulemustalso opewiththerendering ofthe onta t for es andthemanagement of

the ollisions, inorderto avoidthe exitfromwithin thebron hial tube.

•

Visualization Module

The softwarealso requiresaGraphi s Interfa e thatpermits theuserto seethevirtual s ene

and explore it: to make the exploration more intuitive, the appli ation must provide the

graphi alrepresentationofthelungsbothasageneralviewandasseenfromthebron hos ope

tip, sothe user an alwaysknowinwhi hpositionhe isand where to go.

Another important obje tive is to reate a  ross-platform software, able to run on dierent

Chapter 5

Involved Hardware: Hapti Devi e

5.1 Hapti Devi es

Hapti devi es (or hapti al interfa es) are me hani al devi es that allow users to tou h, feel and

manipulate 3Dobje tsinavirtual environment orinatele-operatedsystem.Unliketheothermost

ommon omputerinterfa edevi es(mouse,keyboard orjoysti k)whi hareinput-only devi es,the

hapti devi es are input-output devi es, meaning that they tra k a user's physi al manipulations

(input) andproviderealisti tou handsensations oordinatedwithon-s reenevents(output):they

givethefor efeedba ktothesubje twhoisintera tingwithvirtualorremoteenvironments,giving,

somehow, the sensation ofbeingpresent inthes ene.

The word hapti derives from the Greek haptikos meaning being able to ome into onta t

with. Inhuman- omputer intera tion,hapti feedba kmeansbothta tile andfor e feedba k. The

term ta tile, or tou h feedba k isused to express the sensationsfelt by theskin. Ta tile feedba k

allows users tofeel things su h asthetextureof surfa esand vibration.For e feedba kreprodu es

dire tional for es that an result from solid boundaries, the weight of grasped virtual obje ts,

me hani al omplian e of an obje t and inertia. All these features enhan e the task performan e

and in reasethe realismofa simulation.

Consideringthehumansensorial hara teristi s,theyimposemu hfasterrefreshratesforhapti

feedba kthanforvisualfeedba k:forexample,ta tilesensorsintheskinrespondbesttovibrations

higher than

300

Hz.For thisreason HAPI 1

,andusuallyalso theother hapti renderingAPIs

man-agethehighprioritythreadsforhapti renderingat

1000

Hz[5℄.Thisorder-of-magnitudedieren e between hapti s and vision bandwidths requires that the hapti interfa e in orporate a dedi ated

ontroller. Sin e it is omputationally expensive to onvert en oder data to end ee tor position

and translate motor torques into dire tional for es,a hapti devi e will usually have its own

ded-i ated pro essor. This removes omputation osts asso iated with hapti s and the host omputer

andedi ateitspro essingpowertoappli ation requirements, su hasrenderinghigh-levelgraphi s

[6℄.

The hapti interfa es sold at present an be lassied as either ground-based devi es (for e

ree tingjoysti ks and linkage-based devi es) or body-based devi es (gloves, suits, exoskeletal

de-vi es).Themostpopulardesignonthemarketisalinkage-basedsystem,whi h onsistsofaroboti

armatta hedtoastylus.Thearmtra ksthepositionofthestylusandis apableofexertingafor e

onthetipof this stylus.

An alternative to a linkage-baseddevi e is one that istension-based. Insteadof applying for e

through links, ables are onne ted to the point of onta t in order to exert a for e. En oders

determinethelengthofea h able.Fromthisinformation,thepositionofagrip anbedetermined.

Motorsareusedto reatetensioninthe ables,whi hresultsinan appliedfor eatthegrip. Inthis

ase,ithasto be onsideredthe onewhi hbetterwouldrepresent thebron hos ope,soithadbeen

hosen alinked basedsystem.

Manykindofdevi es anbefoundamongthelinkage-basedsystems.Sin ethe ongurationof

one bron hos ope is determined by the tip's position and orientation, we entered on the 6 DOF

devi es. Among the brands, we de ided for PHANTOM by Sensable Te hnologies, whose devi es

were already used in our laboratory (PHANTOM Premium) and whi h was one of the rst in

ommer ingfor efeedba k devi esand, nowadays,is the most popular.

AmongthePHANTOMs there arethreedevi es that ould suit ourneeds:

•

Premium: 6DOFand for efeedba k for theposition andmomentum for theorientation.

•

Desktop: 6 DOFbut for e feedba k for only 3 joints (theposition ones),i.e. only three ele -tri a tuators to virtually x a point in a 3D spa e. Also his workspa e is smaller than the

Premium's one.

•

Omni: like the Desktop butwith lesspowerfor thefor e-feedba kand more fri tion.

Considering the degrees of freedom of the bron hos ope and above all the harateristi and

wanting to make the do tor feel the bron hos opy and, in the future, provide a sort of guidan e,

the6 DOF of Phantom Premium would be the best hoi e. The momentum given by this devi e

will in rease the reality of the appli ation and moreover it would allow to for e the user to adopt

Figure5.1: ComparisontablebetweenPhantomOmniandPhantomDesktop

to build a program on a expensive devi e if an a eptable result an be a hieved also with mu h

heaperdevi es (e.g.Phantom Omnithatis 25 times heaper than thePhantom Premium).

Asit an be seen on the omparison table (g. 5.1), Desktop and Omnibasi ally diers only

on thestrength of the for e feedba k. Sin e it wasseen that the for e provided by the Omni was

su ient,nallyPHANToM Omniwas hosenfor this appli ation.

5.2 PHANTOM Omni

PHANTOM Omni has 6 DOF: 3 digital en oders to read the position and

±5%

linearity poten-tiometers toreadthe stylusgimbalangles. Asmentionedbefore, it annotprovide torquebut only

Control Algorithm

TheOmni(astheotherPHANTOMdevi es)usesanimpedan e ontrolalgorithm:theusermoves

thedevi eanditrespondswithafor e,ifitisne essary.Inthiskindofalgorithmthe ontrolloop an

beresumedasfollow.The position sensorsdete tthemovement madewiththedevi ebytheuser,

thenthe hardware ontroller (driver) sends thetranslation information to thesoftware simulator,

whi h determines ifarea tion for eis required andits magnitude value. Thehost omputer sends

feedba k for es to the devi e. A tuators (motors within the devi e) apply these for es based on

mathemati al modelsthat simulatethe desired sensations.

For example, when simulatingthe feel of a rigid wall, the driverorders to themotors to stayin a

determined position and a tuators (motors within the devi e) exerts the for es needed to stay at

thatposition. The farther the wallis penetrated, theharder themotorspush ba kthedevi e.

Workspa e Denition

Nominal Worspa e (NW): it is the volume in front of the devi e, in whi h the manufa turer

guarantees the spe ied for e feedba k and pre ision. No physi s limitations forbid to exit this

volume,sin e itisonly denitionof partofthespa e andthejoint rangeallows widermovements.

For theOmnidevi eisa parallelepiped whosedimensions are

160 W × 120 H × 70 D [mm]

. Thenominal workspa e isdierent from thereal workspa e (RW) whi his thevolume

on-taining all the points rea hable by the devi e end ee tor, exploiting the maximum range of all

joints. The RW omprises marginal zones where the devi e behavior would not be a eptable for

ertainappli ations, thiswouldbe a questionto take are ofineveryprospe tive work.

5.3 Hapti Renderers

The hapti renderer's task (or hapti -rendering algorithm) is to ompute the orre t intera tion

between the representation of the hapti interfa e inthevirtual environment and theobje ts

pop-ulating that environment. Furthermore, these algorithms ensure that the hapti devi e orre tly

renders su hfor es on the humanoperator.

Following the ideaof professors Salisbury,Conti and Barbagli des ribed in[7 ℄, three parts an

be distinguishedina hapti -rendering algorithm:

•

Collision-dete tion algorithms dete t ollisions between obje ts and avatars 2

in the virtual

2

i allyoperateontheavatars'positions,thepositionsofallobje tsinthevirtualenvironment,

and the ollision state between avatars and virtual obje ts. Theirreturn values arenormally

for e andtorqueve torsthat areapplied at thedevi e-bodyinterfa e.

•

Control algorithms ommand the hapti devi e in su h a way that minimizes the error be-tween idealandappli ablefor es.Thesealgorithmstakeasinputthefor eandtorqueve tors

omputed by for e-response algorithms and return the values of the a tual for e and torque

ve torsthat willbe ommandedto thehapti devi e.

For example, HAPI (hapti rendering API whi h will be des ribed better in se tion 6) is a

librarythatallowtobuildhapti interfa es,givingmethodsandfun tionwhi himplementthethree

parts just mentioned. There are four hapti renderers available in this library: two are internally

implemented in HAPI and two use external hapti s libraries. Ea h of them an work better than

theothers on aspe i shape modelor ina spe i appli ation.

GodObje t renderer

The god-obje t algorithm was introdu ed by Zillers and Salisbury [8℄. It uses a point proxy and

supports ustom user dened surfa es. Moreover, it is Open Sour e and devi e independent but

ould have problems wththe on ave partofa urved surfa e.

Ruspini renderer

The RuspiniRenderer is based on the algorithm presented by Ruspini et al. in [9℄. It is dierent

from all the other renderers in HAPI in that it uses a sphere proxy making it possible to have

an intera tion obje t with a size instead of just a point. As the GodObje t, also this renderer is

Open sour e, devi eindependent and supportusersurfa es, butit is alittle slowerthan theother

renderers.

Chai3D

devi eindependent, butthispresentssomefallthroughproblems onmovingobje tsand itdoesnot

allowuser dened surfa es[10 ℄.

OpenHapti s

OpenHapti s is a proprietary hapti s library developed by SensAble Te hnologies. It uses a point

proxy basedapproa h and provides a stablehapti feedba k. It is however not very extendable in

termsofuserdenedsurfa esandonlyworkswithhapti sfromSensAbleTe hnologies, moreoveris

losedsour e.Amongtheprosofthisrenderer,we anmentiontheavailabilityofMagneti Surfa e 3

anda goodbehaviorwithmoving obje ts.

3

ItisaH3DAPIsurfa enodewithaparti ularhapti propertythatmakesthesurfa emagneti :thesurfa ehas

asortofmagneti eld(modeledlikeaspringfor e,whosespring onstant andtherangeofa tion anbeset)that

Chapter 6

Involved Software

6.1 CMake

Asitisdes ribedinthe o ialsite [11℄,CMake isanextensible,open-sour e, ross-platformmake

system that manages the build pro ess in an operating system and in a ompiler-independent

manner. CMake has many fun tionality: it an ompile sour e ode, generate wrappers, reate

libraries andbuildexe utablesinarbitrary ombinations. CMake alsosupportsstati and dynami

library builds.

Unlike many ross-platform systems, CMake is designed to be used in onjun tion with the

native buildenvironment.Moreover,CMake generatesa a he lethatisdesigned to beusedwith

agraphi aleditor.Forexample,whenCMakeruns,itlo atesin ludeles,libraries,andexe utables,

andmayen ounteroptionalbuilddire tives.Thisinformationisgatheredintothe a he,whi hmay

be hanged bytheuserprior to thegeneration ofthenative build les.

Itisveryeasy to use:thebuild pro essis ontrolled by reatingone or more onguration les

inea hsour e dire tory( alledCMakeLists.txtles)thatmakeuptheproje t.These onguration

les are usedto generate standard build les (e.g., makeles on Unix and proje ts/workspa es in

WindowsMSVC)whi hareemployedintheusualway.CMakeprovidesmanypre-dened ommands

but,ifneeded,userdened ommands anbemade.Furthermore,itispossibletoaddothermakele

generator(Unixand MSVC++ issupported urrently)fora parti ular ompiler/OS onguration.

6.2 Graphi Rendering: H3DAPI

For graphi rendering there are many solutions too. S ene-graph based API seemed to be a good

tool to make omplex appli ation more manageable, so we will enter on this kind of API. The

s ene graphstru ture uses a hierar hi al representation of a s ene withnodes to divide the world

into smaller sub- omponentswithits hara teristi sand propertiesset inspe i elds.

OpenInventorisanobje t-oriented3Dtoolkitoeringa omprehensive solutiontointera tive

graphi sprogramming problems. It an be dened thede fa to standard for 3D visualization and

visual simulation software in the engineering ommunity. It presents a programming model based

on a 3Ds ene database, in luding a ri h set of obje ts (su h as ubes, polygons, text, materials,

ameras, lights, tra kballs, handle boxes,3D viewers) and editors that speed up theprogramming

time.ThistoolkitisbuiltontopofOpenGL,providesastardardleformatfordatainter hangeand

it is ross-platform, window-system and platform independent. It allows animations and reating

newusers ustomized obje ts.

Coin3Disanotherexampleofhigh-level3Dgraphi stoolkitfordeveloping ross-platform

real-time3Dvisualization and visual simulation software. AsOpenInventor is, also Coin3D is built on

OpenGLand uses s ene graphdatastru tures to render3D graphi sinreal-time. Infa t, Coin3D

isfully ompatible withOpenInventor. Inaddition, itprovides 3Dsound,3Dtextures,and parallel

renderingonmultiplepro essors.Anotherinteresting featureistheseamlesslyintegration intheQt

developmentenvironment.Coin3D,whi hisonlyorientedto thegraphi rendering,isaverymu h

developed instrument tomanagegraphi rendering: itoersawide rangeoftoolsto managingthe

amera and reating several indipendent window with dierent point of view perfe tly integrated

withQt(thanksto the SoQt library 1

).

Both of these mentioned API only provide graphi -rendering, so in appli ations that require

hapti devi es,hapti renderinghastobedevelopedinparallel.Ifyouwanttouses ene-graphAPIs,

thesolution isto reate a dupli ation of the s ene-graph, one ointaining thehapti spe i ations

and the other for the graphi rendering. This would imply to keep the two in syn (Coin3D and

OpenInventor have allba ks me hanism to handle this, unless you have many dynami obje ts

movingaround)and mayimplydataredundan y (everyobje thastobedupli ated inboth

s ene-graph).

H3DAPIisan open-sour e, ross-platform, s ene-graph API.H3D iswritten entirely inC++

andusesOpenGLforgraphi srenderingandHAPIforhapti srendering.Itprovidesas ene-graph

APIthatmerge graphi sand hapti features:itperformsgraphi and hapti rendering froma

sin-gle s ene des ription. H3D leverages the de fa to industry standard hapti library OpenHapti s.

ThroughtheuseofHAPIthereisalsohapti srenderingsupportforseveralotherdevi esthatdoes

fun tionality inamodular way.

XML 4

. Extensible Markup Language, XML is the standard markup language used in a wide

variety of appli ations. The X3D le format is based on XML, and H3D omes with a full XML

parser for loading s ene-graph denitions.

OpenGL 5

. Open Graphi s Library, the ross-language, ross-platform standard for 3D

graph-i s.Today,all ommer ialgraphi spro essorssupportOpenGLa elerated renderingand OpenGL

rendering isavailable onnearly every knownoperating system.

STL - The Standard Template Library is a large olle tion of C++ templates that support

rapid development of highly e ient appli ations. It provides also a quite rapid development: by

ombining X3D, C++ and the s ripting language Python, H3Doers three ways of programming

appli ationsthat givesyou thebestofbothworlds- exe utionspeed where performan eis riti al,

and development speed where performan eis less riti al.

6.3 Hapti s Rendering Engine: HAPI

HAPI is an open-sour e, ross-platform, hapti s rendering engine written entirely in C++. It is

devi e-independent and supports multiple urrently available ommer ial hapti s devi es. This

means that the appli ation an be written just on e and no ode needs to be modied to use

another devi e. It gives the possibility to hoose between dierent rendering algorithms, dierent

for e ee ts and several kinds of surfa es to reate the feeling that you want or reate your own

ustommade ee ts. HAPIhasbeen designed to be highlymodular and easilyextendable.

Furthermore, HAPI is integrated in the H3DAPI. This make the development faster, sin e both

graphi and hapti rendering an be a hieved bybuildingjustone s ene-graph.

AsotheralternativestotheuseofHAPI,thereareGHOSTorthemorere entOpenHapti s,both

2

ThisversionofH3Dsupportsthefollowingdevi es:PhantomDevi e,For eDimensionDevi eandNovintFal on.

3

http://www.web3d.org

4

http://www.w3.org/XML

developed bySensAble. Ghost isthe original API and uses graph stru ture, but does not perform

any graphi al rendering (it has allba ks to allow users own graphi al rendering). OpenHapti s is

theSensAblenewestsolution forhapti rendering. Itismorepowerful thanGHOST,but sin eitis

not based on graphstru ture it an be more di ult and not so qui k to work with. AsGHOST,

OpenHapti s does not deal withgraphi rendering. Among other things, the OpenHapti s toolkit

in ludes:

•

Qui kHapti s, a mi roAPIthatmakesiteasy to writenewhapti appli ationsoradd hapti feature to existingappli ations;

•

Hapti Devi e API (HDAPI),whi hprovidesalow-levela esstothehapti devi e(enabling to renderfor es dire tlyoering ontrolover onguring theruntimebehaviorofthedrivers)

and provides onvenient utility featuresanddebugging aids;

•

Hapti Library API (HLAPI) provides high-level hapti rendering and is designed to be fa-miliar toOpenGLAPIprogrammers. Itallowssigni antreuseofexistingOpenGL odeand

simplies syn hronization ofthe hapti s andgraphi sthreads.

6.4 Framework for software development: Qt

Lookingforaframeworkfordeveloping ross-platformGUIappli ation,thetwobestsolutionsseems

tobewxWidgetsandQt. Tosatisfytheobje tives,thesetwoC++frameworksareverysimilarand

oer,more or less, the same possibilities. The opinionson them are very dis ordant depending on

theprogrammer.

Anotherpoint isthatusing wxWidgetsyouhaveto write themakele byyour own(that ould

benot thateasy),whereas inQt QMake isverysimple touse, sothere'sno need toeditMakeFiles

manually; usingCMake thisis not arelevant dieren e forthis work.

Qtintrodu esaninnovativealternativeforinter-obje t ommuni ation, alledsignalsandslots,

thatrepla estheoldandunsafe allba k te hniqueused inmanylega yframeworks:Qt

automati- ally onne tssignalstoslotsbasedonthenames.Moreover,Qtprovidesaveryusefulandpolished

IDE (QtCreator) tailored to the needs of Qt developers. It in ludes C++ ode editor, integrated

GUIlayout and forms designer, proje t and build management tools,integrated, ontext-sensitive

helpsystem,visual debugger, rapid ode navigation toolsand supportsmultiple platforms.

Both of them provide another very useful tool for graphi ally designing user interfa e: wxBuilder

forwxWidgetsand QtDesigner for Qt.

Chapter 7

Hapti s Interfa e

The main obje tive of this module is to allow the navigation within a generi tra heobron hial

tree usinga hapti devi easifit werea bron hos ope, seeing thesame s ene andfeelingthesame

sensationsasiftheuserwere arryingoutarealbron hos opy.Theideaisto allowtheusertomove

a amera (setting the orientation and position) with the same movements as if he were handling

a real bron hos ope. This means nding a good orresponden e between the movements that the

do torhastomakewiththe bron hos opeto explorethetra heobron hial tubeandthemovements

availablewiththe Phantom Omnihapti devi e.

PSfragrepla ements

A

B

C

4

5

6

head link1 link 2 stylus

PSfragrepla ements

θ

1

PSfragrepla ements

θ

2

X

W

Y

W

Z

W

PSfragrepla ements

θ

3

Figure 7.2:Angles of the joints of the Phantom Omni. The red frame represents the origin of the devi e

world oordinates.

Before des ribing the main features of the navigation and how the Phantom Omni is used for

simulating the bron hos ope, it is ne essary to spe ify some global referen es of the devi e itself.

Asit an be seeningure7.1, the Phantom Omnihasa pen-liketip alledstylus,whi h isxedto

thebase by two links and a round head. This part rotates around the

A

axis (

θ

1

), while the rst linkrotatesaround

B

(

θ

2

) andthe se ond linkaround

C

(

θ

3

)axis whi h is parallelto

B

(g.7.2). Thelastthreejoints(gimbal elements), markedwithnumbers

4

,

5

and

6

,allowthestylusto rotate around the orthogonal axis system entered in the joint

5

, and the onsequent orientation (with respe tto theworld oordinates

(X

W

, Y

W

, Z

W

)

) an be readfrom thedevi e.

The endpoint lo ation of the physi al hapti interfa e as sensed by en oders is alled Hapti

Interfa e Position (HIP).Inproxy-basedrenderingalgorithms(su hasthosein ludedinHAPIand

des ribedinse tion5.3),theHIP(andthewholehapti sdevi ewithit)hasavirtualrepresentation

alledproxy. The proxy follows thepositionof the devi e, but it annotpass through theobje ts'

surfa es: when a shape is tou hed, the proxy stays on the surfa e, even though the hapti devi e

a tuallyhaspenetratedthesurfa e.For esarethengeneratedto bringthehapti devi eoutofthe

surfa etowardstheproxy.Whentheusermovesthehapti sdevi etheproxyfollowsthemovement

butonthe surfa e.Thekindoffeedba kfor es,themovementof theproxy and,therefore, theway

Inthedefaultsituation,usingH3D,thedevi epositionandorientation areidenti ally

transmit-tedto thevirtual world, sotheprobe (ortra ker)positionandorientation oin ide withthedevi e

position and orientation, respe tively.H3DAPI supplies theuser withtwo elements (a matrix and

a rotation ve tor) that an modify thewaythe devi e position and orientation are translated in

thevirtual world. These two elements arepositionCalibration and orientationCalibration 1

.

positionCalibration:Itisaeldthata eptsaMatrix4f 2

,a

4 × 4

matrixwhi hhastheform of atransform matrix:

T

calib

=













P

x

R

P

y

P

z

0

0

0

s













where

R

denesa rotation,

P

x

, P

y

, P

z

,

dene atranslation ve torand

s

as ale fa tor.Itworksas follows:

P

probe

= T

calib

· P

device

,

with

P

device

=













P

x

dev

P

y

dev

P

z

dev

1













where

P

x

dev, P

y

dev

, P

z

dev

dene the devi e position as obtained by the en oders. The important thing to underline here is that positionCalibration inuen es just the way the devi e position

is translated in the virtual world and does not ae t the orientation. The rotational part of the

matrix(ifitisnottheidentity)willtransformadevi etranslationalmovementin

+X

,forexample, in a movement of the virtual HIP that does not o ur on a horizontal line. It will be, instead, a

movementonastraightlineorientedasdes ribedbytherotationalpart, orrespondingtothevirtual

world

X

-axis. Moreover, this rotational part does not inuen e the way the devi e orientation is translated into the virtual world. For example, ifthe viewpoint orientation were asso iated to the

one of the stylus, the orientationCalibrationwerethe identity (default situation)and the user

werehandlingthe stylussoasto have thesameorientationastheworld,theloadeds enewillshow

the world in the normal position:

Y

-axis pointing upward,

X

-axis pointing to the right and the

Z

-axispointingout of the s reen. 1

positionCalibrationandorientationCalibrationaretwoeldsofthenodePhantomDevi e,whi hisanodeofthe

H3DAPI.

2

orientationCalibration: It is the eld that provides the alibration of the orientation. This

elda epts obje tsoftypeRotation,whi hisave tordeninganorientation inaangle-axisway.

Thisve tor has four elements: the rst three des ribe theaxis around whi h the rotation is done

and the fourth element is the angle in radians. In H3D Rotation is also a fun tion that takes as

inputaMatrix3f 3

and returnsa Rotation ve tor.Considering orientationCalibrationa matrix

( alled

R

calib

),it worksasfollows:

R

probe

= R

calib

· R

dev

(7.1)

where

R

probe

istheorientation oftheprobeand

R

dev

istheorientationofthedevi easitisread bytheen oders.

As said for positionCalibration, orientationCalibration only ae ts the way the devi e

orientation istranslated into the probeorientation. Inotherwords, ifthedevi eismoved ina

+X

dire tion and its positionCalibrationis theidentitymatrix, thenthe probe will translate along

theworld

X

-axis (irrespe tive of the valuesof orientationCalibration

R

calib

).

7.1 Bron hos ope Modeling

Asseen before, in a bron hos ope we an distinguish three degrees of freedom (g. 2.1):

forward-ba kward movement, the rotation along the tube entral axis and the rotation of the tip of a

±90

o

range. The rst and the se ond a tions are performed by the do tor a ting dire tly on the

bron hos opehandleandtube,pushingandpullingorrotatingthebron hos opetubeitselfthrough

thetra heobron hial tree. The tip an be rotated just in one plane whose orientation depends on

the orientation of the tube. The tip rotation is provided by a wheel (element (B) in the g. 2.1)

that an be rotated tillthe tiprea hes anorientation of

±90

o

with respe tto thebron hos ope

D

axis.

Considering how the Phantom Omni is handled by the user and wanting to reprodu e the

movements of the bron hos ope (as mu h similar as possible), it has been de ided to adopt the

following orresponden e between the threebasi movements:

•

The forward-ba kward movement made by pushing or pulling the tube will be performed by translating the devi e tip along thedevi e

Y

-axis.A negative value will orrespond to a forward movement,while apositivevalue will orrespondto a ba kward one.

•

The rotationalongthe tube entral axiswill orrespondto therotationofthe

6

th

jointofthe

devi e.

3

7.2 Navigation System

The basi idea is to position the amera in the proxy and make it move with it. The movements

done withthe devi etomakethe ameramove willbeasmu hsimilaraspossible totheonesdone

when handling areal bron hos ope.

Asexplained in hapter 6,H3Dhasbeen usedboth todes ribethegraphi s ene andto render

thehapti sproperties.TheS eneisdes ribedwithaXML-likegraph(X3Dle).Thenode<S ene>

is the highest node in a X3D graph ( no other X3D node an ontain s ene) and ontains all

the other nodes des ribing the shapes and hapti s properties. To render the s ene, a window

in-heriting fromH3DWindowNodemust be reated.Tothis kindof window, whoseimplementation will

be expounded later on, in hapter 8, a viewpoint an be asso iated, whi h represents the

am-era of the H3D s ene. This Viewpoint node inherits from the lass (whi h represents a node)

X3DViewpointNode, whi h, in turn, ontains two elds named position and orientation. They

spe ify,asthe namestell,thepositionandtheorientation oftheviewpoint,respe tively.The

view-pointnodehasa frame( oordinatesystem)asso iatedto it.Perdefaulttheviewerisonthe

Z

-axis looking downthe

−Z

-axis toward theorigin with

+X

to therightand

+Y

straight up(g. 7.3).

PSfragrepla ements

X

Y

Z

Figure7.3: Defaultviewpointorientation drawnon theappli ation mainwindow.

The ongurationofthebron hos opetipisdened byapositionand anorientation.Tryingto

PSfragrepla ements

A

B

C

D

D

E

Opti Fiber FlexibleTip

(a)Drawing of a real general endos ope showing

theDOFandtheaxisof rotation.

X

′′

Y

′′

Z

′′

X

′

Y

′

Z

′

X Y

Y

Z

base tip 90 o

L

(b) Simplied representation of the bron hos ope

tip. The frame

(X

′′

_{, Y}

′′

_{, Z}

′′

₎

is the same as

(X

′

_{, Y}

′

_{, Z}

′

₎

when the tip is turned in the opposite

dire tion(

+90

◦

).

Figure7.4: Degreesof freedom andreferen eframesinthe onsideredmodel of bron hos ope.

and the other is ontrolling the amera from the base of the tip (des ribed in se tion 7.2.2), that

isthebeginning of the exiblebron hos ope nalpart(thatone that an bemoved from

−90

o

till

+90

o

,seegure7.4).Bothof themhave been implementedand thenalde ision about whi h one

isthebest willbe letto thedo tors whowill testthe appli ation.

Lookingatgure7.4it anbebetterseenhowthebron hos opeismodeledandit anbebetter

understood what ea h of the implementations arereferringto. Figure7.4(b) showsthe two frames

asso iated to the bron hos ope tip whi h the two dierent implementation of amera ontrolling

referto.Thenavigationimplemented ontrollingdire tlythe amera(se tion7.2.1)willrefertothe

blue oordinatesystem

(X

′

_{, Y}

′

_{, Z}

′

₎

,whereastheotherimplementation(se tion7.2.2)willrefertothe

greenone

(X, Y, Z)

.Furthermore,betweenthetwoimages(7.4(a)and7.4(b))some orresponden es anbe underlined(leavingout theorientation):

•

The

D

-axis oin ideswiththe

Z

-axiswhereasthe

E

-axis orrespondstothe

Z

′

-axis.Thistwo

axes have thesame orientation only ifthetipis totallyextended.

• C

-axis refers to

X

-axis and orresponds to the beginning of the bron ho ope exible tip whi h anberotated, asalready mentioned,inarangeof

±90

o

.Inotherwords, itmarksthe

4 Compute_ ontrols(); // Dz: bron hos ope advan e, angles of the rotations to apply to the amera 5 Update_viewpoint(); 6 Update_ alibration(); 7 Compute_for es(); 8 9 }

7.2.1 Moving dire tly the amera

The rstpossibleimplementation of theHapti s_loop() isusing thehapti devi emovementsto

ontrol dire tlythe positionand orientation ofthe amera.

Rotation Movement

Whenaskingthedevi efortheorientation,thedevi ereturnsaRotationobje twhi h anbeeasily

onvertedto a

3 × 3

matrix usingthefun tionsofH3D.Thismatrixdenes arotationbetween the xedframeofthedevi eandtheframeasso iatedto theHIP(

(X

W

, Y

W

, Z

W

)

and

(X, Y, Z)

frames of pi ture7.5, respe tively), whi h dependsonthe joints

4

,

5

and

6

(g.7.1).

Exploring as ene witha ying amera ontrolled byan external devi e an present some

di- ulties. The problem isthat the devi ereads the values of position and orientation with referen e

to hisphysi al world framewhi h,ina defaultsituation, oin ides withthes eneworld frame.So,

when the amera and the s ene world frame orientations are not the same, the amera will not

respond to the devi e input as expe ted: when the amera is rotated around Z-axis,for example,

its top stops oin iding with thetop of thes ene world. Inthis ase, if theobserverwants to look

upward (withrespe tto what heislooking),he will spontaneously move thestylustipup but this

movementwillnotbetranslatedtothe ameralookingupward,be auseit orrespondstoanupward

movement (positive rotation along

X

) with respe t to the s ene world oordinates. This problem is a entuated when the 3D s ene has no referen e points to distinguish whi h are the s ene top

and bottom or what is right and left. A tra heobron hial tree exploration an be onsidered su h

theexpe ted amera movements. That an be done byapplyingtheinnitesimal variation inthe

devi eorientation (every lo ksignal)to therotation ofthea tual amera frame.

Before ontinuing, a lari ation hasto be madeonthemeaningof thejoint angles.Thevalues

readby theen oders ( alled gimbal angles) arethe absolute angles of thejointsbut do not dene

the devi e orientation (

θ

5

and

θ

′

5

in gure 7.5). The dieren e an be seen in gure 7.5: doing a translation in world

Y

-axis (where only the devi e position is supposed to hange, but not its orientation) will modify the gimbal angle of the

5

th

joint but the orientation, orre tly, will not

hange.Fromnowon,ifnootherthingisspe ied,whenitwillbe talkingabout hangesinthe

5

th

or

6

th

joint,itwill bereferring to hanges that ause avariation inthe devi eorientation.

Y’

PSfragrepla ements

X

X

Y

Y

Z

Z

X

W

Y

W

Z

W

θ

5

θ

′

5

Figure 7.5:Visualrepresentationofthe dieren ebetweenthegimbalangles(

θ

5

)andthe devi eorientation. In a

+Y

translation hangethe value ofthe gimbal anglesbutnotthe orientation.

The orresponden e between movementsof the devi eandthose ofthe amera hastotakeinto

onsideration the DOF of the real bron hos ope. That being so, the rotation of the

6

th

joint an

Summarizing all what has been told, the aim here is that a rotation in the

5

joint always orresponds to a rotation about the amera

X

-axis and that a rotation in the

6

th

joint always

orrespondstoarotationaboutthe amera

Z

-axis.Thesetworeadanglesdonot orresponddire tly to the devi eorientation about

X

and

Z

-axis,be ause a rotationof the

5

th

ae tstheorientation

dependingon thepositionofthe

6

th

one. Figure7.6 explainsbetterthis issue.The

(X, Y, Z)

frame is the one related with the amera and

(X

W

, Y

W

, Z

W

)

frame is the world frame, whi h has been drawn twi e with the origin oin ident with that of the HIP to see better the orientation whi h

relates thetwo referen esystems.It an be usefulto rememberthatthe amera lookstowards

−Z

, with

+Y

pointingupand

+X

pointingright.In ase

1

themovementof

5

hangetheorientation of the amera upanddown, thatisarotationaround the amera

X

-axis.Whenthejoint

6

isrotated

−90

o

,therotationof

5

provokesarotationaround the

Y

-axis,whi hmeansaright-left hangingin theorientation ofthe amera,whi h isnot theexpe ted behaviour.

Asolutiontothisproblemisto al ulatetheorientationofthedevi easamatrix andfromthis

al ulate the rotation angles with respe t to the xed axis. After that, it will be su ient to use

therotationangles aroundthe

X

W

and

Z

W

to rotate the amera frame (whi hisasso iated tothe probe)aroundits urrent

X

-axis and

Z

-axis,respe tively.

To a hieve this,the rst stepis ndingthe angles withrespe tto theworld axis. Dening the

rotation matrix:

R =









r

11

r

12

r

13

r

21

r

22

r

23

r

31

r

32

r

33









= R

x

(ϕ) R

y

(ϑ) R

z

(ψ),

what iswantedto ndarethe values oftherotation angles

ψ

and

ϕ

around

Z

and

X

axis, respe -tively.Theseare:

ϕ = − arctan 2(r

12

, r

11

)

(7.2)

ψ = − arctan 2(r

23

, r

33

)

(7.3)

where

arctan 2(y, x)

istheangleinradiansbetweenthepositive

X

-axisofaplaneandthepoint given bythe oordinates

(x, y)

on it.

Whenthe devi eis asked to return its orientation, itreturnsa rotationmatrix ofthis kind:

PSfragrepla ements

X

X

Y

Y

Z

Z

X

W

X

W

Y

W

Y

W

Z

W

Z

W

1

2

Figure 7.6:The ee tofthe jointvalues onthe orientation. Whenthe

6

th

isrotated

−90

o the

5

th

jointwill

rotatearoundthe world

X

-axis(theredone)whi h orrespondstothe HIP

Y

-axis.

Itispossibleto apply(7.2) and(7.3) to gettheangles

γ

and

α

,whi h dene theorientation ofthe HIPwithrespe tto the world oordinates

4

.

On e these angles values are found their variations have to be al ulated to rotate the amera

inthe right way. So,when one of these angles hanges by an innitesimal

∆

5

, this rotation of

∆

alongone axiswill be applied to the a tual amera frame withrespe tto the orrespondingaxis.

Thereareotherfeaturestotakeintoa ount.Oneimportantpointisthe

5

th

jointwhi h ontrols

the rotation of the wheel, rotating the amera

90

o

up or

90

o

down (around the

C

-axis of the gure7.4(a)). When the do tor wantsto turn right or left,he hasto orientate thebron ho ope in

4

The HIPreferen e system isleft-handed systemwiththe

X

-axisoppositewithrespe tto theonedrawnhere. Forreasonsof learness,ithasbeende idedtousearight-handedframeforthemodeling hangingthesignof

α

in theimplementation.

(using the devi eto ontrol dire tlythe amera), ithas been de ided to rotate the amera on this

planebyarate ontrol.Therotationangle about amera

X

-axis(

γ

)is in reased(orde reased) by aquantityequalto thedieren ebetween the urrentdevi eangleandareferen eanglepreviously

set. Considering the waythe bron hos ope ishandled (see gure 2.3),this referen eangle was set

to

45

o

.

Implementing all these features, the movements made during the exploration will better tra e

the realmovement. Infa t, when theuser wantsto turn right or leftbe ause he getsto a fork,he

hastoturnthe sixthjointtillhe obtainsthetwobron hial tubesinaverti alplane, thenhe moves

theHIP upor down(depending on whi h bron hus he wants to explore) rotatingthe

5

th

joint,till

the amera gets to the needed orientation to enter the bron hus. Subsequently, he has to put the

stylus in the referen e positionto make the amera stop rotating and be able to go straight, asif

he wasrotating thewheelba kto theoriginalposition.

In light of what has been shown up to this point, the wanted orientation alibration matrix

is the result of several innitesimalrotations applied to an a umulated rotation matrix. In other

words, theorientation matrix at instant

i

is:

R

i

= R

i−1

· R

z

(∆ α) · R

x

(∆ γ),

(7.4)

where

R

_i−1

istheorientation matrix ofthe amera atinstant

i − 1

∆ γ = γ

i

− γ

i−1

and

∆ α = α

i

− α

i−1

.

Considering the way

γ

isin remented,

∆ γ

an be writtenas

∆ γ = π/4 − γ

i

.

Sin etheaimisalsotointera twiththevirtuals enetheobserverislookingatand,forexample,

tou hthebron hus,theproxyhastobeinthesamepositionandsameorientationastheviewpoint.

With regard to theorientation, this means setting the orientationCalibration eld inorder to

obtain in(7.1):

R

probe

= R

i

this an be madebysetting:

This makes theprobe (and so the proxy, if there are no ollisions) and the viewpoint to have

thesame orientation.

Finally,ithastobetakeninto onsiderationalsothefa tthatthebron hos opetip annotbend

more than

90

o

in both sides (g. 7.4). Every time the angle hanges, the appli ation will ontrol

thattherotationangle aroundthe lo al

X

-axis neverex eeds

±90

◦

.

Insertion Movement

Theinsertionmovement impliesamodi ationofthe positionof the amera (positionedonthetip

ofthetube),whi h anbe onsideredtheviewpointofthereals ene.Imaginingthereal amerawith

thesame frameofthe viewpoint,thepushingmovement ausesatranslationof thetiptowardsthe

amera

−Z

-axis.So,knowingtheorientation ofthe amera,itispossibletotranslatetheviewpoint inthe orre t dire tion.To make the viewpoint move following the hapti devi e movements, it is

hanged withthepositionof the proxy.

A onsiderationaboutthe workspa ehastobedone,beforedes ribing indetailsthe

implemen-tationofthispart.Asithasbeen showedinpreviousse tions,thereareseveralkindsofworkspa e.

When building a hapti display system, the optimal system is theone whi h is able to make the

appli ation workspa e(AW) 6

oin idewiththatzoneinwhi h thehapti devi e provideits better

performan e.

Inthis ase,theappli ationworkspa eismu hbiggerthanthenominal(NW)andtherealworkspa e

(RW),sosomethingisneeded torea h and be abletonavigateall theAW.There an be two

solu-tion:use amouse-jump to translate theNWinanother volume spa e of theAWor use thehapti

devi eto establisha velo ity value and translate the probe and theNW where needed, inthe

ap-pli ationvolume.

These ond solutionhasbeen hosen, be ause this one seemsto bemore omfortable ompared to

thehundreds ofmouse-jumpsneeded to explorethewholetra heobron hial tree.

The insertion movement is provided by moving thestylusup (ba kward) and down (forward)

the

Y

-axis.At every lo k y le, thedevi eis askedto readthepositionvalue onthe

Y

-axis.This readvalueis taken asthe linear velo ityof theviewpoint (VP): if itis negative theviewpoint will

translateinits

−Z

-axisdire tionandifitispositiveitwillgo ba kward (inthe

+Z

-axisdire tion). Inotherwords, the devi e

Y

position orresponds toan in rement (

∆ z

) inthe amera

Z

-axis. Thisbehaviourisperformedbypassing(every lo kloop)totheVPpositionthetipposition(where

where

R

probe

is the tra ker (or probe) orientation,

∆ z

isthe linear in rement al ulated from the devi easexplainedbeforeand

P

camera

isthepositionofthe amera,whi h oin ideswiththeoneof thetip.Sin ethe amerapositionandtheprobepositionhaveto oin ide,thepositionCalibration

matrix hasto beset to:

T

calib

i

=













I

P

camera

i

0 0 0

1













·













I

P

dev

i

0 0 0

1













−1

(7.7) where

P

camera

i

and

P

dev

i

arethe amera anddevi e position ve tor atinstant

i

,respe tively. The real bron hos ope an only translate along its longitudinal axis, sin e, on e inserted, the

tra hea or the bron hus donot enable movementson theplane perpendi ular to thebron hos ope

tube.To render alsothis hara teristi , thedevi emovement alongits

X

-axis hasbeen blo ked by sending itafor eproportionaltothedevi e

X

- oordinatesoasto bringthedevi etothe

X

-origin. To make the pro edure learer, the Hapti s_loop an be summarized as follows (in

Update_viewpoint()

{

R

camera

i

= R

camera

_i−1

· R

z

(∆ α) · R

x

(∆ γ)

// fromeq. (7.4)

P

camera

i

= P

camera

_i−1

+ R

probe

·









0

0

∆ z









//from eq.(7.6) } Update_ alibration() {

R

calib

= R

i

· [R

dev

]

−1

// fromeq. (7.5)

T

calib

i

=













I

P

camera

i

0 0 0

1













·













I

P

dev

i

0 0 0

1













−1

//from eq. (7.7) } Compute_for es() {

// Constraint alongx-axis

// sendto thedevi eafor e proportionalto its X-position

omputeFx(P_devi e.x)

// ifthe surfa enode per eivesa onta t, it sendsa for e

//alongdevi e Y-axis topush the amera ba kward

if (surfa e.isTou hed()) ompute_Fy()