30 July 2021
Original Citation:
Virtual reality framework for editing and exploring medial axis representations of nanometric scale neural structures
Published version:
DOI:10.1016/j.cag.2020.05.024
Terms of use:
Open Access
(Article begins on next page)
Anyone can freely access the full text of works made available as "Open Access". Works made available under a
Creative Commons license can be used according to the terms and conditions of said license. Use of all other works
requires consent of the right holder (author or publisher) if not exempted from copyright protection by the applicable law.
Availability:
This is the author's manuscript
ARTICLE
IN
PRESS
JID:CAG [m5G;June26,2020;16:48]
Computers & Graphics xxx (xxxx) xxx
ContentslistsavailableatScienceDirect
Computers
&
Graphics
journalhomepage:www.elsevier.com/locate/cag
Special
Section
on
STAG
2019
Virtual
reality
framework
for
editing
and
exploring
medial
axis
representations
of
nanometric
scale
neural
structures
Daniya
Boges
a,
MarcoAgus
b,∗,
RonellSicat
c,
PierreJ.
Magistretti
a,
MarkusHadwiger
c,
Q1
Corrado
Calì
da Biological and Environmental Science and Engineering Division, King Abdullah University of Science and Technology, Thuwal, 23955-6900, Saudi Arabia b Visual Computing Group, Center for Advanced Studies, Research and Development in Sardinia (CRS4), Cagliari, Italy
c Visual Computing Center, King Abdullah University of Science and Technology, Thuwal, 23955-6900, Saudi Arabia
d Department of Neuroscience ”Rita Levi Montalcini”, Neuroscience Institute ”Cavalieri Ottolenghi”, University of Turin, Turin 10043, Italy
a
r
t
i
c
l
e
i
n
f
o
Article history: Received 14 February 2020 Revised 21 April 2020 Accepted 12 May 2020 Available online xxx Keywords: Ultrastructural analysis Medial axis representation Immersive environments Virtual reality in neurosciencea
b
s
t
r
a
c
t
Wepresent anovelvirtualreality(VR)basedframework forthe exploratoryanalysis ofnanoscale3D reconstructions ofcellular structures acquiredfrom rodent brain samples throughserial electron mi-croscopy.Thesystemisspecificallytargetedonmedialaxisrepresentations(skeletons)ofbranchedand tubularstructures ofcellular shapes,and itisdesignedfor providingto domainscientists:i)effective andfastsemi-automaticinterfacesfortracingskeletonsdirectlyonsurface-basedrepresentationsofcells andstructures,ii)fasttoolsforproofreading,i.e.,correctingandeditingofsemi-automaticallyconstructed skeletonrepresentations,andiii)naturalmethods forinteractiveexploration,i.e., measuring,comparing, andanalyzinggeometricfeaturesrelatedtocellularstructuresbasedonmedialaxisrepresentations. Neu-roscientistscurrentlyusethesystemforperformingmorphologystudiesonsparsereconstructionsofglial cellsandneuronsextractedfromasampleofthesomatosensorycortexofajuvenilerat.Theframework runsinastandardPCandhasbeentestedontwodifferentdisplayandinteractionsetups:PC-tethered stereoscopichead-mounteddisplay(HMD)with3Dcontrollersandtrackingsensors,andalargedisplay wallwithastandardgamepad controller.Wereportonauserstudy thatwecarriedoutforanalyzing userperformanceondifferenttasksusingthesetwosetups.
© 2020TheAuthor(s).PublishedbyElsevierLtd. ThisisanopenaccessarticleundertheCCBY-NC-NDlicense. (http://creativecommons.org/licenses/by-nc-nd/4.0/)
1. Introduction 1
The brain cells, together with their processes, are complex 2
three-dimensional structures, and improving the visual under-3
standingoftherelationshipsbetweenmorphologicalfeatures and 4
functionalaspects ofthesecellsisofprimary importanceto neu-5
roscientists.Therecentprogressindigitalacquisitionandanalysis 6
ofbiologicalsamples,e.g.,braintissues,isofferingunprecedented 7
possibilitiesofinsightsforneuroscientists.Forinstance,automated 8
serialsectionelectronmicroscopy(3DEM)provideselectron micro-9
graphsthatcanreacharesolutionofananometerperpixel, there-10
forerevealingfeatures rangingfromfull structuralcellular details 11
such asaxons,dendrites, andsynapses (the socalled“neuropil”), 12
to smallerintracellular organelles likesynaptic vesicles. However, 13
∗ Corresponding author.
E-mail addresses: magus@hbku.edu.qa , magus@crs4.it (M. Agus),
corrado.cali@unito.it (C. Calì).
neuroscientists still require effective tools and applications to 14 handlethislargeandcomplexdata.Morphologydataatnanoscale 15 resolutionprovide domain scientistsfundamental informationfor 16 understanding neural processes and interaction between cellular 17 structures [1]. Quantifications have particular relevance when 18 extracted data are used to infer parameters allowing mathemat- 19 ical modelization of biological processes [2,3]. Furthermore, the 20 challenge of making qualitative and quantitative assessments of 21 complex and visually occluded individual cellular structures, or 22 groupsofthem,isbeginningtoattractneuroscientiststowardsthe 23 use of immersive visualization paradigms. Hence, during recent 24 years,variouslaboratoriespioneeredtheuseofvirtualreality(VR) 25 insupporting electron microscopy (EM)structural analysis[4–6]. 26 However, previouspipelines were engineeredaround theneed of 27 exploratory analysis of brain structures for specific morphology 28 studies[4],or neuroenergetics investigations [5,7].More recently, 29 theneedformoreefficientextractionoffeatures ofbranch-based 30 whole cell structures, either for quantification and classification 31
https://doi.org/10.1016/j.cag.2020.05.024
0097-8493/© 2020 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license. ( http://creativecommons.org/licenses/by-nc-nd/4.0/ )
Pleasecitethisarticleas:D.Boges,M.AgusandR.Sicatetal.,Virtualrealityframeworkforeditingandexploringmedialaxis represen-tations of nanometric scale neural structures, Computers & Graphics,https://doi.org/10.1016/j.cag.2020.05.024
purposes [8], has emerged. Especially neurons,but also glia,can 32
be adequately schematizedthrough skeleton representations, and 33
nowadays,various laboratories are investing important resources 34
on creating faithful and smooth medial axis representations of 35
braincells.Theseskeletonrepresentationscanbeusedforvarious 36
kinds of novel visual and statistical analysis. To this end, time 37
consumingimage-based manual tools [9–11] are commonlyused 38
fortracingneuralprocessesonconfocalimages.Morecomplicated 39
automatic methodsfor recovering medialaxis representationson 40
nanometric scaleelectron microscopy stacksexist butare still in 41
theirinfancy [12]andnot yetroutinely usedforprocessingbrain 42
cells. 43
Inthispaper,wepresentanovelVR-basedframeworktargeted 44
oncreating, proofreading,and exploration ofskeleton-based rep-45
resentationsof nanoscale brain cellssurface reconstructions. The 46
systemintegratesthefollowingcomponents: 47
• Fastservo-assistedsemi-automaticmethodsforcreating skele-48
tonsofcomplexbraincellularreconstructions. 49
• Tools for proof-reading (checking, correcting, comparing) me-50
dialaxisrepresentations. 51
• Explorationtools,e.g.,forperforminggeometricmeasurements 52
and statisticalcomputations related to cellular structures and 53
theirskeletalrepresentations. 54
The system is currently used by expert domain scientists for 55
analysisofvariouscellsreconstructedfromthesomatosensory cor-56
tex of a juvenile rat [1]. We report on a preliminary subjective 57
evaluation of the immersive environment performed by domain 58
expertsduringcreation andproofreading ofcomplex medial axis 59
representations,aswell asduringanalysis oforganelles distribu-60
tions.Thispaperisan extendedversionoftheproceeding contri-61
butionpresentedatSmartToolsandApplicationsforGraphics con-62
ference[13].Weprovidehereamorethoroughexposition,together 63
withamoreeffectiveexternalsemiautomatictracingtoolfor edit-64
ing medialaxis branches. Moreover, we extended the framework 65
towork inamonocular setupwithalargescaledisplaywall,and 66
wecarriedoutauserstudyforevaluatingtheperformancesofthe 67
systemforcreatingandeditingskeletonrepresentations,eitherin 68
the monocular display wall setup and in the stereoscopic HMD-69
basedvirtualrealitysetup.Toourknowledge,thisisthefirst inter-70
activesystemtargetedto thecreation,proof-reading andanalysis 71
ofskeleton-basedrepresentationofcellularstructures,andthe pre-72
liminaryreportsofusage by expertandnoviceusers providedus 73
promisingindicationsthatthesekindofsystemscanpositively ef-74
fectthewayultrastructuralanalysisiscarriedoutinneuroscience 75
domain. 76
2. Relatedwork 77
Ourworkdealswiththeapplicationofvirtualreality(VR) tech-78
nologiesto neuroscience investigations coupledwith the compu-79
tationof medialaxis representations ofhighly detailedbranched 80
cellularbrainstructures.Inthefollowing,we discusstheprevious 81
workmostlyrelatedtoourcontribution. 82
Virtual reality in neuroscience. Due to the ubiquity of desktop 83
systems,mostcommonlyusedvisualanalysistoolsinneuroscience 84
are designed as desktop applications [14,15]. However, more re-85
cently,thereisgeneralconsensusthattheuseofstereoscopic tech-86
niques,e.g.,inVRsystems,canprovidea moreimmersivewayto 87
explorebrainimagingdata[16],andthattheincreased dimension-88
alityprovidedbystereoscopyisbeneficialforunderstandingdepth 89
in the displayed scenery [17,18]. With respect to immersiveness, 90
theeffectofstereoscopyhasbeenpreviouslyevaluatedinthe con-91
text ofvisual analysis ofvolume data,particularly for semitrans-92
parentvolumerendering[19,20],isosurfaces[21],confocalvolume 93
images[22],andforinteractive graphanalysis[16,17,23].Success- 94 fulexamplesofapplyingVR technologiestoneuroscienceinvesti- 95 gationsinclude analysisofglycogendistributionrelatedto neural 96 morphologies[4],systemsforexploringconnectomes[24],systems 97 for tracing neurons in microscope scans of primates’ visual cor- 98 tex[6],andtheuseofheatmapsforrepresentingabsorptionprob- 99 abilitiesonnanoscalesurfacereconstructions[5].VeryrecentlyXu 100 etal.[25] introducedTempoCave,asystemforanalyzingdynamic 101 brainnetworks by exploring activitypatterns indifferentregions 102 inthebrain,computedbyprocessingrawdataretrievedfromfunc- 103 tionalmagneticresonanceimaging(fMRI)scans[25].Inthiswork, 104 wedescribeanimmersiveenvironmentforperformingshapeanal- 105 ysisthat is mainly targetedon skeleton representations of nano- 106 metricreconstructions.Toourknowledge,itisthefirstapplication 107 ofaVRenvironmenttowardsmorphologicalanalysisofmedialaxis 108 representations,particularlyofbraincells. 109
Skeleton-basedrepresentationofsurfacemeshes.Medialaxisrep- 110 resentations or skeletons can be considered descriptors which 111 jointly describe the geometry, topology, and symmetry proper- 112 ties of a shape in a compact and intuitive way, providing a 113 means to capturethe essence of a 3D shape [12]. Automatically 114 orsemi-automaticallyproducingaccurate skeletonrepresentations 115 is a challenging task. During the last decades, many techniques 116 have been proposed, particularly by the computational geometry 117 community, for different kinds of 3D models. For a comprehen- 118 sivediscussion oftherecentmethodsforcreating3Dmedialaxis 119 representations,wereferthereaderstostate-of-the-artreportsby 120 Tagliasacchietal.[12],andbySobieckietal.[26].Ingeneral,there 121 is a huge collection of methods to obtain 3D skeletons, which 122 can be classified according to the input representation: mesh- 123 based[27–30]andvoxel-basedrepresentations[31].Sinceoursys- 124 temisfocusedonsurfacerepresentations,wewillmostlyconsider 125 methods that use meshes, even if our systemcan be considered 126 independentfromthemethod usedforobtaining themedialaxis 127 representationofthemorphologyconsidered.Thesystemhasbeen 128 designed to import skeleton representations coming from differ- 129 entautomatic frameworks: forourinitial analysis, we considered 130 theMeanCurvatureSkeleton(MCS)algorithm[27],andtheCenter 131 Line Treemethod[32],which areimplemented inthe Avizo [33] 132
framework. 133
Medial axis representations in neuroscience. Since medial axis 134 representations provide an adequate and convenient description 135 forbranchedstructures,recently,neuroscientistsstartedexploiting 136 them for representing complicated cellular structures, especially 137 neurons.Tothisend, they derived specific metrics forcomparing 138 branched structures,i.e.,trees,basedongeometrical andtopolog- 139 ical features [34–36]. Thesemetrics are then used for investigat- 140 ing differences and analogies between morphologies or in gen- 141 eral for performing identification and classification [37–39]. Fol- 142 lowing this philosophy, recently Kanari et al. [40] developed a 143 classification framework for neurons completely based on skele- 144 tons,whichisbasedonspecifictopologicalrepresentations,called 145 persistence diagrams.The framework has been successfully used 146 forobjectivemorphologicalclassification ofneocortical pyramidal 147 cells[8].Ithasalsobeenintegratedintoamoregeneralcollabora- 148 tiveframeworkfortheanalysisandvisualizationofneuronalmor- 149 phologyskeletonsreconstructed frommicroscopy stacks[41].Our 150 proposed immersive environment addresses similar needs,and it 151 iscustomizedfortheproofreadingandanalysisofskeletonsofdif- 152 ferentcells,while leveragingthebenefitsofaVR system.We be- 153 lievethat3D branchedstructuresderived bybraincellmorpholo- 154 giescanbemoreeffectivelyanalyzedbyleveragingcuesprovided 155 by stereoscopy and full immersion which are well suited for3D 156 scenes.Ourframeworkisgeneralandcustomizable,anditcan be 157 extended to integrate other geometric representations andvisual 158
D. Boges, M. Agus and R. Sicat et al. / Computers & Graphics xxx (xxxx) xxx 3
ARTICLE
IN
PRESS
JID:CAG [m5G;June26,2020;16:48]
Fig. 1. Our proposed virtual environment enables neuroscientists to immersively create, proofread, and explore medial axis representations or skeletons of nanoscale recon- structions of brain cells. In the example scenario above, skeletons are represented as connected nodes (yellow spheres) and edges (black cylinders), while brain cells are depicted as shaded surfaces (using a light blue color in this example). (For interpretation of the references to color in this figure legend, the reader is referred to the web
version of this article.) Q2
Fig. 2. Data preparation. Left: we tested the proposed immersive system on mod- els reconstructed from an image stack acquired by serial electron microscopy of a sample from a juvenile rat’s somatosensory cortex. Right: sparse reconstruction provides high resolution surface representation of full cellular morphologies.
3. Applicationdomain:Morphologyanalysisinneuroscience 160
Beforedetailing theproposedimmersive environment,wefirst 161
provide a brief description of our particular application domain 162
in neuroscience:the ultrastructuralinvestigation of braincellular 163
morphologiesatnanometricscale. 164
Ultrastructural analysis. Neuroscientists often perform ultra-165
structural analysis of brains samples through ex-vivo digital ac-166
quisitionof verysmall brainportions. Tothisend, they use high 167
resolution electron microscopy systems equipped with high pre-168
cision cutters [42]. Through this methodology, domain scientists 169
get 3D8bit imagestackscontainingcellularmembranes at nano-170
metric resolution (see Fig. 2 left). These datasets allow them to 171
visually individuate andannotate cellular andmolecular features, 172
such as compounds, synapses, and organelles like mitochondria. 173
vesiclesandendoplasmaticreticulum(ER).Nowadays,EMimaging 174
techniqueisbecomingincreasinglypopularinthefieldof connec-175
tomics,sinceitenablesaccuratereconstructionsoftheconnections 176
betweenneurons[43]. 177
Processing pipeline. Given a 3D stack of images acquired by 178
an electron microscope (Fig. 2 left), neuroscientists need to pass 179
through differentprocessingtasksinorderto extractrelevant 3D 180
shape representations of cellular structures, in form of surface 181
meshes(Fig.2right),thatcanbeusedforstatisticalcomputations, 182
simulation, or rendering.The processing pipeline consistsof car-183
ryingoutdenseorsparsereconstructions,byusingmanual, semi-184
automatic orautomatictools,whichlabeltheimage pixelsinthe 185
stack, i.e., assigning them witha unique object identifier forthe 186
various structures ofinterest,such asneuralaxons,dendrites, or-187
ganelles, nuclear envelopes, etc. In this work, we used a hybrid 188
two-steppipeline[44],composedby: 189
• Aroughautomaticsegmentationperformedofflinethroughthe 190 iLastiktool[45],forfindingthegrossfeaturesandprocessesof 191
acell. 192
• Amanualproofreadingphase,performedthroughtheTrackEm2 193 tool [46],for specifyingexact objectboundaries andfinerde- 194
tails. 195
Morphologyfeatures.Oncethevariouscellsandsub-partsarela- 196 belledonaper-pixel levelintheimagestack,neuroscientistsper- 197 formvariousultrastructuralanalyses by studyingthemorphology 198 ofthefollowingbiologicalstructures(Fig.2right): 199
• Neurons: composed of axons and dendrites, which are the 200 terminals respectively sending and receiving electric signals 201 throughboutonsandspines.Boutonsandspinesarelinkedand 202
formsynapses. 203
• Glialcells:neuroscientistsmainlyfocusonastrocytes,whichare 204 metabolicallyinvolvedinfeedingneurons,microglia,whichare 205 themainformofactiveimmunedefenseinthecentralnervous 206 system byacting asmacrophages,andoligodendrocytes, which 207 producethemyelinsheathinsulatingneuronalaxons. 208
• Organelles:domainscientistsmainlyfocusonmitochondriaand 209
endoplasmatic reticulum, which are contained in axons, den- 210 drites, andglialcells.Theycontainthemachineryforchemical 211
transformations. 212
Neuroscientistsare interestedinstudyingtherelationshipsbe- 213 tweentheaforementionedstructures,andperformgeometricanal- 214 ysisforrecoveringparametersto beusedforsimulationpurposes 215
orforclassification[40,47]. 216
Medial axis representations. Most of the considered cells have 217 complicatedbranching structures, which are very difficult to an- 218 alyze using standard mesh representations (see Fig. 2 right). To 219 thisend,skeletonrepresentationsprovideaneffectivetoolforde- 220 scribing them and classifying the various branches, according to 221 thesizeandthebranching level,startingfromthesoma.Forthis 222 reason, neuroscientists are increasingly focusing on technologies 223 thatcan supportthemin recoveringaccurate skeletalrepresenta- 224
tions[8,35]. 225
4. Systemoverview 226
The proposed systemis a standard 3D framework customized 227 tobeusedwithastereoscopicHMD-basedsetupusingroomscale 228 tracking technology (VR), or with a large screendisplay for col- 229 laborative sessions. In VR, the system allows the user to inter- 230 actwitha3Denvironmentthroughtwomotion-trackedhand-held 231 controllers,i.e.,by pointing/selectingobjects,orselectingmotions 232 through menus. When working with the display wall, a generic 233 Pleasecitethisarticleas:D.Boges,M.AgusandR.Sicatetal.,Virtualrealityframeworkforeditingandexploringmedialaxis represen-tations of nanometric scale neural structures, Computers & Graphics,https://doi.org/10.1016/j.cag.2020.05.024
Table 1
Notations/formats used for skeleton data.
Algorithm/Tool Notation/ File Format Data Type
Centerline Tree (Avizo) [ Point ID, Thickness, X Coord, Y Coord, Z Coord ]/.CSV Points (file1)
[ Segment ID, Node ID1, Node ID2, Point IDs List ] /.CSV Branches(file2)
Mean Curvature Flow [ x, y,z ] / txt Points (file1)
[ NodeID1, NodeID2 ] /.txt Branches (file2)
[ Sum of points(n), X, Y, Z, Xn,Yn,Zn ] /.txt Points and Branches
Simple Neurite Tracer (Fiji) [ NodeID, Cell Type, X, Y, Z, radius, ParentID ]/.SWC Points with Branches
gamepadcontrollercanbeusedforinput.Theframeworkwas de-234
velopedontopoftheUnitygameengine. 235
Scenerepresentationandrendering.Theimmersiveenvironment 236
providesreal-timeexplorationofscenescomposedofsurface rep-237
resentationsofbraincellsandschematicrepresentationsofthe as-238
sociatedmedialaxesorskeletons.Theleveloftransparencyof sur-239
faces can be interactively controlled in a way to providecontext 240
forskeletonexploration.Sincethesystemisalsodesignedfor pro-241
vidingendoscopicanalysisofcellularprocesses,atorchtoolis pro-242
videdforshadingmeshwallsanddarkcornersduringexploration. 243
Thetool isattached to one ofthe manipulatorsand can be eas-244
ilyused to illuminate dark areas.Basic 3D manipulation options 245
areprovided,e.g., objectscaling andplacement,aswell as mate-246
rialandcolorassignment.Moreover,userscan flipthemesh nor-247
mals,ina wayto havea moreconvenient wayofexamining the 248
inner/outermesh surfaces. Withrespect to skeletons, the system 249
usesthreedifferentrepresentations: 250
• sprite-based: 2D line segments/ribbons represent the whole 251
skeleton geometry (implemented using Unity line renderer 252
module); 253
• Node-based:onlyspheresrepresentskeletonnodes;depending 254
ontheskeletondata,thesystemcanutilizeonlyprimarynodes 255
toprovidearoughrepresentationofskeletons. 256
• Complete: skeleton nodes are represented by spheres while 257
skeletonedgesarerepresentedbycylinders. 258
Main features. After loading the cellular morphology, the sys-259
temenablesuserstooperateonmedialaxisrepresentationsintwo 260
modes:createmodeforcreatingskeletonsfromscratch,and
proof-261
read mode for correcting/editing previously computed skeletons. 262
Inproofreadmode, thesystemrequiresthatpreviously computed 263
medialaxesrespectspecificnotationsrepresentedinTable1.This 264
notationisvalidformostgraphrepresentationsandiswidelyused 265
bymanygraphprocessingsoftware.Specifically,inthispaper,we 266
focusedonskeletonscomputedthroughthreemethods: 267
• An automatic volume-basedmethod [32], implementedinthe 268
Avizo framework [33] - it uses connected components for 269
graphs,combiningaunion-findandarecursivealgorithm. 270
• An automatic mesh-basedmethod[27] -ituses iterative con-271
tractionthroughmeancurvatureflowevolution. 272
• A manual image-based tracer implemented in the Fiji sys-273
tem[10]. 274
Thesystemprovidessupportforimportingandexporting stan-275
dardskeletonfileformatsthatarecompatiblewiththepreviously 276
mentionedsystems.Itcanalsobeeasilyextendedtosupportother 277
formats/notations. 278
5. Interactivetools 279
The proposed system provides interactive tools for edit-280
ing/manipulatingmedial axis representations. We describe avail-281
able interactions using 3D controllers for VR as well as for 282
gamepad controllers forthe display wall.We tried to keep most 283
interactionssimilar forthe twocontrollers inordertoreducethe 284
Fig. 3. Interactive tools. Left: an arch-shaped menu attached to the left controller allows users to select interaction mode with skeletons. Right: a stabilizer servo- assisted tool (in red) guides users through the process of skeleton branch tracing. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
effortforusersintermsofswitchingbetweenthetwoifnecessary. 285 In thefollowing, we discussinteractions based on3D controllers 286 but thesame applies to a gamepad with the trigger buttoncor- 287 responding to gamepad buttons with pre-mapped functions, and 288 3D pointingcorresponding tousingthe gamepadarrowandana- 289 log buttons tomove the camera andthe pointer. The coreinter- 290 actions are summarized in 7 options laid out in an arch-shaped 291 menu(seeFig.3left),attachedtotheleft3D controller.Theuser 292 can choose one of the options by first rotating his/herwrist be- 293 tween 0◦ and180◦,andthen,oncesettledonan option,pressing 294 thetrigger buttonsto select. Theoptions providedby thesystem 295
arethefollowing: 296
• AddNode:usingthetriggerbutton,theusercancreateanode 297 in3D space.Thisprocesscanbe fullymanual orcontrolled by 298 aservo-assistedstabilizer.Upon creation,thesystemautomat- 299 icallypairsupnodeswitheachother andconnectsthemwith 300 anedge,hence,creatingasingleconnectedpath. 301
• Grab and Move: as part of the proofreading/editing process, 302 nodescan be moved anywhere simply by grabbingthem and 303 moving them.This can be achievedthru a combinationof an 304 actiongrabinitiatedbypressingandholdingofthecontroller’s 305
gripbuttonwhiletouchingthesurfaceofthetargetnode. 306
• SelectandConnect:using acombinationofpoint andtrigger 307 click,theusercan selecttwo nodessubsequentlyandthesys- 308 temcreatesanedgeconnectionbetweenthem(seeFig.4left). 309
• Delete Skeleton Element: the system allows the right con- 310 troller to shoot a laser pointer by pressing on the controller 311 touchpad.Theusercanthendeletenodesandedgesbypoint- 312 ingatavalidskeletonunitobjectfollowedbyatriggerbutton 313
click(seeFig.4right). 314
• TagRoot,Junction,andLeaf:asimilaractionofpointandtrig- 315 gerata specifiednode willsaveit initscorresponding skele- 316 ton file as one of these values: 0=Root, 1=Internal, 2=Leaf, 317 3=Junction. Tagging a node with “Leaf”, “Junction”, or “Root” 318
D. Boges, M. Agus and R. Sicat et al. / Computers & Graphics xxx (xxxx) xxx 5
ARTICLE
IN
PRESS
JID:CAG [m5G;June26,2020;16:48]
Fig. 4. Skeleton editing. Our system provides effective tools for rapid editing of skeleton branches. Left: adding connection between nodes. Right: removing a wrong edge from a skeleton branch.
Fig. 5. Skeleton creation. We propose a semiautomatic and guided method for creating skeletons, based on endoscopic exploration of cell branches, and using a servo-assisted stabilizer.
marksitwithaspecialcolormaterialandfinalizesthecurrent 319
pathasasinglebranch. 320
• Undo: an action that saves users the troubles of having to 321
delete a mistake node manually,and instead, they can revert 322
tomultiplestepsbackduringtheskeletoncreationprocess. 323
Path stabilizer: tunnel metaphor. The system provides a semi-324
automatic methodfor creatingskeleton branchesthrough one of 325
theVRinputcontrollers.Thismethodisbuiltaroundavisualuser 326
guide,thatoperatesasreferencewhentracingthetunnel-like cel-327
lular processesthrough endoscopic navigation.During the explo-328
rationoftheprocess,a pathstabilizertransparentlyand automat-329
ically placesskeleton nodesin themiddleof theprocess section. 330
The automaticnode positioncomputationisperformedby shoot-331
ing straightrays ontoa numberofradialdirections,and comput-332
ing theaveragedistancetothesurroundingwall boundaries.This 333
simplebuteffectivemethodprovidesawaytorapidlytracemain 334
cellularprocesses,andcreatefullycontrolledskeleton representa-335
tions.Incurrentimplementationweuse16raysforcomputingthe 336
averagedistance. 337
Pathstabilizer:tracingmetaphor.Inordertospeed upthe trac-338
ingprocess oflongbranchesexhibitinglowcurvature,likeit hap-339
pensforsomeneuraldendritesandaxons,we introduceda semi-340
automatic external tracing metaphor. With thistool, useris able 341
Fig. 6. External tracing. Users can trace branches through external 3D pointing, since a semiautomatic algorithm computes running barycenters through multiple iterative ray casting.
tofollowthe pathof thespecific through 3Dpointing, while the 342 systemuses ray casting forintersecting thebranch, and, starting 343 fromtheray,an iterativeapproachshoots differentrays similarly 344 tothepreviouslydescribedtunnelmetaphor.Inordertoaccelerate 345 thecomputationofnearestneighbors,weuseaKD-treedatastruc- 346 ture.Therunningbarycentersofthedifferentrayintersectionsare 347 addedasnodesinthecurrentskeletonbranch.Eveninthiscase,in 348 currentimplementationweuse16raysforcomputingtherunning 349 barycenter. This method proved to be very fast andeffective for 350 processesnotexhibitingsharpfeaturesandbumps(seeSection7). 351
Fig.6showsanexampleofthealgorithmforcomputinganode. 352
6. Setupanddataset 353
Our proposed immersive system is used by neuroscientists 354 forperformingreal-timecreation,proofreading/editing,andexplo- 355 rationofbraincellreconstructionsbasedonmedialaxisrepresen- 356
tations. 357
Implementationdetails.The immersive systemhas beendevel- 358 oped and deployed using the Unity game engine (version 5.6.3, 359 viaC# scripting).ForVR,ituses SteamVRandtheVRTKsoftware 360 packages[48],whichprovidesmoothimmersivesystem-userinter- 361 actionaswell ascross-hardware setupcompatibility. Inthisway, 362 thesameapplicationcan beusedonvariousVR setups,likeOcu- 363 lusRift[49]orHTCVive[50].Forcomputingautomaticskeletons, 364 andforother preprocessingtasks,weimplementedandusedC++ 365 applicationsandPythonscripts.Inaddition,weusedAvizo(acom- 366 merciallyavailabledataanalysis/visualizationsoftwareframework) 367 forcomputinghigh-quality skeletons, andpreprocessingwas car- 368 riedoutonaworkstationequippedwithtwoCPUsof10coreseach 369 (seeTable2foradditionaldetails). 370
Datapreparation.Fortestingpurposes,weconsideredfivecom- 371 plex cellular structures reconstructed from a p14 rat somatosen- 372 sorycortex.Weselected differentkinds ofcellstoshow different 373 levels ofcomplexity:two neurons, two microglia,andone astro- 374 cyte[1].The cells were reconstructedfrom a high-resolution EM 375 stackwithapproximatedsizeof100
μ
m X100μ
m X76.4μ
m(see 376Fig.2 left). The reconstruction process wasperformedthrough a 377 semiautomaticprocess[44]involvingcustomizedcomponentsand 378 public domain software like iLastik [45] and TrakEM2 [46]. The 379 output of the reconstruction process is a series of high resolu- 380 tiontriangularmeshesrepresentingthecellularmorphologies(see 381
Fig.2right).Furthermore,eachcellwasoptimizedinawaytobe 382 Pleasecitethisarticleas:D.Boges,M.AgusandR.Sicatetal.,Virtualrealityframeworkforeditingandexploringmedialaxis represen-tations of nanometric scale neural structures, Computers & Graphics,https://doi.org/10.1016/j.cag.2020.05.024
Table 2
Machines used for immersive environment and data preparation.
Machine OS Task Specs
Asus ROG G703G Windows 10 Pro- Immersive environment 32GB DDR4, Intel Core i9-8950HK 4.8 GHz, Nvidia GTX 1080 8GB GDDR5X, 2X 256GB PCIE SSD + 2TB SSHD FireCuda. Supermicro Linux CentOS 7 Data processing and skeleton creation 1TB memory, Intel(R) Xeon(R) Gold 6150 CPU2.70GHz (18 Cores), Nvidia GK104GL Quadro K5000, N/A
watertightandwithoutnon-manifoldedgesandvertices,andina 383
waytopreserveallimportantmorphologicalfeatures.Tothisend, 384
we used public domain mesh processing tools like Blender [51], 385
Meshlab[52],andUltralizer,ageometryprocessingtoolcontained 386
insidethe suite NeuroMorphoVis[41].For gettingautomatic me-387
dialaxisrepresentationsoftheconsideredmorphologies,weused 388
theMean Curvature Skeleton algorithm [27],as well asthe Cen-389
terline Tree module both available in Avizo [32]. In Table 3 we 390
reporton the cell morphologiesandthe associated skeleton rep-391
resentations.Specificallywe provide visualrepresentations ofthe 392
morphologies, together with information about their shapes and 393
sizesintermsofvertexcounts,visualrepresentationsofautomatic 394
skeletons,andskeletongraphstatistics(numberofnodes,number 395
ofedges,andnumberofbranches). 396
Hardwaresetups.WetestedtheVRapplicationonagaming lap-397
topequippedwithanNvidiaGTX10808GBGDDR5XGPU.Weused 398
thelaptoptodrivetwodifferentdisplaysetups(seeFig.8): 399
• AstereoscopicimmersiveHeadMountedDisplay(HMD)Oculus 400
RiftS[53] withsensorsembeddedinawaytolowerthe bulk-401
inessofthesystemandincreaseportability. Thedisplayusesa 402
singlefast-switch LCD panel witha resolution of2560× 1440 403
witharefreshrateof80Hz,fieldofviewof110degreesona 404
workspaceof3.5m × 3.5m; 405
• Amonocularcollaborativelarge-scaledisplaywallcomposedof 406
an arrayoftiled 3 × 4Narrow BezelMonitors(55”)(7680 × 407
3240pixels)foratotalresolutionofaround25Mpixels.Forthe 408
monoscopic display setup, the Oculus Riftinput devices were 409
substitutedbyanXboxgamepadcontroller. 410
The two different setups provide different working environ-411
ment in which the toolscan be used as direct pointersfor per-412
forming editing operations on the scene. The proposed interact-413
ingmetaphorsaregeneralandcanbe adaptedtodifferentsetups, 414
using different kind of controlling devices, like touch devices or 415
gestures [54]. According to the taxonomy presented by Mendes 416
etal.[54],the consideredsetupsarethefollowing: animmersive 417
one,with3Dcontrollers physicallyprovidingadirect metric con-418
troloverthescenerepresentedinavirtualworkspace,anda gam-419
ing one, withgamepad controllers controlling a 2D display wall, 420
in an indirect way. Regarding the design choices, we decided to 421
donot use the samecontrollers in both setups since the oculus 422
controllersarenativelyconnectedtoareal3Dphysicalsetup,and 423
webelievetheywouldnotbenaturallyunderstoodifcoupledtoa 424
monoscopicdisplaysetup. 425
7.Results 426
We carriedouta preliminaryassessmentofthesystemto un-427
derstandhowistheusability inthecontextofexploratory analy-428
sisofdifferentkindofcells. Wereport hereontwo kindof eval-429
uations: a subjective qualitative assessment performedby expert 430 users on complete creation of skeleton representations of entire 431 cells,andanuserstudyforevaluatingtheperformanceofthesys- 432 temforeditingtaskseitherinHMD-basedstereoscopicsetupand 433 Wall-basedmonoscopicsetup.Finallywereport onausecasefor 434 measurementanalysisondistributionsofbranched-likeorganelles. 435
7.1. Expertevaluation 436
Apreliminaryevaluationofthesystemwasperformedby two 437 expertneuroscientists oncellsofTable3.Domain scientists were 438 particularlyinterestedinobtainingaccurateandclearskeletalrep- 439 resentations to be used asdescriptors ofhighly intricate cellular 440 structures.Ingeneral,theywanttohaveprecisecontrolofmedial 441 axisrepresentations,inawaytobe abletoclearlyseparate main 442 processes from fine details that have different biological mean- 443 ing (for example dendritic shafts and spinesin neurons). In this 4 4 4 sense, mostautomatic systemsprovide “dirty” medialaxis repre- 445 sentations, thus we expected that an interactive tool helping in 446 cleaningskeletonswouldreceiveapositivefeedback.Moreover,we 447 expectedthattheimmersivenessprovidedby virtualreality could 448 improvethecreationandeditingprocess. 449
Skeleton creationfrom scratch. Neuroscientistsused thesystem 450 for creating skeletons fromscratch on two neural morphologies. 451 InTable4,weshow statisticsabouttheskeletoncreationprocess, 452 withdifferenttracingmetaphors(internalandexternal)anddiffer- 453 entdisplaysetups(VR-basedandWall-based).Theprocedurecon- 454 sistedofexploringthesurfacemodels inordertoselectthemain 455 processes,andtracethebranchesfrominsidethecells,i.e.,similar 456 to anendoscopic navigation/view.Domain scientists feltcomfort- 457 ableinrecognizingmainprocesses,e.g.,dendritesandspines,ina 458 waytocorrectly tracethe medialaxisof interest.Moreover, they 459 feltquite comfortablewiththepathstabilizer,whichreducedthe 460 numberofinput actions oncontrollers. A comparisonofcreation 461 timesbetweenthe two differentdisplay setupsand thedifferent 462 tracinginterfacesshowsthat expertuserswerefasterwhenusing 463 external tracingmetaphor withstereoscopicVR setup (almost2X 464 withrespecttotheworstcaseaccordingtimingsinTable4). 465
Skeletonproofreading/editing.Automaticallycomputedskeletons 466 were examined by domain scientists through the proposed sys- 467 tem (see Fig. 7). They used the system for comparing skeletons 468 automaticallycomputedthroughMeanCurvatureFlow(MCS[27]), 469 andCenterlineTree(CLT[32]).Theyconcludedthatboththemeth- 470 odsconsidered were abletocoverall themorphology featuresof 471 interest. However, skeletonsproduced by CLT appearedto be too 472 highly detailed, with a number ofwrongly assigned branchesas 473 wellasdisconnectedparts.Table5showsthedifferenceintheto- 474 talnumberofbranches,nodesandedgesforeachalgorithmforall 475 five cells. In general, domain scientists found that skeletonspro- 476 ducedbyMCSalgorithmcontainedalowernumberofartifacts.For 477 thisreason,inallconsideredcases,theypreferredtoperformedit- 478 ing andcleaning on skeletonscomputed through MCSalgorithm. 479 Tothisend,they carriedout aseriesofchecksdependingonthe 480 type ofcell,andon thebiologicalsignificance ofthevarious fea- 481
tures: 482
• Identify main branches by tagging their nodes as either leaf, 483 endofbranch,orinternalnodes.Thesystemidentifiesallnode 484 typesbasedonthedegreeofeachoneinthegraphtree.How- 485 ever, some needs to be adjusted based on the cell’s biologi- 486 calfeatures. UsingtheVRinteractive menu,theuserpointsat 487 a node with the VR controller’s laser pointer and then clicks 488 onthe trigger buttonto tag it.The node’s colormaterial will 489 switchcolorindicatingthat itissaved inthesystembasedon 490 thetaggingfeature.Inthecaseofneurons,themainbranches 491 wouldbealldendrites,excludinganyotherfeaturese.g.,spines. 492
D. Boges, M. Ag u s and R. Sicat et al. / Com p ut ers & Gr aphics xxx (xxxx) xxx 7
AR
TICLE
IN
PRESS
JID: CA G [m5G; June 26, 2020;16:48 ] Table 3Morphologies and Mean Curvature Skeletons (MCS) of 5 biological cells. Cells are computed automatically through [27] and proofread and cleaned through our Virtual Reality system. Together with pictorial representations, we report on cell sizes, total times for proofreading and cleaning, and skeleton statistics.
Name Picture Vertices Time Before cleaning Nodes-Edges-Branch After cleaning Nodes-Edges-Branch
Neuron1 49,628 10.00 1569| 1573| 201 1318| 1321| 25 Neuron2 78,215 15.28 1,619| 1,629| 357 1215| 1223| 20 Microglia1 48,015 09.00 1,463| 1,479| 165 1443| 1456| 62 Microglia2 125,532 13.53 2,105| 2,122| 260 2060| 2077| 111 Astrocyte 211,004 25.16 4,055| 4,137| 854 3906| 3983| 296 Please cit e this article as: D. Bog e s, M. Agus and R. Sicat et al., Virtual re a lit y fr ame w or k for e d iting and ex ploring me dial axis re pr esen-tations o f n anome tric scale neur al st ructur es, Com put e rs & G ra phics, https://doi.or g /1 0.1 0 1 6/j.cag.2020.05.02 4
Table 4
Statistics on skeletons generated semi-automatically from scratch for Neuron1 and Neuron2 morphologies.
Cell Name Skeleton Time Stereo Int Time Stereo Ext Time Mono Int Time Mono Ext Nodes| Edges| Branches
Neuron1 25:13 14:24 29:55 25:28 481| 480| 25
Neuron2 30.50 14:35 22:54 24:11 629| 628| 20
Fig. 7. Skeleton proofreading. Our system enables domain scientists to perform effective proofreading of skeletons by using endoscopic and external metaphors. Table 5
Neuron1 and Neuron2 skeleton properties as generated via Mean Curvature Skele- ton (MCS) and Centerline Tree (CLT) algorithms.
Cell Name Algorithm Nodes|Edges|Branches
Neuron1 MCS 1569 | 1573 | 201
CLT 7719 | 7328 | 361
Neuron2 MCS 1619 | 1629 | 357
CLT 9655 | 9530 | 516
• In the caseofhighly-detailed skeletons, one wouldencounter 493
duplicate nodesandedges, disconnected parts,loops, andout 494
of trackskeletonization.Neuroscientists tried to deleteall de-495
fectsthroughaniterativemanualprocess. 496
• Thesomaareashouldbeclearfromanybranchingsoneurosci- 497 entists“cleaned” thesepartsbydeletingallbranchesandmerg- 498
ingthemintoone. 499
InlasttwocolumnsofTable3,weshowtheproofreadingout- 500 putsforalltheconsideredcells.Ingeneral,domainscientistsfound 501 the proofreading taskcomfortable andaccurate, and they partic- 502 ularly appreciated the immersiveness of the system forchecking 503 featuresandrecognizingdefects.Forcomparison,weaskedexpert 504 neuroscientists to perform skeleton cleaning by using the exter- 505 nal tracing metaphor together with the wall display setup: tim- 506 ingsrecorded forNeuron1 (27m58s) andforNeuron2 (46m10s) 507 providedusevidencethatthemonoscopicsetupassociatedtothe 508 externalinterfaceisnotcomfortableforcleaningskeletons. 509
Discussion.Ingeneral,oneofthedrawbacksofdealingwithan 510 immersiveenvironmentonlongsessions(20minandmore),isthe 511 symptomsofcybersicknessandfatigue.Thishappenedalsoforour 512 system, and,duringthesessions,usersneededto takebreaks ev- 513 ery 15 min when performing each task.To this end, the system 514 allowsformultiplesavesacrosssessions,andtheusercanretrieve 515 thefileanytimeandcontinuewherehe/shelaststopped.Fromthis 516 pointofview,userslikedthe displaywallsetup incaseofcyber- 517 fatigue, since they could sit, and rest while still working on the 518 task.Asgeneralimpression,the systemwasconsidered veryuse- 519 fulforboth proofreading/cleaningpre-exported skeletons,aswell 520 asforcreatingskeletonsfromscratch.Inparticular,neuroscientists 521 trained in neurite tracing found very effective how the tool au- 522 tomaticallyprovides a centerline,without thehassleof havingto 523 placeitmanually.Althoughtheprocessofcreatingskeletonsfrom 524 scratchinVRcanbetime-consuming,automatictoolscanmakea 525 lotofmistakes, andthetime savedonthe manual tracingwould 526 be lost on the proofreading anyways. This largely balanced the 527 costs/benefitsofthetwoapproaches.Severalfactorscontributedto 528 makethecreationprocess timeconsuming, mainlytechnical.One 529 factorinvolves the orderof tracing the variousbranches. Specifi- 530 cally,insome cases,usersstartedtracing fromthesomaandpro- 531 ceededtowardsthetips,whileinothercasestheymadetheoppo- 532 sitechoice,by startingfromthetipofthe mostextendedbranch 533 andtracing towards the soma. Wealso experienced that another 534
Fig. 8. Display setup for user study. We evaluated the system performance under different display conditions: large scale monoscopic display (left), and head mounted stereoscopic display (right).
D. Boges, M. Agus and R. Sicat et al. / Computers & Graphics xxx (xxxx) xxx 9
ARTICLE
IN
PRESS
JID:CAG [m5G;June26,2020;16:48]
source of error was the path-stabilizer, in cases where the user 535
happenedto releaseanode atabifurcation spot. Sincethe path-536
stabilizeris basedon theconcept ofray-casting, usersneededto 537
takecareofcorrectlykeepingtheVRcontrollerwithinthewallsof 538
the cellularstructure.Issues withthestabilizerwere experienced 539
also in cases when the cell’s main branch has too many spines 540
withinclosedistancetoeachother.Insuchsituations, neuroscien-541
tistssometimespreferredtodisablethestabilizerandoperateona 542
full-manualmode.Ingeneral,wenoticedthatmostcreationissues 543
were alleviated asusers gained experience withthe system, and 544
wethinkthatperformanceshoulddramaticallyimproveonceusers 545
repeat the process for many cells, i.e., after further training and 546
experience. From the quality point of view, neuroscientists were 547
satisfied withskeletonsgenerated fromscratch in VR, since they 548
appeared well-structured and represented precisely the biologi-549
cal structure of thecells, tailored accordinglyto their experience 550
and knowledge aboutcell morphology. Regarding the proofread-551
ing task,domain scientists performedvery well in checking and 552
editing all five skeletons. They experienced some problems only 553
withtheAstrocyte,whichtookaround25minutestobeproofread 554
andedited.Thisisduetothefact thatastrocyteshavevery com-555
plicated branched structures (see Table3), where mainprocesses 556
areverydifficulttorecognizeatfirstsightevenforexpertdomain 557
scientists. 558
7.2. Userstudy
559
The use of Virtual Reality in Neuroscience is still atits early 560
stages[5,55],thereforetheuserstudiesforevaluating the perfor-561
manceofVirtualEnvironmentsforeditingtasksarecurrently de-562
signed almost from scratch, because of the lack of standardized 563
guidelines[56].In ourcase, we aimedto assessthe performance 564
ofthesystemforcreatingandmodifying skeletonrepresentations 565
of brain cells. To this end, we involved 12 users with different 566
level ofexperience,andwe askedthemto performspecific tasks 567
underdifferentconditions.Wemeasured timesandaccuracy, and 568
we assessedthefatigue andcomfortthroughNASA-TLX question-569
naire[57].Inthefollowingwedetailthedesignoftheuserstudy 570
andthe outcomesof evaluation. Themain goal ofthe studywas 571
to evaluate the effects of HMD and wall display on system per-572
formance,andwhetherusersfeltmorecomfortablewithexternal 573
orinternaltracing metaphorforeditingpurposes(seeFig.8).The 574
studytook placein thevisualization labfacilityofKing Abdullah 575
University of Science and Technology, and in the department of 576
AnatomyatUniversityofTurin. 577
Designandprotocol.Subjectswereaskedtoperformvarious par-578
tialtasksonthesystem,usingHMDorWalldisplay,andusing in-579
ternalorexternalmetaphorforeditingoperations.Specifically, af-580
ter aperiodfortrainingandgettingcomfortablewiththevarious 581
setupsandtools,weaskeduserstoedittheskeletonofabraincell 582
inthefollowingway: 583
• Traceafullbranch. 584
• Correctabranchbyremovingnodesandlinks. 585
The branches to be traced were chosen randomly from two 586
longdendriticprocessesofNeuron1andNeuron2(seeFig.10). Be-587
fore each task, the user was shown how the task is performed 588
andhowthejoystickcontrolsarebeingoperated.Theyweregiven 589
a few minutes to practice the task until they were comfortable 590
enough to go ahead and start their mission. Water and refresh-591
mentswereofferedatalltimesduringthestudy.Equipmentwere 592
wipedandcleanedthoroughlywithantibacterialwipesaftereach 593
use. 594
The two taskswere repeatedrandomly underdifferent condi-595
tions,dependingonthedisplaytypeandthetracingmetaphorfor 596
a total of 8tasks (see Fig.9). Between each taska break of five 597
Fig. 9. User study protocol: subjects were asked to perform eight skeleton editing tasks under different conditions related to tracing metaphor and display setups.
Fig. 10. User study data: subjects were asked to perform eight editing tasks under different conditions on two long processes from Neuron1 (left) and Neuron2 (right).
minuteswasgiven tosubjects.After one taskwascomplete (un- 598 derthe4conditions),userswereaskedtofilla6questionsNASA- 599 TLXformforcomparingmentaldemand,physicaldemand,tempo- 600 raldemand,performance,effortandfrustration(seeTable6),with 601 a5valueLikertscalescorerangingfromlowto high[57].During 602 thetaskswemeasuredthetotaltimeforperformingthetasks,and 603 thepathsofthe tracedbranch.The testsweredesignedinaway 604 thatusersdidnotneedmorethan60minfortraining,completing 605 alltasks,andfillingtheforms.Think-out-loudcommentswerealso 606
recordedduringsessions. 607
Quantitative performance. For measuring performance in cre- 608 ationandediting taskswe compared branchesobtainedby users 609 withrespecttogroundtruthobtainedbyMCS[27].Fig.11shows 610 an exampleofa branchcreatedby a user(in pink),andthecor- 611 respondinggroundtruth(in blue).Fig. 12showsperformance re- 612 sultsfor thetests relatedto accuracy fortrace branchesand the 613 completiontime, obtainedafterfilteringvery fewoutliersofsub- 614 jectsexhibitingreallypoorperformance.Weshowresultsinform 615 of boxplots: the bottom and top of each box are the first and 616 Pleasecitethisarticleas:D.Boges,M.AgusandR.Sicatetal.,Virtualrealityframeworkforeditingandexploringmedialaxis represen-tations of nanometric scale neural structures, Computers & Graphics,https://doi.org/10.1016/j.cag.2020.05.024
Fig. 11. Accuracy measurement. We measure the accuracy of branch creation through Hausdorff distance between the ground truth branch computed by MCS [27] (in blue in this example), and the branch traced by subjects (in pink in this example). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Fig. 12. Quantitative performance. Boxplots of accuracy performance for creation task (Hausdorff distance with respect to ground truth computed with Mean Curva- ture Flow in top row), and time performance for creation task (middle row) and editing task (bottom row). The bottom and top of each box are the first and third quartiles, the black line inside the box is the second quartile (the median), and the ends of the whiskers extending vertically from the boxes represent the lowest da- tum still within 1.5 IQR (inter-quartile range) of the lower quartile, and the highest datum still within 1.5 IQR of the upper quartile.
third quartiles, the black line inside the box isthe second quar- 617 tile (the median), and the endsof the whiskers extendingverti- 618 cally fromthe boxes representthe lowest datum still within 1.5 619 IQR(inter-quartilerange)ofthelowerquartile,andthehighestda- 620 tumstillwithin1.5IQRoftheupperquartile.Outliersareindicated 621 assmallcircles.Ontopweshowtheaccuracyoftracinginformof 622 Hausdorff distancein
μ
mbetweenthebranchcreatedbysubjects 623 and the ground-truth branch computedthrough MeanCurvature 624 Flow [27]: given two point sets X, Yrepresenting two branches, 625 we measure the symmetric Hausdorff distance [58] as the max- 626 imum of the asymmetric directed Hausdorff distance H(
X,Y)
= 627 max(
Hˆ(
X,Y)
,Hˆ(
X,Y))
, where the directed Hausdorff distance is 628definedas 629
ˆ
H
(
X,Y)
= maxx∈X
(
min y∈Yx− y)
. (1)ANOVA on the Hausdorff distance showed no effects due to 630 the display for accuracy (1.49± 0.88 with Mono setup versus 631 1.74± 0.72withStereo setup).Alsothedifferentinterfacesdonot 632 appearto affectaccuracy (1.63± 0.65withExternalinterface ver- 633 sus 1.60± 0.96 with the Internal interface). On the bottom we 634 compare the task completion time, either for the creation pro- 635 cess(middle row) and theediting process (bottom row).ANOVA 636 revealedan effectoninterfacewhen usedwithStereosetup (p= 637
.06withF=3.893314) forthecreationtask,withT=218± 64.6s 638 forexternalinterface,andT=292± 117.7s forinternal interface, 639 indicatingthat theexternal interfaceappears tobefasterforcre- 640 ation especially in VR setup. Moreover, ANOVA revealed an im- 641 portant effectforediting eitherrelatedto the display setup (p= 642
.007withF=8.106935)andtheinterface(p=.001forF=12.24), 643 indicating that external interface is perceived more comfortable 644 and users perform editing tasks faster when they use the VR 645
setup. 646
Qualitativeperformance.Table6showsthequestionsandresults 647 ofNASA-TLXquestionnaireproposed tosubjectsafter tasksinor- 648 der to evaluate their perception of performance, satisfaction, fa- 649 tigueandstressunderthedifferentconditions. 650
Fig. 13 shows the boxplots of answers on a Likert scale 651 (1=low,5=high).ANOVAonanswersrevealedaslighteffectforself 652 satisfaction duringthe tracing task dueto display setup (p=.03 653 and F=5.29 for question Q4 in favor of VR setup), and effects 654 on stress of display (p=.1 and F=2.81 for question Q6 in fa- 655 vorofVRsetup),andinterface(F=3.93andp=.05forquestion 656 Q6 infavorof externaltracing). Withrespectto theediting task, 657 ANOVArevealedsignificanteffectsrelatedtothedisplayformen- 658 taldemand(F=12.67andp=.001forquestionQ1infavorofVR 659 setup),physicaldemand(F=12.86andp=.0008forquestionQ2 660 infavorof VRsetup),fatigue (F=8.14and p=.006 forquestion 661 Q5infavorofVRsetup),andstress(f=3.32andp=.07forques- 662 tionQ6infavorofVRsetup).Nosignificanteffectswerefounddue 663
tothedifferenteditinginterface. 664
Discussion.Byobservingthebehaviorofuserswiththesystem, 665 we could notethat the learningcurve wasrapidandthe process 666 per-se waspretty straight forward. In general, while performing 667 thetasks,all operationsrequiredsome timefortheuserto learn 668 how toswitch fromone functionto anotherone. Tonote alsoin 669 thiscasethelearningcurvewasprettyfastforexperiencedtracers. 670 Fortheusageofthesystemondisplaywallsetup,werealizedthat 671 itis afactorof advantageiftheuseris agamer, whenusingthe 672 XBOXcontroller:expertgamersappeared tobemorecomfortable 673 andtonavigate blindly andeffortlessly. Ingeneraltheuserstudy 674 revealedthat subjectsfeelmoreincontrol wheninVR, sincethe 675 orientation, navigation and interaction is more natural, andthey 676 can contribute withbody,head andhands andnot justtwo joy- 677 sticksthat restrict movement, eyesightandperspective. Since we 678
D. Boges, M. Agus and R. Sicat et al. / Computers & Graphics xxx (xxxx) xxx 11
ARTICLE
IN
PRESS
JID:CAG [m5G;June26,2020;16:48]
Fig. 13. Qualitative evaluation. Boxplots of answers in Likert scale for questions in Table 6 for two different editing metaphors (external and internal), two different setups (Mono and Stereo), and two different tasks (creating and editing). The bottom and top of each box are the first and third quartiles, the black line inside the box is the second quartile (the median), and the ends of the whiskers extending vertically from the boxes represent the lowest datum still within 1.5 IQR (inter-quartile range) of the lower quartile, and the highest datum still within 1.5 IQR of the upper quartile.
Table 6
User evaluation: 12 subjects were asked to compare two different editing metaphors (external and internal) on two different setups (Mono and Stereo) for two different tasks (creating and editing). Table shows the results of the answers of a 6-item questionnaire [?] on a Likert scale (1 = low,5 = high).
Question
Results (Likert scale: 1 = low 5 = high)
Creation Editing
Stereo Mono Stereo Mono
External Internal External Internal External Internal External Internal Q1 : How mentally demanding was it? 2.0 ± 0.7 2.4 ± 1.2 2.2 ± 0.9 2.7 ± 1.1 1.4 ± 0.5 1.8 ± 0.8 2.5 ± 1.1 2.7 ± 1.2 Q2 : How physically demanding was it? 1.4 ± 0.7 2.0 ± 1.3 1.8 ± 0.9 1.7 ± 0.7 1.3 ± 0.5 1.6 ± 1.2 2.3 ± 1.1 2.3 ± 0.7 Q3 : How hurried was the pace? 1.9 ± 0.8 2.2 ± 1.3 2.1 ± 1.2 2.3 ± 1.3 1.3 ± 0.5 1.5 ± 0.8 1.8 ± 1.2 1.9 ± 1.0 Q4 : How successful were you? 4.3 ± 0.5 4.2 ± 0.7 3.8 ± 1.0 3.7 ± 1.0 4.4 ± 1.2 3.8 ± 1.5 3.9 ± 1.1 4.0 ± 1.1 Q5 : How hard did you have to work ? 2.1 ± 0.8 2.3 ± 1.3 2.2 ± 0.8 2.7 ± 1.2 1.6 ± 0.7 2.2 ± 1.5 2.8 ± 1.4 3.0 ± 1.2 Q6 : How stressed were you? 1.3 ± 0.5 1.7 ± 1.1 1.7 ± 1.0 2.3 ± 1.1 1.1 ± 0.3 1.6 ± 0.8 1.9 ± 1.0 1.7 ± 1.2
designedthedurationoftestsinawaytodonotletsubjects per-679
ceiveanyproblemofcybersickness(maximum10minutesforeach 680
task,and5minutesbreakbetweenthe tasks),resultsofthe user 681
studyappeartobeincontradictionwithrespecttotheevaluation 682
performedbyexpertusersduringthetracingofentirecells.Itwas 683
a consciousdecision duringthedesign of theuser study,even if 684
weareawarethatitwouldbeimportanttoevaluatetheeffectsof 685
cybersicknessandtofindwaystoreduce it.Weplantocarry out 686
furtheruserstudyinvestigations infuture,withdifferenttask du-687
ration,inordertobetterevaluatetheeffectsofcybersicknessand 688
physicaleffortsonourframework. 689
7.3. Casestudy:Analysisofbranch-basedintracellularorganelles.
690
One of the significant benefits of having skeleton representa-691
tions of braincellsis thepossibility ofcomputing accurate mea-692
surementsofmorphologicalfeatures.Asapreliminarytest, neuro-693
scientists performedanalysisofmitochondria,which are intracel-694
lularstructures withintheneuralcellsNeuron1andNeuron2(see 695
Table 7). Since scientists are particularly interestedin measuring 696
specific geometric features of organelles, like lengths and radii 697
(maximum, minimum, andaverage), adequate skeleton represen-698
tationsareneededforperformingaccuratemeasures.Tothisend, 699
our systemuses the samefunctionality equippedinthe VR path 700
stabilizer. Users can point ata particularnode froma branchof 701
Fig. 14. Branch-based measurements. Our system performs calculations of mea- surements on intracellular structures. Left: User points the laser pointer at any node of a branch of interest to display node-relevant measurements. Right: Mea- surements of a mitochondrion branch are displayed.
interest,andthesystemusestheskeletoninformationforprovid- 702 ing the measure of the full length, along with the radius values 703 ateachskeletalnodecontainedinthatbranch.Themeasuredval- 704 uesareshownastextlabelsinthesceneontopofeachnodeand 705 recordedforsubsequentstatisticalanalysis(seeFig.14). 706 Pleasecitethisarticleas:D.Boges,M.AgusandR.Sicatetal.,Virtualrealityframeworkforeditingandexploringmedialaxis represen-tations of nanometric scale neural structures, Computers & Graphics,https://doi.org/10.1016/j.cag.2020.05.024
Table 7
The intracellular structures of Neuron1 and Neuron2 showing mitochondria morphology, side by side with their skeletons generated via the MCS algorithm.
Cell Name Morphology MC Skeleton Nodes| Edges| Branches
Mito Neuron1 2246| 1963| 1396
Mito Neuron2 1749| 1656| 811
8. Conclusion 707
We presentedan immersive systemfor creating,proofreading 708
and exploring medial axis representations from highly detailed 709
braincellularmorphologiesreconstructedfromserialelectron mi-710
croscopy. The framework is designed for stereoscopic HMD dis-711
play setups andextended to large-scale monocular displaysin a 712
waytoalleviate unpleasantside-effectslikecyber-sicknessand fa-713
tigue,while still providing the ability to edit the skeleton in an 714
immersive way. The system is currently used by neuroscientists 715
forderiving accurateskeletonrepresentationsto beusedfor clas-716
sification, measurements, and simulation purposes [8]. We pre-717
sentedtheoutcomesofauserstudytoevaluateandcomparethe 718
strengthstheproposedsystem. 719
Oursubjectivepreliminaryevaluationshowedthatdomain sci-720
entistsfeelparticularlycomfortableinusingthesystemfor proof-721
readingandeditingpreviouslycomputedskeletonswhiletheystill 722
considertheprocess ofcreatingmedialaxisrepresentations from 723
scratch to be comparable to automated or semi-automated 3D 724
tools,intermsoftimeconsumption,althoughtheyrecognisedhow 725
powerfulthe path stabilizer approach is to find the medial axis 726
automatically, andthecombination ofexternal andinternal trac-727
ing metaphors dramatically speed-upthe creation process. Plans 728
toimprovethe systemincludethe implementationofonline col-729
laborative schemes, in order to distribute the creation process 730
among multiple users, reduce the working time and effort, and 731
atthe same time increase the quality of the output representa-732
tion;andintegrationofvisual analyticstoolsforexploringfeature 733
distributionsinsidemorphologies[59]andtoolsforperforming vi-734
sualanalysisoftopologicaldatarepresentationsassociatedto me-735
dial axisrepresentations [40].Finallywe customized the system 736
fortwodifferentsetups, consideringdirectmetricinteraction,and 737
standard gaming indirectinterfaces [54]. We did not yet investi-738
gatealternativeinterfacesthatcouldspeed-uptheeditingprocess 739
andexploration, like touch-based systems to be attached to the 740
display wall setup or gesturerecognition systems to be attached 741
tothestereoscopicHMDsetup.Weplantoexplorethisavenue,in 742
ordertounderstandwhichinterface ismoreperformingforthese 743
kindofneuroscienceinvestigations. 744
DeclarationofCompetingInterest 745
Theauthorsdeclarethattheyhavenoknowncompeting finan-746
cialinterestsorpersonalrelationshipsthatcouldhaveappearedto 747
influencetheworkreportedinthispaper. 748
CRediTauthorshipcontributionstatement 749 Daniya Boges: Methodology, Formal analysis. Marco Agus: 750 Methodology,Formalanalysis,Supervision,Writing-originaldraft, 751 Writing - review & editing. Ronell Sicat: Methodology, Supervi- 752 sion, Validation. PierreJ. Magistretti: Funding acquisition, Inves- 753 tigation, Formal analysis.MarkusHadwiger:Methodology, Super- 754 vision,Writing -review &editing.Corrado Calì:Project adminis- 755 tration,Datacuration,Methodology,Writing-review&editing. 756
Acknowledgment 757
Thiswork is supported by KAUSTKing AbdullahUniversity of 758 ScienceandTechnology KAUST-EPFL AllianceforIntegrative Mod- 759 elingofBrainEnergyMetabolismhttps://www.kaust.edu.sa/enun- 760 derKAUSTCRG6GrantNo.2313. Wethankalltheparticipantsof 761 the user study from KAUST andfrom the Neuroscience Institute 762 ”CavalieriOttolenghi”.Wealsothanktheanonymousreviewersfor 763
usefulcommentsandsuggestions. 764
Supplementarymaterial 765
Supplementary material associated with this article can be 766 found,intheonlineversion,atdoi:10.1016/j.cag.2020.05.024 767
References 768
[1] Calì C, Agus M, Kare K, Boges D, Lehväslaiho H, Hadwiger M, et al. 3D cellular 769 Q3 reconstruction of cortical glia and parenchymal morphometric analysis from 770 serial block-face electron microscopy of juvenile rat. Prog Neurobiol 2019. In 771
press 772
[2] Coggan JS, Keller D, Calì C, Lehväslaiho H, Markram H, Schürmann F, et al. 773 Norepinephrine stimulates glycogenolysis in astrocytes to fuel neurons with 774
lactate. PLoS Comput Biol 2018;14(8):e1006392. 775
[3] Markram H, Muller E, Ramaswamy S, Reimann M, Abdellah M, Sanchez C, 776 et al. Reconstruction and simulation of neocortical microcircuitry. Cell 777 2015;163(2):456–92. doi: 10.1016/j.cell.2015.09.029 . 778 [4] Calì C, Baghabra J, Boges DJ, Holst GR, Kreshuk A, Hamprecht FA, et al. Three- 779 dimensional immersive virtual reality for studying cellular compartments in 780 3d models from EM preparations of neural tissues: 3d virtual reality for neural 781 tissue. J Comp Neurol 2016;524(1):23–38. doi: 10.1002/cne.23852 . 782 [5] Agus M, Boges D, Gagnon N, Magistretti PJ, Hadwiger M, Cali C. Glam: 783 glycogen-derived lactate absorption map for visual analysis of dense and 784 sparse surface reconstructions of rodent brain structures on desktop systems 785 and virtual environments. Comput Graph 2018;74:85–98. 786 [6] Usher W, Klacansky P, Federer F, Bremer P-T, Knoll A, Yarch J, et al. A vir- 787 tual reality visualization tool for neuron tracing. IEEE Trans Vis Comput Graph 788
2017;24(1):994–1003. 789
[7] Vezzoli E, Cali C, De Roo M, Ponzoni L, Sogne E, Gagnon N, et al. Ultrastruc- 790 tural evidence for a role of astrocytes and glycogen-derived lactate in learning- 791 dependent synaptic stabilization. Cerebral Cortex 2019. 792
