Application For immersive virtual reality rehabilitation through utilization of limb orientations

Academic Year 2016 - 2017

POLITECNICO DI MILANO

Master of Science in Design & Engineering

School of Design

Application for Immersive Virtual Reality

Rehabilitation Through Utilization of Limb

Orientations

Supervisor:

Prof. MARIO COVARRUBIAS RODRIGUEZ

Co-Supervisor: Prof. MONICA BORDEGONI

Graduation Thesis of:

ESER KÖKTÜRK

ID 849958

Abstract

English

Advancing Virtual Reality systems have provided compelling solutions to limb

related medical conditions through rehabilitation. Availability of cost-efficient

commercial VR applications is a determining factor to reach more people who seek

treatment with or without the help from medical facilities. In this thesis, we proposed a

low-cost wireless VR input device and a software application LIMBrehabVR for upper

and lower limb rehabilitation. Implementation and design choices are explained clearly

to represent the capabilities and potential of the VR systems. Patients can interact with

the virtual environment using the IMU based input device which detects the orientation

of the limb segment. LIMBrehabVR can generate user defined simulations that patient

can practice with. User interface of the software presents many settings for the patient

to customize and it is integrated with a method to allow the patient to use a smartphone

as a VR headset display. According to the case study we conducted, viability of the VR

system is proven by healthy subjects for limb rehabilitation. Further developments of

VR technology will be the deciding aspect for the reliability of VR rehabilitation and

what is presented in this thesis can be improved to produce potent results according to

the state of the art immersive VR applications.

Italiano

L'avanzamento dei sistemi di realtà virtuale ha fornito soluzioni convincenti alle

condizioni mediche correlate agli arti attraverso la riabilitazione. La disponibilità di

applicazioni commerciali a basso costo per i consumatori commerciali è un fattore

determinante per raggiungere più persone che cercano di trattare con o senza l'aiuto di

strutture mediche. In questa tesi abbiamo proposto un dispositivo di input wireless a

basso costo di VR e un'applicazione software LIMBrehabVR per la riabilitazione dell'arto

superiore e inferiore. Le scelte di implementazione e progettazione sono chiaramente

spiegate per rappresentare le capacità e il potenziale dei sistemi VR. I pazienti possono

interagire con l'ambiente virtuale utilizzando il dispositivo di immissione basata su IMU

che rileva l'orientamento del segmento degli arti. LIMBrehabVR può generare

simulazioni definite dall'utente che il paziente può praticare con. L'interfaccia utente del

software presenta molte impostazioni per il paziente personalizzandole ed è integrato

con un metodo che consente al paziente di utilizzare uno smartphone come display

auricolare VR. Secondo lo studio di caso condotto, la vitalità del sistema VR è dimostrata

da soggetti sani per la riabilitazione degli arti. Ulteriori sviluppi della tecnologia VR

saranno l'aspetto decisivo per l'affidabilità della riabilitazione VR e ciò che è presentato

in questa tesi può essere migliorato per produrre risultati potenti in base alle ultime

applicazioni VR immersive.

V

Contents

1

INTRODUCTION ... 1

1.1

T

HESIS

O

BJECTIVES

... 2

1.2

T

HESIS

O

RGANIZATION

... 3

RESEARCH FIELD BACKGROUND ... 4

2.1

V

IRTUAL

R

EALITY

... 4

2.2

S

TATE OF

I

MMERSIVE

V

IRTUAL

R

EALITY

S

YSTEMS

... 12

2.2.1

Market Analysis of VR systems ... 12

2.2.2

Immersive Virtual Reality Products ... 14

2.2.3

Technical Requirements ... 19

2.2.4

Fields of Application... 20

2.3

V

IRTUAL

R

EALITY

R

EHABILITATION

... 22

2.3.1

Physical Medicine and Rehabilitation ... 22

2.3.2

Rehabilitation Application of VR ... 23

2.3.3

Medical Problems Related to VR Rehabilitation ... 25

2.4

R

ELATED

W

ORKS

... 27

LIMB ORIENTATION ESTIMATION ... 29

3.1

O

RIENTATION

T

RACKING

... 29

3.2

S

ENSOR

M

ODULE

... 32

3.3

D

ATA

A

CQUISITION

... 35

3.3.1

Microcontroller ... 38

3.3.2

Initialization ... 41

3.3.3

Sensor Outputs ... 43

3.4

C

OMPUTATION OF

A

NGLES

... 50

3.4.1

Euler Angles Approach ... 50

3.4.2

Quaternion Approach ... 54

PROTOTYPE DESIGN ... 58

4.1

C

ONCEPT

... 58

4.2

E

LECTRONIC

C

OMPONENTS

... 61

4.2.1

Data transmission ... 62

4.2.2

Component Assembly ... 65

4.3

L

AYOUT

... 67

4.3.1

Cover Design... 69

4.3.1

Cover Production ... 70

4.4

F

INAL

A

SSEMBLY

... 74

VIRTUAL INTERFACE DESIGN ... 78

5.1

D

ESIGN

A

PPROACH

... 78

5.1.1

Unity Game Engine ... 79

LIST OF FIGURES

……….

VII

LIST OF TABLES

……….

X

LIST OF ABBREVIATIONS

……….

XI

VI

Contents

5.2

A

PPLICATION

F

EATURES

... 80

5.2.1

User Interface ... 81

5.2.2

Main Menu ... 81

5.2.3

Limb Selection ... 83

5.2.4

Position Selection... 85

5.2.1

Instructions Menu ... 87

5.2.2

Settings Menu ... 88

5.3

P

RACTICE

S

IMULATION

... 92

5.3.1

Set Practice Menu ... 92

5.3.2

Practice Testing ... 94

5.4

D

EVICE

I

NTEGRATION

... 97

5.4.1

Bluetooth Setup ... 97

5.4.2

Port Menu ... 98

5.4.1

Device Testing ... 99

5.5

VR

O

PERATION

... 100

5.5.1

VR Settings ... 100

5.5.2

Oculus Rift ... 102

5.5.3

Trinus VR ... 103

CONCLUSION ... 105

6.1

C

ASE

S

TUDY

... 105

6.1.1

Discussion... 106

6.2

R

ESULTS

... 109

6.3

F

URTHER

I

MPROVEMENTS

... 110

BIBLIOGRAPHY ... 111

APPENDICES ... 117

A

A

RDUINO

C

ODE

... 117

B

T

ECHNICAL

D

RAWINGS

... 121

C

U

NITY

C

ODE

... 125

a.

Settings.cs ... 125

b.

LimbFigure.cs ... 132

c.

TestDevice.cs ... 135

d.

PracticeMotion.cs ... 148

e.

SetMenu.cs ... 154

f.

Menu.cs ... 157

g.

LimbPractice.cs ... 168

h.

VRScene.cs... 175

VII

List of Figures

Figure 2.1 The Allegory of the Cave of Plato ... 5

Figure 2.2 Non-immersive, semi-immersive and fully-immersive VR systems ... 8

Figure 2.3 Revenues of VR Applications (by Greenlight Insights, 2017 Report) ... 9

Figure 2.4 Racławice Panorama ... 10

Figure 2.5 Stereoscope of Charles Wheatstone ... 10

Figure 2.6 Telesphere Mask of Morton Heilig ... 11

Figure 2.7 Headsight Headgear ... 11

Figure 2.8 Average head mounted display prices over the years (by Kzer Worldwide) ... 12

Figure 2.9 2016 Global VR Headset Shipment Market Shares (by Strategy Analytics) ... 13

Figure 2.10 Global VR Headset Shipments Forecast (by BI Intelligence, 2016) ... 13

Figure 2.11 VR headsets in the presented order ... 15

Figure 2.12 Immersight positional head tracking ... 16

Figure 2.13 Kinect for Xbox One ... 17

Figure 2.14 Geomagic Touch Haptic Device ... 17

Figure 2.15 Cyber Grasp Data Glove ... 18

Figure 2.16 Oculus Touch controller ... 18

Figure 2.17 Fully-Immersive VR first person shooter ... 20

Figure 2.18 Immersive VR education of preteen students ... 21

Figure 2.19 VR training demonstration by the Netherlands Army ... 21

Figure 3.1 Nickel tuning fork gyroscope ... 30

Figure 3.2 Working procedure of a capacitive accelerometer ... 31

Figure 3.3 Working principle of a Hall magnetometer ... 32

Figure 3.4 MPU-9255 IMU Module ... 34

Figure 3.5 Two wired serial interface that uses clock pulses to transfer bits of data ... 35

Figure 3.6 Connection of multiple slave and master devices to I2C bus ... 36

Figure 3.7 8-bit I2C Communication ... 36

Figure 3.8 Configuration of master device and slave devices for SPI Protocol ... 37

Figure 3.9 7-bit SPI Communication... 37

Figure 3.10 Arduino Nano microcontroller ... 39

Figure 3.11 Arduino Nano 3.0 pinout ... 40

Figure 3.12 Uncalibrated gyroscope measurements for 2000 samples ... 45

Figure 3.13 Uncalibrated Gyroscope Means of Measurements for 10 seconds ... 45

Figure 3.14 Calibrated gyroscope measurements for 2000 samples ... 46

Figure 3.15 Calibrated Gyroscope Means of Measurements for 14 seconds ... 46

VIII

List of Figures

Figure 3.17 Uncalibrated Magnetometer measurements around X axis ... 49

Figure 3.18 Calibrated Magnetometer measurements around X axis ... 49

Figure 3.19 Basic block diagram of open-loop orientation estimation process ... 50

Figure 3.20 Tait–Bryan angle representation ... 51

Figure 3.21 Filtered pitch and roll angles inertial frame correction ... 52

Figure 3.22 Representation of gimbal lock on an object ... 53

Figure 3.23 Madgwick filter correction with β=0.1... 55

Figure 3.24 Madgwick filter correction with β=0.4... 56

Figure 3.25 Madgwick filter correction with β=1... 56

Figure 4.1 Concept layout with three IMUs and wired connection ... 60

Figure 4.2 Improved layout (microcontroller + battery + wireless transmitter + IMU) ... 60

Figure 4.3 ZNTER 9V rechargeable battery with Micro USB connection ... 62

Figure 4.4 HC-06 Bluetooth module ... 63

Figure 4.5 Wiring diagram of the components ... 66

Figure 4.6 Assembly of the electronic components ... 66

Figure 4.7 Layout I with overall dimensions ... 68

Figure 4.8 Layout II with overall dimensions ... 68

Figure 4.9 Testing cover design choices through cardboards... 70

Figure 4.10 Printing configurated for models in Simplify3D ... 71

Figure 4.11 Top cover printing tests ... 72

Figure 4.12 Base cover printing tests ... 72

Figure 4.13 Final CAD model of base cover ... 73

Figure 4.14 Final CAD model of top cover ... 73

Figure 4.15 CAD assembly of the input device... 74

Figure 4.16 Exploded View of the input device ... 75

Figure 4.17 Assembled physical input device ... 76

Figure 4.18 Battery charging indicator when USB connected and disconnected ... 76

Figure 4.19 Input device power OFF and ON ... 77

Figure 4.20 Three input devices attached to left arm ... 77

Figure 5.1 LIMBrehabVR Main Menu ... 82

Figure 5.2 Limb Selection Menu ... 83

Figure 5.3 Limb amount switch in Limb Selection Menu ... 84

Figure 5.4 Selection of left arm limb segments in Limb Selection Menu ... 84

Figure 5.5 Position Menu initial view ... 85

IX

List of Figures

Figure 5.7 Default Standing, Sitting and Laying positions ... 87

Figure 5.8 Instructions Menu... 87

Figure 5.9 Controls Settings ... 89

Figure 5.10 Models Settings ... 90

Figure 5.11 Man Model Clothes (1 to 4) ... 90

Figure 5.12 Woman Model Clothes (1 to 4) ... 91

Figure 5.13 Environment Settings ... 91

Figure 5.14 Available virtual environments (1 to 4) ... 92

Figure 5.15 Set Practice editor ... 93

Figure 5.16 Positions added to the list in Set Practice Menu ... 94

Figure 5.17 Content of an example file saved in Set Practice Menu ... 94

Figure 5.18 Practice View Menu ... 95

Figure 5.19 Open File browser in Practice View menu... 95

Figure 5.20 Practice View mirrored limb motion ... 96

Figure 5.21 Paired input devices in Windows 10 Bluetooth settings ... 97

Figure 5.22 Checking for the input device COM ports in Windows 10 Bluetooth settings .. 98

Figure 5.23 Device Setup menu to select COM ports... 99

Figure 5.24 Dropdown list displaying the available ports in Device Setup menu ... 99

Figure 5.25 Test Device operation... 100

Figure 5.26 VR Operation Settings menu ... 101

Figure 5.27 Simulation presented on the screen for Oculus Rift with input device ... 102

Figure 5.28 Oculus Rift with the custom input device ... 103

Figure 5.29 VR Operation with Trinus ... 104

Figure 5.30 Smartphone and PC wireless Trinus VR connection... 104

Figure 6.1 Questions that are asked to the subject before testing for the case study ... 106

Figure 6.2 Test 1 of the case study with input device and Oculus Rift ... 107

Figure 6.3 Test 2 of the case study with simulation and Oculus Rift ... 107

Figure 6.4 Test 3 of the case study with simulation and Trinus VR ... 108

X

List of Tables

Table 2.1 Comparison between various VR implementations (Kalawsky, 1996) ... 7

Table 3.1 9-Axis IMU comparison ... 33

Table 3.2 Arduino Nano 3.0 Specifications... 39

Table 3.3 Gyroscope Configuration Chart ... 42

Table 3.4 2-bit and 8-bit values for gyroscope and accelerometer sensitivity ... 43

Table 3.5 Gyroscope sensitivity scale factors ... 44

Table 3.6 Accelerometer sensitivity scale factors ... 47

Table 4.1 HC-06 Bluetooth module specifications ... 64

Table 4.2 PLA cover printing specifications for CTC Bizer 3D printer ... 71

XI

List of Abbreviations

DOF

Degrees of Freedom

HMD

Head mounted Display

I

2

_C

_{Inter-Integrated Circuit}

IMU

Inertial Measurement Unit

MEMS

Microelectromechanical Systems

UI

User Interface

PC

Personal Computer

PLP

Phantom Limb Pain

SCI

Serial Communication Interface

SPI

Serial Peripheral Interface

VE

Virtual Environment

VR

Virtual Reality

1

Introduction

Over the recent decades, technological advancements played a key role in the

development of contemporary ideas which led to the creation of various industries and

products that changed the everyday life of society. Evolution of smartphones from

analog phones can be one of the best examples to present the effect of the developing

technology on the form of a commonly used product. Although the evolution would

not have been possible if microprocessors [1] could not provide the sufficient

computation for complex operations and graphics. Technology of processors is

progressing rapidly and according to Moore's Law [2] processors will continue to

improve in performance while decreasing in size. Therefore, the remarkable potential

of high-speed computing can advance many applications to be more efficient in the

near future and Virtual Reality systems are in a great position today to benefit from

the upcoming improvements. VR technology requires rendering of three-dimensional

graphical structures created in a digital environment with low latency between

real-time and simulated data to present accurate visuals. User experience with VR is very

dependent on how real and robust the interactions with virtual environment are and it

is still on the initial stages of development but expected growth of the VR market [3]

shows that companies are willing to invest further into these systems.

Currently there are various products [4] related to VR systems that are

commercially available such as hardware, games, videos and many different

applications. Experiencing virtual environment is mostly through a head mount

displays which are headsets you wear to visually observe visual scenes. HTC Vive,

Oculus Rift and PlayStation VR are the most common headsets that are available with

sensors, a screen and other devices that can help you to interact with VE but they are

required to be connected to a computer or a console while operating. A more practical

HMD approach uses your smartphone as a screen such as Google Cardboard or

Samsung Gear VR.

2

Introduction

Today the usage of VR is mostly for entertainment purposes while there are many

professional fields that can use the potential of this system. More technical adaptation

of this technology can be very effective in the medical field. There are several examples

[5] for VR practices in medical field such as exposure therapy, rehabilitation therapy,

surgery simulations to present different solutions to problems that can be treatable. This

paper will focus on the rehabilitation aspect of virtual reality.

Rehabilitation using virtual reality is getting more notice each day. Versatility of

practices that can be developed for therapy purposes is a crucial factor to be considered

for rehabilitation with VR, to test the efficiency of a solution that is implemented for a

certain condition. Change of parameters in virtual environment is relatively simpler

than in physical world so achieving effective methods would be more common than

the physical applications that are depended on advanced mechanisms. Therefore, the

potential of VR rehabilitation will be improved with the advancements in technology

of the VR systems.

1.1

Thesis Objectives

In this thesis, we propose a method to rehabilitate patients that have medical

problems related to their upper and lower limb segments through immerse virtual

reality interaction therapy. Limb motion is generated using a custom device or a

simulation that is created in VR application. Device is designed to be an inexpensive

solution to track rotation using a sensor and sending data wirelessly to the application.

Virtual environment is created using Unity 3D Game Engine for Windows OS. User

interface is designed to be easy to operate and visually communicative while providing

different options regarding to the preferences of the user. Custom simulations can also

be created for limb motion from the application for generating a sense of the limb

motion without the real-world movement. These simulations can also be practices for

the patient to be observed in virtual environment from a screen by mimicking the same

movements. Application will available for VR glasses that use smartphones and Oculus

Rift headset. In the end, thesis will present a practical and cost-effective solution to a

rehabilitation method that can help patients to get treatment they need from their

homes or medical centers.

3

Introduction

1.2

Thesis Organization

Chapter 2

This chapter discusses the developing technologies related to Virtual Reality and how

they are used in medical field to be a solution for different treatment methods. Medical

problems that are treatable with VR are explained. Further applications related to the

topic are analyzed.

Chapter 3

This chapter focuses on converting limb rotations, to digital information to be processed

by the VR interface. Extracting data from a commercially available inertial

measurement unit using a microprocessor is explained. Discussion of the angle

measurement method is conducted.

Chapter 4

Development of the sensor as a product is shown in this chapter. Each component of

the device is explained. Production of the parts is presented. Final analysis of the device

is conducted.

Chapter 5

This chapter shows the capabilities of the interface designed for VR environment in

Unity Game Engine. User interface and features of this application is explained. Device

and Virtual interface co-operation is discussed.

Chapter 6

The last chapter will be about the conclusive results of the project with a conducted case

study and future improvements that can be applied to the topics that are presented.

.

Research Field Background

In this chapter, we present general information about topics that are related to

thesis such as basis of human perception with its relation to virtual imaging, how

virtual reality systems emerged to their current stage and providing technical

information about how VR systems function. Afterwards, interaction between physical

world and virtual environment is explained according to sensory devices and modules.

Further information will be provided about rehabilitation with virtual reality, analysis

of the medical problems that can be solved using VR therapy and how are these issues

being treated currently including for both VR and other solutions. Last part concludes

with a discussion about the studies and projects related to the aim of the thesis.

2.1

Virtual Reality

Perception of Reality

Concept of reality has been a topic of discussion since the earlier ages of humanity.

One of the most well-known example about the notion of reality is presented by the

Ancient Greek philosopher Plato through “The allegory of the cave” [6]. In this allegory,

there are prisoners locked in a cave since childhood and they are observing the shadows

of figures on the cave wall created by the puppet masters through the light of fire. What

Plato suggests is that the prisoners perceive what they observe as reality but if one of

the prisoners discovers a way out of this cave, he will understand that it was only a

mere reflection of reality. This thought experiment alone can explain how our reality

can be manipulated through delusive information that we acquire and how we perceive

the world was a matter of questioning even thousands of years ago. Although we are

starting to learn about the mechanisms that drive our understanding, it is still early for

us to fully uncover our perception.

5

Research Field Background

An American neuroscientist David Eagleman claims that [7] all the sensory

information that we experience through our senses such as hearing, touching, sight,

smelling and gustation take place in an organ that has no direct contact with the outer

world, the brain. Therefore, the sensation of feeling is the processed information by the

brain from the mechanical inputs gathered by the senses. It has been discussed [8] that

there are more than five senses as we know of and how we perceive the real world is

from the combination of all this information. Although, we have yet to understand the

brain completely, there are ways to implement new sources of information to achieve

various methods of sensing through sensory augmentation. The Vest [7] is a product

designed by David Eagleman which translates soundwaves into patterns of vibrations

trough a wearable vest for the people with hearing disabilities. It is also tested that in a

brief time a deaf subject began to understand words that are presented to them through

vibrations from the device. This revolutionary innovation suggests that we have the

capacity to understand sensory information through non-biological accessories that we

are not born with. Since we can program our brains for new type of learning methods,

6

Research Field Background

suggestion of using virtual information to manipulate our perception is not that far

from this topic.

We tend to imagine ourselves in unreal situations or fantasies to have a sense of

comfort, happiness, anxiety and various feelings time to time which can also be

described as a virtual production of our brains. Therefore, creating a virtual world is

something that is natural to us and today we can utilize the technology to achieve our

requirements. All the actions that we do to interact with the world can be simulated

through electro-mechanical devices and a digital world can be visually generated for

us to observe. When recreation of the reality can be performed in an accurate manner

through technological devices, our perception can be exploited for any required

purpose.

General Information about Virtual Reality

The near-reality experience that we perceive through simulated information can be

described as Virtual Reality [9]. Currently, the most solid approach to VR is by using

highly advanced computer graphics to render three dimensional objects with accurate

visuals to form a virtual environment as almost as it is real and considering the potential

of the sensor technology, extracting information from the real world is a way to enhance

the VR experience by allowing it to be interactable.

Recent developments in processors [2] and LCD [10] technology enhanced VR with

sharper visuals, more detailed image quality and improved performance related to

older applications. Introduction of higher resolution screens is an extensive

advancement for virtual reality systems since the concentration of high number of

pixels in a designated space can present real world looking images. Although, it

requires more computation for each frame, accordingly central and graphical

processing units must be capable of providing calculations for each pixel while

rendering the 3-D graphical structures. We are observing VR technology getting more

notice than before with the evolution of processors that can carry out billions of

operations per second. Potential of VR systems are correlated with the performance of

the technological applications and in the near future there would be more intriguing

versions of VR.

7

Research Field Background

Each virtual reality system has a certain level of interaction depending on its

application. Immersion is the common term that is related to the level of engagement

of user with VR systems and it can be described as a mental and physical involvement

without the awareness of being in an artificial world [11]. More immersive a VR system

is, there will be more separation from real world perception and attraction to virtual

environment will be more compelling. Levels of immersion are implemented

depending on the particular requirement while there are pros and cons of each

application. Virtual reality systems can be generally categorized in three distinct

immersion levels as non-immersive, semi-immersive and total-immersive [12]. Most

effective VR experience can be acquired from fully immersive VR systems (see Table

2.1) according to Kalawsky and even though decades have passed since his analysis,

technical difficulties and limitations he claimed about fully immersive systems are still

persistent. Performance of the system is highly dependent on the relation between

latency (lag) and resolution of the application. The delay we observe in the virtual

application causes lower frame rates and directly affects the VR experience. Therefore,

to minimize the delay, optimization of resolution is required depending on the

specifications of the hardware. Development of capable processors are a solution to

overcome the problems related to the latency-resolution optimization issues and

immersive systems have an immense potential for the future implementations.

Main features

Non-immersive

Non-immersive

Full Immersive

Resolution

High

High

Low - Medium

Scale (perception)

Low

Medium - High

High

Sense of situation awareness

Low

Medium

High

Field of regard

Low (50°)

Medium (150°)

High (360°)

Lag

Low

Low

Medium - High

Immersion

None - Low

Medium - High

Medium - High

Table 2.1 Comparison between various VR implementations (Kalawsky, 1996)

Fully-immersive systems use head mounted displays (HMD) and sensory input

devices that can properly track the motion of the user [12]. Thanks to that, simulation

8

Research Field Background

would be more accurate representation of the reality. On the other hand,

non-immersive systems are designed to present virtual environment through windows or

portals such as high definition screens that can be processed by desktop computers.

User can interact with the virtual environment but would be aware of the distinction

between the virtual and the real world. Therefore, VR experience would rather be

limited. Another solution between fully and non-immersive VR is semi-immersive

applications. They require more processing power than non-immersive and interaction

with the VE is generally more detailed. Although, there would still be real world visuals

present in the system and total simulation is not achieved. One of the best examples for

this type of system are flight simulators, which provide a sense of virtual environment

that is not digitally dominant.

According to Greenlight Insights more than half of the VR market revenues of 2017

are based on HMDs (see Figure 2.3). This information presents the fact that individuals

are likely to invest into VR systems and fully immersive devices seems to be the main

choice due to commercial availability. Many enterprises that have taken notice into

developing their own brand of devices and applications for the VR industry. Overall

capabilities of HMDs are currently based on motion tracking through camera, head and

eye tracking with internal sensors and displaying VE from an LCD screen though lenses

accompanied by gadgets to provide a better interaction with virtual world.

9

Research Field Background

Earlier Applications of Virtual Reality

Recognition of virtual reality technology today is related to computer generated

images and digital environment but there were earlier attempts of implementing the

basis of near reality experience throughout history. One of the best examples can be the

panoramic paintings that were available in the early 19

th

_{century [13]. These paintings}

consisted of a 360-degree field of view of a landscape (see Figure 2.4) including

additional props that were implemented to present the sense of depth in a closed exhibit.

This approach can also be considered as a non-digital and non-immersive attempt for

virtual reality. Panoramic pictures are still being used in digital VR applications for

background images or directly for entertainment purposes.

Another essential step was the invention of stereoscope (see Figure 2.5) by Charles

Wheatstone [14] that shaped how we visually observe the virtual reality today. After

the invention of photography, this device is used to form 3D images from pictures by

coupling them together. This particular approach is used by many VR headsets to

achieve the illusion of depth and larger field of view.

10

Research Field Background

In the mid-20

th

_{century the first VR headset called is designed by a}

cinematographer named Morton Heilig [13]. This headset contained a sound system

with earphones, for visuals it had television tubes combined with vison lenses and even

air nozzles for nostrils that can pump out scented air. Overall shape of the innovation

was very familiar to what we have today as HMDs. Another invention he made was

called Sensorama which was a fully immersive VR system that had an appearance

similar to an arcade game machine and it functioned with the similar features of his VR

headset.

Figure 2.4 Racławice Panorama

11

Research Field Background

After 60s VR systems as we know of began to emerge. A military funded device called

Headsight [13] involved a video screens for the eyes and a motion tracking system using

a camera which can be categorized as the first head mounted display. The term “virtual

reality” was officially declared for applications that create simulations of reality and at

the end of the century gaming industry has seen the potential of the VR systems.

Nintendo Virtual Boy and SEGA VR glasses were the initial commercially available VR

headsets for gaming industry.

After the millennia, accelerated technological developments opened many doors

for virtual reality. Limited capabilities of the systems are immensely improved.

Although it is undeniable that, present VR applications were mostly concepted decades

and maybe even centuries ago which was fulfilled by current advancements of today.

It is an indication for VR systems that they will be expanding excessively in the near

future, comparable to previous examples.

Figure 2.6 Telesphere Mask of Morton Heilig

12

Research Field Background

2.2

State of Immersive Virtual Reality Systems

In this thesis, we would like to focus on fully immersive virtual reality systems

therefore more technical research about their market value, state of the art products and

technical specifications will be discussed to explain our reasoning behind the design

approach.

2.2.1

Market Analysis of VR systems

Essential information can be gathered from the open market analysis and trends

about the products and the industries. Virtual reality applications are currently a

growing industry and immersive VR products such as headsets are getting attention of

the consumers and companies (see Figure 2.3) considering their revenue percentage.

There are many well-known global tech companies that are invested in to VR industry

such as Google, Facebook, Sony, Samsung [15] which creates an encouragement for

competition. Expansion of commercially available VR products mostly exists as in

terms of VR headsets or input devices (sensors) which can be sold to individual

consumers. Competition between the enterprises are lowering prices (see Figure 2.8)

and generating more innovative solution to VR systems.

13

Research Field Background

Consumer demand is a crucial factor to determine the course of the industry.

According to the analysis by Strategy Analytics [16] for 2016, Google Cardboard, which

is a smartphone based VR headset, dominated the market (see Figure 2.9) in terms of

amount of sales. Although most revenue was made by Samsung Gear VR with 35% of

overall VR shares. Considering both devices for designed for mobile phone usage, safe

to say that consumers are preferring inexpensive and practical solutions for immersive

VR systems. A different study by BI Intelligence [17] estimates a trend for smartphone

headsets will continue and dominate the market (see Figure 2.10).

Figure 2.9 2016 Global VR Headset Shipment Market Shares (by Strategy Analytics)

14

Research Field Background

The market analysis of the VR applications and products was an essential source

of influence for the thesis. Proposal of the was required to reflect the requirements of

the individual consumer since our main aim is to design for both home and medical

center usage. Cost effectiveness should also be taken into consideration for the

application of more practical solutions.

2.2.2

Immersive Virtual Reality Products

Variety of the commercially available VR devices have been presented in the

previous section and in this thesis, we propose to design a system that can be used

practically by anyone similar to these products. Moreover, further research is based on

state of the art devices which are leading the industry in VR technology.

Headsets

Headsets are the most essential parts of the fully immerse virtual reality application.

These products are equipped with head orientation trackers, LCD screens, additionally

proximity sensors and sound systems. They are strapped to the head of the user as

goggles and transmit the visual information from the display to the eyes through lenses.

Best performing headsets are generally wired to computers to operate while they

provide high resolution visuals although they can be costly. A different type of headsets

uses the potential of the smartphones. They can operate wirelessly and they are

inexpensive compared to desktop solutions but their performance is limited to the

specifications of the mobile device. The most commonly used HMDs are presented

below:



Oculus Rift: This product was the initial headset that started current competitive

market for VR headsets. It was presented through Kickstarter website and acquired

by Facebook Inc later [18]. There are two displays implemented for each eye in 2160

x 1200 resolution at 90 Hz refresh rate. Visual data is transferred through HDMI

cable and an USB cable to power the system (an additional adapter can be used for

additional power). System also contains a position tracker camera and earphones

[19].

15

Research Field Background



HTC Vive:

This headset was created by the cooperation of HTC, a Taiwanese

smart mobile devices company, and Valve Cooperation which is the owner of

largest video game distribution platform Steam [20]. It provides similar resolution

with the same refresh rate of Oculus Rift but aspect ratio is 9:5 which presents a

taller field of view. There are 70 sensors implemented to this system.



Sony PlayStation VR:

Sony developed this headset as an accessory for its

gaming console PlayStation. It is the cheapest among the popular head mounted

display systems. It contains a single display with a resolution of 1920 x 1080 display

with a refresh rate up to 120 Hz [21].



Samsung Gear VR:

This headset is designed to be compatible with latest

Samsung smartphones. It can connect to a mobile phone through USB and can

provide manual control over VR application [22].



Google Cardboard: Highest shipped headset in the world is design by Google. It

is a headset made from actual cardboard and added a socket for the mobile phone

with optional vision lenses for each eye [18].

16

Research Field Background

Input Devices

Headsets can provide most of the interaction required for the virtual environment

but more detailed tasks require more specific hardware to assist the user. Devices that

can provide other functions for the VR systems are called input devices. Input devices

are generally integrated with sensors to gather information from the user actions.

Depending of the method of interaction, input devices can be categorized as listed

below [23]:



Tracking: The main objective of the tracking devices is to determine the position

or the orientation of the real object. The information can be achieved by several

techniques such as magnetic, optical, acoustic, mechanical and inertial tracking.

 Position Tracking:

This method of tracking can be achieved by calculating

absolute position of an object relative to a fixed coordinate system or relative

position to the initial position of the object.

 Orientation Tracking:

Similar to the position tracking it can be achieve

depending on a global reference system (gravitational vector or magnetic field

of the Earth) or a local reference system such as previous/initial orientation.

Tracked orientation can be represented in angular form with several

approaches such as Euler angles, Quaternions etc.

17

Research Field Background



Kinect:

This input device was designed for Xbox 360 console by Microsoft. It is

capable of body and face tracking with integrated voice recognition. It is also

equipped with an infrared camera to determine the depth of field through the

distortions occurred in a projected pattern form by the camera. Additionally, thanks

to the software it can detect gestures and create a 3D rig for the human figure. Kinect

can be considered as a tracking device although, it is a more complex system with

many extra functions. There are also devices using similar approach such as Leap

Motion sensor which is for hand and finger tracking.



Haptic Devices: These devices are designed to simulate the sense of touching

when the user interacts with a virtual object. It is possible to feel and influence

objects in a passive or an active manner. Interaction can be achieved by different

contact shapes such as point, line or three-dimensional and the force inputs/outputs

must be optimized depending on the application.

Figure 2.13 Kinect for Xbox One

18

Research Field Background



Data Gloves: These devices are gloves that are designed to detect the position and

orientation of each finger segment. It can be used for manipulating virtual objects

and interacting with the virtual environment. Typical application methods are optic

fibers (uses changes in the light that goes through the fiber), mechanical (use of

potentiometer or elongation of a tube), resistance gloves (uses piezoresistor) and

optic gloves (positional tracking by cameras using markers).



3D Input:

These devices can be categorized as controllers with a capability of

multiple degrees of freedom inputs. Joysticks, wheels, spheres and styluses are used

as a common approach to generate information. Many HMDs come with their own

3D input controllers since it is the most generic way to interact with VE.

Figure 2.15 Cyber Grasp Data Glove

19

Research Field Background

2.2.3

Technical Requirements

Immersive VR demands a system with the highest performing hardware available

to operate efficiently. Most of the VR headsets contain double displays that causes task

of processing visual information to be exhaustive for both CPU and GPU. If there is

occurrence of noticeable latency and frame rate drops during operation, it can cause

dizziness for the user. This problem is called “simulation (motion) sickness” [24] and it

might also occur after extensive usage of the headset. Therefore, VR systems should be

used carefully.

Many hardware manufacturers and VR application developers state certain

specifications for the desktop systems. It is possible to automatically test a PC for VR

performance. Hardware recommendations (by GPU producer Nvidia, 2017) for the

latest HMDs that are released are listed below:



GPU: NVIDIA GeForce GTX 1060 or greater



CPU: Intel Core i5- 4590 equivalent or greater



Memory/RAM: 8GB+ RAM



Video Output: 1x HDMI 1.4



Ports: 3x USB 3.0



OS: Windows 7 SP1 (64bit) or higher



Driver: Oculus – 361.91 and newer



HTC – 361.75 and newer

Development of the new head mounted displays, more detailed VR applications

and upgraded driver versions can affect the hardware requirements. Moreover,

hardware specifications for desktop VR will involve newer models for future

appliances.

Smartphone headsets also require certain specifications for a decent VR experience.

For active head tracking and larger field of view, a magnetometer and an accelerometer

is mandatory to determine the orientation of the smartphone. Resolution of the display

must be at least Full HD (1080p) and mobile device must contain minimum two core

processor while running at 60 Hz refresh rate. Bluetooth and USB connection can also

be a requirement for several headsets [25].

20

Research Field Background

Figure 2.17 Fully-Immersive VR first person shooter

2.2.4

Fields of Application

Potential of the VR systems has been noticed by many industries that led them to

design various VR applications in their fields of work [26,27]. Some of the major

examples are listed below:



Entertainment:

Commercially available devices played a significant role for the

development of the VR technology. Many global gaming system developers such as

Sony and Valve entered into the VR market and it encouraged other game

developers to focus on this innovative technology. Gaming systems are an

important part of the growing VR industry (mobile and desktop applications) and

one of the most recognized examples. Another innovative approach was VR movies

and concerts which has a lot of potential for future investments.



Medical: Potential of VR in medical field is something that cannot be overlooked.

Technology was initially used for phobia treatment and overcoming post-traumatic

stress disorder, currently there are applications such as VR surgery and

rehabilitation. In this paper, further discussion of virtual reality rehabilitation will

be made.



Education: VR allows many costly and complicated tasks to be done in simulated

environment which allows inexperience people to practice their skills in various

fields such as medicine, engineering and aviation. Another, fascination method is to

visualization of events and locations for the elementary school students for more

interactive learning.

21

Research Field Background

Figure 2.18 Immersive VR education of preteen students



Military: There are extensive exercises and tasks required for the military training

and practice. VR applications are being used to train troops for active duties such as

combat, bomb disposal and decision making.



Automotive:

Industry leading companies such as Ford, Audi and Toyota are

investing into VR systems to test their products and analyze design choices by

presenting them to potential consumers through the 3-D VR model of the

automobiles. Driving simulators with VR is a potential solution to train

inexperienced drivers or the people who are learning how to drive.

22

Research Field Background

2.3

Virtual Reality Rehabilitation

2.3.1

Physical Medicine and Rehabilitation

Physical medicine and rehabilitation is the field of specialty that focuses to

diagnose, prevent and rehabilitate patients who experience functional limitations

resulting from injuries, diseases, or malformations [28] that are affecting the brain,

spinal cord, nerves, bones, joints, ligaments, muscles, or tendons [29]. Treatment

methods mostly use non-surgery approaches such as thermotherapy, phototherapy,

ultasoundtheraphy, electrotherapy, magnetotherapy and physical nerve simulation

with state of the art applications [30]. Medical conditions that affect physical and

physiological functionality of the patients can be categorized as (by the definitions of

World Health Organization, 1980) listed below [31]:



Impairment: Psychological, physiologic and/or functional abnormalities or issues

occurred due to acquisition of a painful condition.



Disability:

Any restriction and/or lack of ability to perform (commonly due to

physical impairments) an activity that is considered in the manner or within the

range considered normal for an individual in society.



Handicap: Disadvantage that an individual possesses due to the impairment and

disability that affects his or her role in society.

Rehabilitation process aims to restore functionality and provide further training to

minimize social or individual discomforts for the people with impairments, handicaps

and disabilities. Duration and intensity of the process can vary depending on the

diagnosis of the medical issue. Therefore, treatment methods may be required to be

performed in medical facilities for more severe conditions or they can be performed by

the patients from their homes if they are comfortable performing [32]. An intact

rehabilitation program can consist of:

23

Research Field Background



Definitive methods of therapy for medical conditions patient possesses



Advice about setting up home of the patients to maximize their function and safety



Aid with wheelchairs, splints and other medical equipment that might be required

for the patient



Aid and cooperation for the financial and social issues of the patient

Activities that are essential to the rehabilitation process must be performed

according to a schedule with specific repetitions and limitations that is approved by a

medical expert. Excessive or uncontrolled exercises might cause further discomfort and

injuries that might be dangerous for the progress and the health of the patient.

Therefore, medical team has a responsibility to clearly present detailed analysis of the

conditions to the patient before active rehabilitation is initiated.

An effective rehabilitation has various benefits depending on the method of

treatment such as capability of managing pain without the requirement of using

medicinal opioids, total recovery from trauma and injury with improved movement

and mobility, recovery from stroke and paralysis, fall prevention with better balance

management and coping with the age-related problems [33].

2.3.2

Rehabilitation Application of VR

Technological developments establish different approaches to numerous

applications in many fields of expertise. Growing potential of immersive virtual reality

technology has been proven to be an effective method to simulate artificial

environments and engage user with various task in the digital world. Medical field of

physical medicine and rehabilitation, which conducts diagnosis of various medical

impairments and provides treatment methods through user activities, has been

effectively practicing solutions that rely on virtual reality systems. Virtual reality

rehabilitation treatment is called virtual reality therapy (VRT).

Reliability and efficiency of the virtual reality rehabilitation has been a topic of

discussion for decades. It has been presented that neuropsychological effect of an

24

Research Field Background

impairment which is related to a brain injury or neurological disorder can be estimated

through stimuli, forced by virtual reality [34] and it can serve as an improvement over

traditional techniques. More precise measurement about effects of an impairment can

be conducted for domains of the brain that are being affected by the dysfunctionality of

the patient. Therefore, more effective diagnosis can be made while cause of the problem

can be observed through immersive virtual reality that can introduce more efficient

methods of treatment. Ability to manipulate parameters in an application of VR

rehabilitation can be effortlessly performed to adapt according to the requirements and

capacity of the patient.

Various medical problems can be potentially treated using VR systems. Moreover,

rehabilitation applications should be implemented according to the diagnosis of the

patient. Classification of VR rehabilitations can be categorized into three groups for

specific patient population and conditions [35]:



Musculoskeletal:

Patients that are suffering from bone or muscle injuries



Post-Stroke:

Patients that have survived

a neural hemorrhage, or blood clot to

the brain, resulting in paralysis in certain parts of the body



Cognitive:

Patients that with psychological disorders, such as hyperactivity,

post-traumatic stress and phobias

The main aim of the virtual reality therapy is to provide an interactive and

encouraging approach to the rehabilitation while presenting an immense variety of

means to accomplish that deed. Therefore, the practice of using games and entertaining

activities is a favorable way to help the patient to engage with the activity. Additionally,

progression of the treatment can be shown to the user through achievements obtained

in the VR game or the application which can be a crucial factor in terms of increasing

mental dedication. A fully-immersive VR option can isolate the patients from

real-world limitations and guide them to overcome their discouraging perceptions about

their physical conditions. Virtual environment can also be designed according to the

preferences of the patients to make them comfortable during therapy.

25

Research Field Background

2.3.3

Medical Problems Related to VR Rehabilitation

Practices of VR has been proven to be effective for numerous medical conditions.

Capabilities of VR rehabilitation have provided a new perspective for the traditional

solutions that are still being practiced.

Stroke

Interruption of blood supply to the brain due to a clot (ischemic stroke) or the

leakage of blood pressures the brain and causes damage to that area (hemorrhagic

stroke) can result in a stroke [36]. Damaged parts can heal over time but there could

also be permanent conditions such as speaking problems, paralysis, co-ordination and

comprehension problems. Most common type of stroke is ischemic which is accounted

for 85% of all the strokes [36]. There is another type of stroke which is called transient

ischemic attack (TIA) that occurs when blood flow is briefly interrupted. TIA generally

has temporary conditions but it can be an indication of a later major stroke. People that

are affected by stroke can present differences from their states in terms of mental and

physical. Stroke patients have to be treated with medication or with surgery to prevent

the effects of the stroke while the post-stroke treatment is also necessary. Rehabilitation

after a stroke is essential to the recovery of the patient. There are many methods for

rehabilitation such as speech therapy, physical therapy and occupational therapy.

Another approach to rehabilitation with virtual reality is proven to be effective by

several studies [37] and it can accelerate the recovery of the stroke patients.

Phantom Limb Pain

After an amputation of a limb (sometimes other body parts), feeling of pain that

occurs in the no longer existing body part is defined as the phantom limb pain. This

sensation of pain generally starts after first few days of amputation and studies show

60–80% of amputees are affected from this medical condition [38]. Duration of the pain

can vary from patient to patient and it has a potential to diminish over time. Causes of

PLP are not very clear but it has been reported that changes in peripheral and the central

26

Research Field Background

nervous system caused by a nerve injury can be one of the reasons. Most effective

treatments are medicinal drugs but there are also non-medical treatments such as

electrical nerve stimulation (TENS), vibration therapy, acupuncture, electroconvulsive

therapy. It has been suggested that “mirror therapy” [39] can potentially be used the

reduce the sensation of pain. This approach can be very effective with the help of VR

systems and there have been several applications regarding to that.

Parkinson’s

Degenerative disorder that occurs in the brain cells that are dedicated for body

movement is diagnosed as Parkinson’s disease [40]. Death of the dopamine inducing nerve

cells cause tremor, slowness, stiffness, and balance issues. Cause of the Parkinson’s is

mostly unknown and there are several studies that are being conducted currently to

discover the reasons behind occurrence of the disorder. Management of Parkinson’s

disease can be accomplished through self-care, medication and surgery. Active and

controlled exercises without forcing the body can help patients cope with the conditions

related to disorder. VR therapy for PD is currently being practiced effective results [41]

have been obtained from it.

Phobia

Phycological disorder that causes extreme and irrational obsession of fear can be

described as a phobia [42]. Obsession of fear can be related to an object, a location or an

action. Risk of developing this condition can depend on the genetics or the

environmental factors. Generally distressing and traumatizing events can trigger

phobias while people who are affected by substance abuse, brain injury and depression

have a higher risk of occurrence than others. Treatment methods can involve

medication such as antidepressants and anti-anxiety drugs to relieve the patient from

their issues or more commonly cognitive behavioral therapy (CBT) which exposes

patient to the source of the fear. Virtual reality rehabilitation can be used for CBT to

face the patients to what they fear in a controlled manner. This can result in declination

of anxiety in social life and reduced negative reactions caused by the phobia.

27

Research Field Background

Post-Traumatic Stress Disorder

Traumatic and terrifying events can damage the mental health of a person resulting

in post-traumatic stress disorder (PTSD) [43]. A person with PTSD displays problems

with adjusting to everyday routine and social interactions. PTSD causes reoccurring of

unwanted memories, flashbacks of the traumatic events, extreme reactions to subjects

related to the event and sleeping problems. Reasons behind the cause of the mental

problem is not clear but there are several hypotheses why it happens such as stressful

experiences, brain structure, features of the personality and inherited mental health

risks. Medical treatment can include drugs such as antidepressants, anti-anxiety

medications and prazosin. Psychotherapy is also an efficient method of treatment

through cognitive therapy, exposure therapy and eye movement desensitization and

reprocessing (EMDR). VR rehabilitation is used commonly for controlled exposure

therapy to treat many patients that have PTSD such as war veterans.

2.4

Related Works

In this thesis, we focused on the treatment of upper and lower limbs through

immersive virtual reality rehabilitation. Our approach was to design a practical and

inexpensive way to treat patients in their homes or medical facilities using orientation of

the limb segments. There are several state of the art studies that have practiced treatment

of various medical conditions that have affected limbs.



Immersive virtual reality mirror therapy is studied among patients that are

suffering from PLP in University of Manchester [44] by using data gloves and

sensors to capture upper and lower limb motion. Patients observed the motion of

their missing limbs in VE with the help of a HMD. According to the conducted tests,

upper limb is treatment seemed to be more effective than lower limb. Patients

reported decreased phantom pain during the sessions. Conclusive results of the

study show that immersive VR therapy can potentially be useful for PLP treatment

although more detailed testing may be required with larger group of patients.

28

Research Field Background



A study about post-stroke VR rehabilitation for upper arm [45] is conducted in

University of Ulster which aimed the encouragement of arm movement in stroke

patients through several tasks that are presented in VE while assessing the effects

of immersive VR on both healthy subjects and subjects with stroke. A custom

system is designed by the research team that uses a HMD and gloves to interact

with VE. After several tests, subjects with stroke were more effected by the

immersion of VR than the healthy subjects.



A data glove with pneumatic features called PneuGlove [46] is designed to guide

the stroke patient to grasp an object that is created in the VE through immersive

VR rehabilitation. Closing and opening of hand is tested on stroke patients with

hand impairments in VR simulations. Subject group that was using the data glove

showed slightly more improvement than the group which was not using any input

device.



Research about the treatment of the problems related to hand movement which is

commonly observed in Parkinson’s disease and the elderly attended with the use

of immersive VR [47]. Subjects practiced finger tapping test in virtual environment

using a HMD while seeing their hand in the simulation. Analysis of tapping

movement is obtained through the designed equipment from the test subjects with

issues and the young subjects that have no troubles of hand movement. Results of

the tests show that practice of finger tapping test can also produce efficient results

using VR application. Therefore, VR approach can be used to study movement

alterations and control the hand movements for further research analysis on the

subject.