Investigation of Transitory Sensory Feedback for Upper Limb Prostheses

(1)

Università di Pisa

Dipartimento di Ingegneria dell'Informazione

corso di laurea magistrale in Bionics Engineering

Investigation of Transitory Sensory

Feedback for Upper Limb Prostheses

Relatori:

Prof. Christian Cipriani Dr. Leonardo Cappello Controrelatore:

Prof. Enzo Pasquale Scilingo

Candidato:

Waleed Mustafa Ali Alghilan A.A. 2019-2020

(2)

1

Abstract

Myoelectric prostheses are on the pinnacle of their technological development in their complexity and movement patterns. Yet, controlling them imposes a great difficulty, impeding their spread between users. This is accredited to the absence of sensory feedback, where the only feedback of how much force the robotic hand is applying to an object is through vision or sound. Recent advancements in targeted muscle reinnervation surgery, where nerves that served the missing limb are transferred to the residual limb, allows for promising advancements in sensory feedback quality. Sensory transfer occurs when the nerve fibers that are transferred to denervated muscle segments of the residual limb regenerate afferent nerves through the muscle and reinnervate nearby denervated skin. Patients reported sensing light touch, graded pressure, different vibration frequencies, hot and cold stimuli as if they were applied to their missing limb. This phenomenon can be exploited to provide sensory feedback to patients such that when they grip something with a prosthesis, a small feedback device pushes into their reinnervated skin. So, they feel as if they are grasping the object directly.

In this thesis we study the strategies and actuation principles used to deliver non-invasive mechanical sensory feedback. We attempt to find an optimal solution to deliver a modality-matched sensory-feedback to TMR patients. The review of currently available solutions nominated voice-coil based actuation as the best solution to deliver sensory feedback, with its main disadvantages of power consumption and heat generation. We tried to overcome these disadvantages using a new feedback policy where we limit the continuous analogous feedback to the first phase of the grasp, where the grasp force changes frequently, and fade out the feedback as it arrives to a stable state. To further enhance this strategy, we attempted to couple it with neural adaptation mechanisms, this way the fading out will not be noticeable by the user. We could not find a method to remove this feedback without the participants noticing.

To study if the proposed strategy of sensory fade out (Transitory feedback) is effective, we run a study in virtual reality where participants grasp and hold a fragile virtual box for a specified time. The transitory feedback showed to be equivalent in performance to continuously delivered feedback and better than the no-feedback condition with respect to broken boxes and successful trials, and participant subjectively received it as good as continuous feedback. Finally, a small involuntary reflex was detected within 500 msec of the sensory feedback removal.

This thesis is the first step for future work surrounding the design and integration of tactors in prosthetic systems, and contributes to the existing body of knowledge regarding sensory feedback, with future plans to translate this devices to a clinical setting, which has the potential to improve the quality of life for upper limb amputees.

(3)

2

Acknowledgements

I want to thank my supervisors Prof. Christian Cipriani and Dr. Leonardo Cappello for their help and the positive and encouraging environment, they were my mentors and advisors and their guidance was invaluable in realizing this work.

This work has been completed with extensive advice and guidance from an interdisciplinary team of researchers in the Artificial Hands Area of the Biorobotics institute, Pontedera. I would like to thank all the researchers and students in the lab for their encouragement and enthusiasm and their practical advices and guidance throughout my time in the lab.

I also, want to thank the team of researchers in PERCRO lab, Pisa. Lead by prof. Antonio Frisoli for their support and hospitality throughout my time in the lab. They have continuously provided advice and feedback and played a large role in making this thesis.

I Also want to thank all the faculty members and professors from the school of engineering in the University of Pisa, and Scuola Superiore Sant'anna for their help and guidance throughout my degree and giving me this unique opportunity to experience advanced research firsthand. Also, I want to thank all my colleagues and friends in the Masters of Bionics Engineering program for their support, company and fun times.

Thank you to my friends who I have met over the years, I could not mention everyone here, but their friendship is something I truly treasure.

Finally, I would like to thank my family for their love and support. My father Mustafa Alghilan and my mother Reem Alawi have always encouraged me to pursue further education and follow my own path; I truly appreciate all they have done in raising and educating me, and I want to thank them for believing in me and encouraging me to go forward when I was in doubt, and supporting me throughout my journey both here in Italy and back in Saudi Arabia.

(4)

3

Chpater 1: Introduction

The human hand is of critical importance in everyday life. Losing an arm or two is extremely limiting, as most trivial tasks become harder and time-consuming. Patients with upper limb amputations rely on prosthetic devices to overcome these limitations. These devices traditionally had body-powered function-oriented designs, the more advanced and newer designs have life-like battery-powered designs. Despite these technological advances, Body-powered devices are still popular among patients [1]. This inclination can be accredited to their robustness, low costs, and the sensations of contact forces through the actuation cables. This “feedback” sensation allows for a more robust manipulation of fragile objects, detecting slippage, omitting the need to maintain visual contact with the held object. This is a crucial property missing in current battery-powered advanced prostheses.

To provide the sensory feedback, researchers studied different haptic feedback policies (i.e., DESC: Discrete-Event based Sensory Control [2], matched modality feedback [3], and vibration based feedback [4]). All the proposed solutions provide improvements in the manipulation task, but they still suffer from some limitations, mainly their coarse sensory information, and their relatively high power-consumption in the case of continuous sensory-feedback.

1.1 Problem definition

Develop a method to deliver non-invasive sensory feedback for upper limb amputees, while keeping power-consumption and size factors to a minimum.

1.2 Objectives

• Limit the power consumption and energy losses of the system. • Make the system as discrete and lightweight as possible. • Run initial tests on the feasibility of the proposed solution/s.

1.3 Chapter summary

In chapter 2, neural sensory mechanisms in humans are discussed, how the mechanisms of human-machine interfaces work with examples of applications, and finally the non-invasive sensory feedback policies and devices existing in literature are reviewed.

In chapter 3, the objectives and design specifications are defined, and we explore current solutions to the non-invasive sensory feedback mechanisms and compare them against a set of specifications the Quality Function Deployment Method.

In chapter 4, the Experimental studies to check the feasibility of the suggested feedback policy are listed with their results.

(6)

5 In chapter 6, the references are listed. In Appendix A and B where the programs and hardware used to carry the experiments are detailed.

(7)

6

Chpater 2: Theory and background

2.1 Sensory mechanisms in the CNS

It is well established in the literature that human movement utilizes a closed-loop motor-control strategy. This strategy manifests as an interplay between the motor output (Muscle movements) and sensory inputs (Neural signals from peripheral nerves). The lack of sensory feedback in the case of myoelectric prosthesis significantly alters this control strategy, which in turn increases the cognitive load on the user and makes the myoelectric hand unpleasant to use. The ideal sensory-feedback system would provide both modality-matching and somatotopic-matching, allowing the user to feel a relevant stimulus in the correct location on their missing limb.

The nerves in the human peripheral nervous system (PNS) fall into two main categories: afferent nerves and efferent nerves. The former sense the stimulus and sends this information to the central nervous system (CNS), the latter delivers the commands from the CNS to the distal organs. The afferent nerve endings can be further divided based on the stimulus that activates them:

1. Mechanoreceptors for mechanical stimuli. 2. Thermoreceptors for thermal stimuli. 3. Nociceptors for pain.

4. Electromagnetic receptors for vision. 5. Chemoreceptors for taste and smell.

Each type of these receptors is selectively sensitive to a single type of stimulus while being almost nonresponsive to other stimuli. [5]

2.1.1 Somatic senses

Somatic senses are the nervous mechanisms that collect sensory information from all over the body. These include mechanoreceptors, thermoreceptors, and Nociceptors. For tactile sensations, our receptors of interest are the mechanoreceptors on the skin.

Touch sensation results from stimulation of tactile receptors under the skin and in the tissues immediately beneath, pressure sensation generally results from deformation of deeper tissues, and vibration sensation results from rapidly repetitive sensory signals. Tactile receptors include:

1. Free nerve endings that are found everywhere in the skin and many other tissues, these can detect touch and pressure.

2. Meissner’s corpuscles found in abundance in the fingertips, lips, and other areas of glabrous skin. Their primary function is to discern spatial locations of touch and contact with the skin they also signal the movement of objects on the skin surface.

3. Merkel’s discs are responsible for giving steady-state signals that allow one to recognize continuous contact of objects with the skin.

(8)

7 4. Ruffini’s end-organs, located in the deeper layers of the skin and deeper internal tissues. They

signal continuous states of deformation of the tissues, such as prolonged heavy pressure. 5. Pacinian corpuscles: They are stimulated only by rapid local compression of the tissues; they

signal tissue vibration or other rapid changes in the mechanical state of the tissues.

6. The hair on skin: slight movement of any hair on the body stimulates a nerve fiber entwining its base. Thus, each hair and its basal nerve fiber, called the hair end-organ, are also a touch receptor.

Figure 1 organisation of neural receptors in the skin. adapted from[6]

2.1.2 Adaptation in neural receptors

Adaptation is the main reason why some neural receptors are sensitive to prolonged pressure stimuli while others respond to rapid stimuli. In general, the application of force on one side of the receptor deforms the viscoelastic structure and transmits a signal to the adjacent central nerve fiber, eliciting a receptor potential. The first mechanism of adaptation is when the fluid within the corpuscle redistributes so that the receptor potential degrades. Thus, the potential appears on the onset of compression but disappears within hundreds of milliseconds, even with continuous compression. The second mechanism of adaptation is a much slower one which occurs when the nerve fiber itself gradually becomes accommodated to the stimulus; this is due to progressive inactivation of the sodium channels in the nerve fiber membrane. Some receptors are slow-adapting like Ruffini end-organs and Merkel discs, while others are much faster like Pacinian corpuscles.

2.2 Human-machine interfaces in prosthesis

An optimal interface to control the prosthesis should decipher the user’s intended movements, command the prosthetic arm, and facilitates the closed-loop control by returning relevant sensory signals to the user. To decipher the user’s intended movements, an interface connects the human with the prosthesis. The different kinds of interfaces include:

2.2.1 Brain interfaces

By using arrays of electrodes that penetrate the cortex, researchers were able to record control signals and operate devices in animal models and primates, including monkeys that were able to control 3 DOF robotic arms [7] [8], [9]. Some studies were performed on humans, where cortical

(9)

8 arrays were implanted into severely disabled patients. The patients were able to control the cursor on a computer screen, and even control robotic arms [10], [11]. The development of systems that provide sensation through brain-machine interfaces is still in its infancy. Some researchers suggested supplying sensory data through the somatosensory cortex [12], but the technological development in this area is still in its beginnings.

2.2.2 Peripheral nerve interfaces

Peripheral nerves carry the centrally processed and integrated motor commands to the organ’s muscles. These signals are generally weak and affected by surrounding EMG signal noise at a similar bandwidth. Also, keeping the microelectrodes stable in the correct position for long periods is a challenging task. Over time, the implants generally fail. These systems are also fragile and frequently require surgical replacement [13]. [14] showed that electrical stimulation through the implanted electrodes elicited discrete, unitary, graded sensations of touch, joint movement, and position, referring to the missing limb. Although research on direct peripheral nerve interfacing has been ongoing for more than 50 years, it has not yet been clinically implemented [15].

2.2.3 Electromyographic interfaces

The most common control signal used nowadays is the EMG signals from residual muscles. Despite its non-invasiveness, poor control and discomfort outweigh the benefits of this technique [16]. Many unilateral upper limb amputees choose not to use a prosthesis at all and most bilateral arm amputees, forced to use a prosthesis to restore any function at all, use body-powered devices [15]. Unfortunately, there are also technical difficulties in acquiring reliable EMG signals to control these prostheses. The signals are prone to many noise sources, including neighboring muscle activity, unreliable electrode positioning, sweating, and distortion due to the signal traveling through different subcutaneous tissues [17]. These noise sources can result in unintentional control signals sent to the prosthesis, causing its activation (opening or closing) even in the absence of voluntary muscle contractions [18].

2.2.4 Targeted Muscle Reinnervation

TMR is a technique to acquire the PNS signals by exploiting muscles as buffers and amplifiers. During TMR surgery, Targeted muscles are denervated, and residual arm nerves are transferred to the target muscles. Usually, target muscles are the residual limb muscles in trans humeral amputees or the thorax of amputees with shoulder disarticulation, and the residual nerves are coopted to the remaining motor points in the targeted muscle. Once reinnervation of the targeted muscles is complete, the muscles contract in response to activation of the transferred nerves. For example, when an amputee attempts to bend his or her missing elbow, the command signal travels down the musculocutaneous nerve to the new target muscle (e.g., a segment of pectoralis major) instead of the

(10)

9 biceps brachii. This muscle segment contracts and generates an EMG signal that can be used to flex the elbow of the prosthetic arm. Four major brachial plexus nerves innervate the arm and hand, hence TMR can produce at least four independent control signals in shoulder disarticulation patients and up to five independent signals in trans humeral amputees (including those from remaining intact nerve muscle connections). [15]

2.2.4.1 Transfer sensation phenomenon

The nerves of the brachial plexus have a mix of motor and sensory fibers. When these nerves are transferred to denervated muscle segments, the afferent nerve fibers could regenerate through the muscle and reinnervate nearby denervated skin[19]. Patients reported sensing light touch, graded pressure, different vibration frequencies, hot and cold stimuli as if they were applied to their missing hand [19], [20].

Targeted reinnervation takes nerves that once served the hand, a skin region of high functional importance, and redirects them to less functionally relevant skin, which questions its efficacy as an interface to receive the sensory feedback. In [21], they examined grating orientation and point localization thresholds on two amputees who had undergone a TMR procedure and reported their ability to orient grating stimuli at threshold levels similar to the contralateral chest skin. Despite the coarse tactile spatial acuity of the reinnervated skin compared to the fingertips, it is still feasible to use the reinnervated skin as a pathway to return sensory feedback from a prosthetic device to the amputee.

2.3 Non-invasive Sensory feedback policies

Sensory feedback policies are broken into two main themes, sensory substitution based, and modality matched sensory feedback. The former is comprised of systems that apply a feedback signal not matched in modality to the stimulus occurring at the end-effector, and the latter is congruent to the external stimulus detected by the prosthetic sensor (i.e., pressure stimulus elicits pressure sensation). In this thesis, we are focusing on the matched modality feedback as it brings us a step closer to the ideal feedback (feedback with both modality matching and somatotopic matching).

Mechanotactile feedback is used to communicate conditions of touch and grasp occurring at the prosthetic prehensor to the user. Most of these systems translate touch or grasp force information into a pressure applied to a strategic location on the amputee’s residual limb or body. This way, somatotopic matching is achieved in the case of TMR patients[4].

Numerous tactile actuator (tactor) designs are present in literature, some are utilizing pneumatics [22], others are based on servomotors [23], and voice coils [24]. Studies on amputee showed that the incorporation of mechanotactile feedback improves the performance during object manipulation tasks[25].

(11)

10 Compared with other feedback systems, mechanotactile systems typically consume more power and are often larger and heavier than vibrotactile or electrotactile devices. As suggested by [3], a sensory feedback tactor should become comparable in size to an electrode, and they suggest limiting its functionality to contact and pressure only, based on patient prioritization and preference.

2.3.1 Discrete-Event based Sensory feedback Control (DESC)

Based on a neuroscientific hypothesis [26] that the mechanisms involved in humans' sensorimotor control are organized by means of multi-modally encoded discrete sensory events. [2] proposed vibratory sensory feedback organized in phases (Object contact, liftoff, Replace and release), then they tested their methodology using a Modified Box and Blocks test called Virtual Egg Test (VET).

Figure 2 Phases and of a pick-lift-replace manipulating task. Vertical lines indicate the mechanical events that separate consecutive grasp phases. Adapted from [2]

The VET replicated the box and blocks test except that breakable blocks were used instead of the standard wooden ones. The performance was measured by the number of blocks transferred and percentage of blocks broken during 1-min trials, similar to the standard box and blocks test.

Upon testing this strategy with upper limb amputees, a statistically significant decrease in the fraction of broken blocks was found, implying improved sensorimotor control. Also, performance speed was not slowed down by the feedback, suggesting that providing DESC feedback minimally taxes participants’ cognitive system.

2.3.2 Continuous modality-matched sensory feedback

[26] carried an experiment in a virtual environment where they used EMG signals to allow the patient to control a virtual prosthetic hand holding a virtual object. The EMG signals were used as velocity commands for the motion of virtual prosthesis. The participants could see the virtual prosthesis and the virtual objects and feel the pressure and shear haptic feedback.

Their results showed improved performance in controlling the grip force with the haptic feedback compared to no feedback condition. However, the simultaneous display of two haptic channels (pressure and shear) does not enhance but instead degrades, the grip force control.

(12)

11

Chpater 3: Design specifications

To limit the design to relevant factors for the user, we employed a modified version of Quality Function Deployment (QFD) method to prioritize and create a device that satisfies the user needs while meeting the technical specifications required. Our scope is sensory feedback in TMR patients in chest area, but the design outcomes can be generalized to other amputation conditions.

The general requirements are:

• Modular design that can be attached to an existing prosthesis. • Non-invasive.

• Silent operation. • Light weight.

• A thin and discrete profile (<2 cm). The technical requirements are:

• Footprint area less than 60 cm2_.

As reported in [27], the sensation area in TMR patients are in the range (15 cm x 20 cm), limiting the device to 60 cm2_{allows several devices to be implemented simultaneously.}

• Applying forces in range (0.002 N to 5 N), and a stroke up to 1 cm.

As reported in [3], a targeted reinnervation participant begins to feel discomfort at 5.8 N and pain at 8.8 N, corresponding to 10 mm penetration with an 8 mm diameter tactor head with minimum pressure limit of 0.003 Newtons.

• Frequencies (70 Hz – 110 Hz).

As reported in [28], vibrations of 30 Hz, 250 Hz and 400 Hz produced sensations in the missing limb.

• Maximum temperature (36 C – warm), preferable set a t 34 C.

As reported in [29], the average limit of warmth sensation in chest is between 34-37. • Response time less than 100 milliseconds.

As reported in [30],transcortical loop delay is approximated to 100 msec.

3.1 Proposed solutions:

Several devices to provide non-invasive sensory feedback are present in the literature concerned with virtual reality, remote surgery, and prosthetics. In this review we will focus on their functioning principles and exclude non-mechanical methods (i.e., electrodes), the possible solutions are:

1. The Rice Haptic Rocker [31]: It conveys proprioceptive information with skin stretch, it has a simple design consisting of a curved contact point (rocker), and servo motor strapped to the upper arm. As the servo rotates the rocker, the skin is stretched and indented.

(13)

12 2. HapPro [32]: A wearable haptic device that conveys proprioceptive information through a

moving wheel on the inside of the forearm.

3. hRing [33]: A wearable haptic device that consists of two mini servo motors, a vibrotactile motor, and a belt. The servo motors are mounted on the finger, and they hold the belt in contact with the inside of the finger. When the two motors spin in opposite directions, the belt presses into the user’s finger. When the motors spin in the same direction, the belt applies a shear force to the skin.

4. pinion/rack [34]: These devices use a rack and pinion mechanism actuated with an electric motor, to apply a normal force on the skin to convey pressure sensation.

5. skin stretching [35]: This device stretches the skin by twisting it using two rotating wheels in contact with the skin.

6. Silicone bladders [36]: They used a closed pneumatic system of silicone encapsulated bulbs, one at each fingertip and they are connected to a corresponding position in the cuff.

7. Off-center motor vibration motors are commonly used for vibratory feedback (i.e., [37]). 8. voice-coil-based actuators [38]: Electrical current running through a coil of wire creates a

magnetic field that reacts with a permanent magnetic field generated by the stator causing push/pull movement.

9. arrays of vibrotactile motors [39], [40], [41]: used to signal the intensity of stimuli or current configuration of the prosthetic hand.

10. Finger haptic thimbles: Some researchers made wearable haptic interfaces on the fingertip. The device is composed of thimbles to wear on the fingers with movable plate actuated by a compact servo motor for orientation and a custom-made voice coil to push the plate into the anterior side of the fingertip [42], or using mini servo motors and a parallel robot architecture to move the plate [43] [44]. These devices can provide information regarding surface stiffness and contact no contact conditions in virtual reality environments.

Table 1 compares between the fore mentioned working principles to filter out unfeasible solutions:

Device working principle Applicable? Reasons fixable? Modifications

Skin deformation using rocker No Rocker is too bulky Yes Use bended stick instead

Wheel rolling Yes

Belt tightening/moving Yes

Pinion/rack No The rack is vertical Yes Use horizontal rack.

Stretching skin Yes

Pneumatic silicon bladders Yes

Vibration motors Yes

Voice coil actuators No Heat dissipation Yes Shorten actuation time/use less force

(14)

13 At this point, we compare between all the proposed solutions using QFD method. Starting by defining technical characteristics and user demands to create a comparison criteria then comparing between the different solutions performance using these criteria to finally arrive at most adequate solution for our objectives: ▲ ▲ ▲ ▼ ▼ ▼ ▼ Wei gh t Re lat iv e w eight (100) Quality Characteristics User Needs n u m b er o f mod e s Ban d w id th n u m b er o f mot o rs Wei gh t Re liab ili ty vo lu m e p o w er con su m p tio n 5 15.6 integrable in cuff ▲ Θ 3 9.37 easy to clean Θ 4 12.5 light weight Θ Ο 5 15.6 minimal irritation Ο 5 15.6 analogous sensation Θ Θ ▲ 2 6.3 silent operation Θ ▲ 5 15.6 discrete (thin) ▲ Θ Θ

3 9.4 Long battery life Θ ▲ Θ

Weighted importance 156.3 140.6 365.6 128.1 31.3 318.8 131.3

Weighted importance scaled (out of 100) 12.3 11.1 28.7 10.1 2.5 25.1 10.3

Weighted importance scaled (out of 5) 0.56 0.50 1.31 0.46 0.55 1.14 0.47

Table 2 House of Quality table comparing the technical characteristics with user needs.

Where:

▲ positively correlated ( = 6 points ) Θ weak positive correlation ( = 3 points ) Ο strong positive correlation ( = 1 point ) And each for each criterion we have:

𝑊𝑒𝑖𝑔ℎ𝑡𝑒𝑑 𝑖𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒 = ∑ 𝑝𝑜𝑖𝑛𝑡𝑠 ∗ 𝑤𝑒𝑖𝑔ℎ𝑡𝑟𝑒𝑙𝑎𝑡𝑖𝑣𝑒

Now that we have weights for each criterion, we will start comparing the different solutions on how they achieve a balance between the Quality characteristics and user needs, to find the optimal potential solution from both the technical viewpoint and customer satisfaction:

(15)

14 User Needs Feedback principle in te grab le in cuf f easy to c lean lig h t we ig h t m in im al fri ctio n / irri tat io n /sw eat in g accur ate an alo go u s se n sa ti o n sil ent o p erati o n d iscrete ( thi n ) lo n g batte ry lif e Weight 5.0 3.0 4.0 5.0 5.0 2.0 5.0 3.0 Score

Skin deformation bended stick 6 6 6 6 6 3 3 3 4.22

wheel rolling 6 6 6 3 3 3 3 3 3.44

belt tightening/moving 6 6 6 3 3 3 6 3 3.83

pinion/rack (Bowden) 6 6 3 6 6 3 3 6 4.14

stretching skin 6 6 6 3 1 3 6 3 3.57

fluid bladders at fingertips 6 3 6 3 6 6 6 6 4.38

vibration motors cuff 6 6 6 6 3 3 6 3 4.22

Voice coils 6 6 6 3 6 6 6 3 4.38

Table 3 performance of different solutions in achieving user needs.

Technical Specifications Working principle n u m b er o f m o d es Ban d width n u m b er o f m o to rs W eig h t Reliab ility vo lu m e p o we r co n sum p tio n weights 0.56 0.50 1.31 0.46 0.55 1.14 0.47 score

Skin deformation bended stick 1 2 2 3 1 3 2 3.50

wheel rolling 1 1 2 2 2 1 2 2.60

belt tightening/moving 2 1 1 2 2 3 1 2.95

pinion/rack (Bowden) 1 2 2 2 3 2 3 3.49

stretching skin 1 1 1 3 3 3 1 3.10

fluid bladders at fingertips 1 2 3 2 1 3 3 3.94

vibration motors cuff 1 3 2 3 2 2 2 3.47

Voice coil actuators 2 3 2 3 3 3 1 4.06

(16)

15 All scores were scaled between 0 – 5 and plotted accordingly. The most adequate solutions (highest user satisfaction and Technical adequacy) will show on the top right corner.

Figure 3 comparing user needs with technical specifications.

In figure 3, the fluid bladders on fingertips and voice coil actuators are the best solutions. We are excluding the fluid bladder due to reliability issues (i.e.: leakage, water vapor formation). This leaves us with voice coils (magnetic actuators) as an optimal solution to the current problem. The main drawback of voice coils actuators, reported in the table, is their power consumption and excessive heating with prolonged activation. The following chapters will tackle this problem.

3.2 Gradual removal of sensory feedback

The problem of excessive heating could be solved using non-back-drivable mechanisms, but this render the device in a disadvantage compared to other proposed solutions, since adding a traditional mechanism bulks up the device, degrades the overall efficiency, and increases the parts count. Another viable solution would be to minimize the force applied as sensory feedback, but excessive force reduction affects the user ability to distinguish between different levels of pressure. Which leaves us with the solution of minimizing the actuation time, or a combination of slightly reducing the force level and actuation time.

To choose when to start a gradual removal of the sensory feedback, we refer to grasping behavior in humans. It is well established in the literature that the grip force increases on initial contact with the object (preload phase), then there is a parallel increase in grip and load forces (load phase). This increase continues until the start of movement. When the load force overcomes the force of

bended stick wheel rolling belt pinion/rack (Bowden) stretching skin pneumatic bladder vibration motors voice coil 2,5 3 3,5 4 4,5 5 2,5 3 3,5 4 4,5 5 Te ch n icha l re q u ire m en ts s core

(17)

16 gravity, the object is lifted, and a static (hold) phase is reached where the grip force stays relatively constant.

We hypothesize that the continuous sensory feedback of pressure is critically important in the preload and load phases and but gradually removing it once the constant hold is reached will not affect the grasp performance. We further hypothesize that we can take advantage of the sensory adaptation mechanisms to remove this sensory feedback without the user noticing it. This way we can decrease the operating time of the actuators, hence solving the heating problem without the user noticing.

We did not find quantitative reports on the neural adaptation characteristics in humans (i.e., how much time until neural adaptation occurs). It could be due to the difficulty of the task, as it is hard to achieve the neural adaptation if the user is cognitively attentive to the sensory feedback, and it is difficult to mark accurately when that occurs.

However, [45] reported that during a steadily maintained indentation, participants perceived a steadily increasing depth of indentation. To perceive a constant depth of indentation, the probe had to be retracted on average 14% of the original depth over an 18-sec period. They add that the average perceived change for all participants combined, from the beginning to the end of an 18-second plateau, was an increase of 23% for a 1-mm indentation and an increase of 12% for a 2-mm indentation.

Another study[46], reported that when continuously moving of the skin with a constant force, sensory adaptation was found to correlate with the rate of skin movement under this force, and complete adaptation occurred when the skin movement was below approximately 10 Micrometer/sec.

To study the neural adaptation, we carried a pilot study to see if the sensory adaptation can be achieved within a reasonable time window for our objective (i.e., we can remove pressure from skin without the participant noticing within 1 minute). After that, we carry a second study to quantify the effectiveness of removing the pressure in grasp and hold task (Transitory feedback). Both studies are detailed in the next chapters.

(18)

17

Chpater 4: Experimental studies

4.1 Sensory adaptation in human fingertips

We hypothesize that there is a threshold of removal rate where it is possible to trick the brain into thinking the pressure removal is due to the neural adaptation and give the illusion of steadily maintained pressure. We used a threshold hunting algorithm to detect this pressure removal rate. Later, we analyzed the data available to see if there was a common trend between successful trails. We used different pressure drop rates and initial pressures. The results were highly subjective, and we couldn’t generalize direct correlation between sensations and drop rates or initial pressure on fingertips.

Objective:

- Study the characteristics of neural adaptation in fingertip.

- study the feasibility of removing pressure applied on the fingertip without the participant noticing.

Participants

seven right-handed participants (4 males, 3 females, age range 36–22 years) with no known neurological conditions were enrolled in this study, which was performed in accordance with the Declaration of Helsinki. All participants gave written informed consent before participating in the study.

Materials and methods: Apparatus:

1) A 3D platform that utilizes stepper motors to move the end effector in 3D space, (VT-80 linear stages, PI miCos GmbH, Eschbach, Germany).

2) A tri-axial load cell connected to the end effector, (NANO17, ATI Industrial Automation Inc., Apex, NC, USA). 3) A computer running custom LabVIEW™ program to control and read data from both the platform and the load cell.

The LabVIEW™ program utilizes a PID controller where the control signal is error in force on the load cell and the output is movement commands to the platform.

Figure 4 Experimental setup: the PID controlled platform will move vertically pressing the load cell against the finger.

(19)

18

Figure 5 Schematic of the controller and connections between systems.

The program structure used in this study is detailed in Appendix A.

Methods:

The experiment is composed of two parts, performed in succession. In the first part, we study the pressure sensation threshold (I.e., from contact what is the minimum pressure the participant recognizes as pressure). In the second part, we study the pressure removal rate threshold.

Threshold hunting algorithm:

Both parts of the experiment utilize a staircase forced-choice method with variable step sizes based on a threshold hunting algorithm suggested by [47]. The algorithm changes the intensity of the stimuli (The pressure applied in the first part of the experiment, the rate of removal in the second part) according to the following law:

𝑥(𝑛 + 1) = 𝑥(𝑛) − 𝐶 𝑛𝑠ℎ𝑖𝑓𝑡+ 1

[𝑧(𝑛) − Φ] Φ is target percentile (Φ = 0.75)

C is the initially tested threshold x(1).

n number of trials that have occurred since the beginning of the experiment. nshift number of reversals that have occurred

X(n) is the intensity at nth step. x(n+1) is the intensity at (n+1)th step.

Z(n) is a binary quantity, that depends on the response at the nth step: 1 for positive response (I.e. participant notices pressure).

0 for negative responses. (I.e. participant doesn’t notice pressure) Consequently, the stimulus level is decremented with a step size δ = 𝐶_𝑛(1− Φ)

𝑠ℎ𝑖𝑓𝑡+1 in the case of

a positive response or incremented with a step size δ = 𝐶_𝑛 Φ

𝑠ℎ𝑖𝑓𝑡+1 in case of a negative response. Since

the step size is inversely related to the number of reversals, both increments and decrements become smaller while running the experiments and converge to the threshold with reasonable accuracy within 20 runs.

(20)

19

Part 1: pressure threshold hunting

To measure the minimum pressure that elicits sensations of pressure against fingertips from initial contact, we used the following procedure:

1. From low contact force with the finger (0.1-0.15 N/cm2_{), apply noticeable pressure to fingertip}

C = 0.4 N.

2. Ask the participant if they feel it as pressure or not.

3. Update the pressure on fingertip depending on their answer, using the threshold hunting algorithm.

4. Repeat steps 2 and 3 until 10 reversals have occurred. (𝑛_𝑠ℎ𝑖𝑓𝑡 = 10) or (n > 20) 5. x(n+1) is taken as the threshold.

Part 2: Feasibility of pressure removal without eliciting sensations

To measure the threshold of pressure removal we start from a constant pressure 1 N/cm2

held for 5 seconds then the pressure is removed with a fixed rate x(n) while keeping the participant distracted, the participants are asked to push a button if they feel any change in their fingertip. The pressure threshold measured in the first part of the experiment is used as the limit to mark successful trials. A trial is success if the pressure dropped below the limit.

The same threshold hunting algorithm detailed before was used with x(n) being the different pressure drop rates. The distractions used to keep the participants cognitively occupied include solving simple mathematical problems like addition and subtraction, talking about what they do and tell stories.

A record for each trail of forces on the end effector position and its position is kept for later analysis.

Data Analysis:

From the data we extract features of interest that include: 1. Time: time until they felt the pressure drop [seconds].

𝑇 = 𝑁 × 𝑇𝑠𝑎𝑚𝑝𝑙𝑒, where 𝑇𝑠𝑎𝑚𝑝𝑙𝑒 = 0.1 𝑠𝑒𝑐

2. Total displacement: Total displacement of the platform when the participant felt the pressure drop.

∆𝑥 = 𝑥𝑓− 𝑥𝑖

3. Change in force: Amount of total force drop for start to the end of trial. ∆𝐹 = 𝐹𝑓− 𝐹𝑖

4. Displacement percentage: Total displacement of the platform expressed as a percentage of the original indentation.

∆𝑥𝑝𝑒𝑟 =

𝑥𝑓− 𝑥𝑖

𝑥𝑖

(21)

20 5. Force Percentage: Force difference when the participant noticed the pressure drop expressed

as percentage of originally applied force. ∆𝐹𝑝𝑒𝑟 =

𝐹𝑓− 𝐹𝑖

𝐹𝑖

× 100%

6. Platform velocity: Average velocity of platform during the last second before the participant felt the pressure drop.

𝑋𝑣𝑒𝑙= 𝑎𝑣𝑔(𝑥̇𝑁, 𝑥̇𝑁−1… . 𝑥̇𝑁−9)

7. Platform acceleration: Average acceleration of platform during the last second before the participant felt the pressure drop.

𝑋𝑎𝑐𝑐= 𝑎𝑣𝑔(𝑥̈𝑁, 𝑥̈𝑁−1… . 𝑥̈𝑁−9)

8. Force removal rate: Average force removal rate in the last second before the participant felt the pressure drop.

𝐹𝑣𝑒𝑙= 𝑎𝑣𝑔(𝐹̇𝑁, 𝐹̇𝑁−1… . 𝐹̇𝑁−9)

9. Force removal acceleration: average Acceleration of force removal in the last second before the participant felt the pressure drop.

𝐹𝑎𝑐𝑐= 𝑎𝑣𝑔(𝑥̈𝑁, 𝑥̈𝑁−1… . 𝑥̈𝑁−9)

The data is divided and flagged into two main categories, Failed trials are the trials where the participant stopped the trial before the pressure reached the threshold level (I.e., they noticed the pressure drop). While successful trials are trials where the pressure reached the threshold level, a trial starts from the moment the pressure release starts and ends when the participant pressed the button in failed trails or when the pressure dropped below the threshold in successful trails.

Results:

The sensitivity threshold of pressure on fingertip is found to be ( 𝜇 = 0.34 𝑁 , 𝑆𝐷 = 0.08 𝑁 ), while the removal rate of pressure is ( 𝜇 = 0.24 𝑁/𝑠𝑒𝑐 , 𝑆𝐷 = 0.28 𝑁/sec ).

31 out of 120 trials were successful, with high pressure drop rates (3 trials) marked as success. Successful trials have higher velocities toward the end of the trail on average. As seen in figure 6.

By grouping the data into bins based on their respective force drop rates x(n), figure 7

Figure 6 comparing the velocity of the platform during the last second before the participants noticing pressure drop in failed trials.

(22)

21 shows the duration of trials, or the participant

pressed the button for failed trials. (𝑀𝑒𝑑𝑖𝑎𝑛 = 5.2 𝑠𝑒𝑐, 𝜇 = 14 𝑠𝑒𝑐, 𝑆𝐷 = 17.7 𝑠𝑒𝑐 ) .

Finally, for a better examination of the data we group the data for each participant. In figure 8, The force-drop rates in the last second of trials is plotted. And in figure 9, we show the amount of force dropped by the end of the trails. The difference in pressure dropped for successful trials is because the threshold is defined for each participant individually.

Figure 7 Time from pressure removal to the end of trial.

Figure 8 force removal rates during the last second of the trial.

Figure 9 Total amount of force removed when the trial was concluded.

(23)

22

Discussion:

While the pressure threshold was found to be consistent ( 𝜇 = 0.34 𝑁 , 𝑆𝐷 = 0.08 𝑁 ), sensitivity thresholds for pressure removal were highly subjective making it impossible to generalize any findings from the collected data.

Some of the successful trials can be credited to the delay between the user feeling the pressure drop and pressing the button, this can be seen by the higher velocity of platform on average on successful trails as demonstrated in Figure 6.

Figure 7 suggests that pressure removal is sensed almost immediately but could be felt later up to 1 minute of removal for extremely slow rates. So, time is not a deciding factor, when it comes to pressure removal.

The subjectivity of the pressure drop feeling is seen when comparing attributes across participants. in figure 8, we see that insensitive participants arrive to extremely high force removal rates without noticing, yet sensitive participants can detect even slow force removal rates. In addition, there is no noticeable difference between successful and failed trials when it comes to force drop rate for the same participant. Suggesting no relation between the force drop rate and trial success for the same participant.

The high dispersity of total force dropped in failed trials suggests no relation between the amount of force removed (except forces less than 0.1 N) and the feeling of pressure drop. Again, the distribution of forces is different from one participant to the next.

We couldn’t find a correlation between any of the attributes studied to target for further analysis in the future. This absence of correlation could be due to the design of the experiment, or that the pressure drop sensation is a function of a combination of factors other than the ones studied here. This question is left for further analysis.

Conclusion:

To our knowledge no published literature attempted to quantify the neural adaptation mechanism in human cutaneous tissue, in this study we tried to evoke this problem and explore the sensibility of human touch to pressure removal. The high variability among participants, and within subject trials, brought the experiment to an early stop. The feeling of pressure-drop appears to be subjective and varies from one participant to the other and even within the participant trials. Hence, we couldn’t find a method to exploit the neural adaptation mechanisms to discretely remove sensory feedback.

(24)

23

4.2 Effects of sensory feedback removal during holding tasks

As we couldn’t take advantage of neural adaptation, we continued to study the effects of sensory removal on user performance during steady hold. In collaboration with researchers in the Perceptual Robotics Laboratory (PERCRO) in Pisa, we carried the following experiment. To study the feasibility of removing sensory feedback in holding task, we used virtual reality environment and on-the-finger actuators to deliver sensory feedback using different feedback policies including the proposed policy in this thesis, transitory feedback. The participants were asked to grasp and hold a fragile virtual box shown on a screen, we compared their performance between four feedback modalities (No Feedback, Transitory Feedback, Discrete Event-driven Sensory feedback Control (DESC), Continuous feedback). Later, we studied different performance parameters to compare between the four conditions. The results show no statistically significant difference between the Continuous feedback and Transitory feedback with regards to the number of broken boxes.

Objectives

- Study the effects of removing the sensory feedback during grasp and hold task.

- Compare between the performance of different sensory feedback policies in grasp and hold task.

Participants

Twenty-four right-handed participants (15 males, 9 females, age range 36–22 years) with no known neurological conditions were enrolled in this study, performed in accordance with the Declaration of Helsinki. All participants gave written informed consent before participating in the study.

Apparatus

1) Haptic thimble device

Custom fingertip mounted devices, adapted from [43], were fixed on the last phalanx of the thumb and index fingers of the participant and they were used to deliver the sensory feedback to the user through the fingertip contact using costume-built voice coil actuator. The device compactness and lightweight allow natural motion during virtual object manipulation.

2) Optical tracking system and virtual environment

An optical tracking system was used (OptiTrack V120 Trio, Motive, USA) to track the position of two reflective markers fixed on the outside of the haptic thimble devices. The virtual scenario and the real-time rendering were developed on XVR framework (eXtreme Virtual Reality [48] ) connected with the optical tracking system. The XVR software also performed physics simulation of the interaction forces generated by gravity, friction, stiffness, and collisions. The physics engine was updated using the position references of the tracked markers at 120 Hz.

(25)

24 3) Virtual environment physical model

The virtual scene is composed of a graphic representation of a virtual cube, a table, and a horizontal target line (150 mm above the table plane in real life). A virtual prismatic guide, graphically represented as a vertical line, constrains all degrees of freedom of the cube except for the vertical translation. Index and thumb fingertips were represented as two spheres with 10 mm diameter, colored in light grey. The point of view was coherent with the body pose of the participant.

For a realistic rendering of the interaction forces and behavior of the box, a virtual spring was used to translate the distance between the two markers to normal and shear forces, where we represent interaction force as a compliant spring (250 N/m) that represents a malleable object. The virtual grip force is the force exerted by the compressed spring. The box is set to break at (1.5 N / 6mm compression) making it a fragile object. To reserve power, and reduce heat produced by the actuators, all forces delivered to the haptic interface are reduced by a factor of 2.

4) Microcontroller:

A microcontroller (Teensy 3.2, Pjrc.com LLC, USA) connected the XVR software with the haptic thimble devices based on the location of the two makers from the optical tracking system and the sensory feedback modality set by the XVR different sensory feedback scenarios can be delivered to the user.

5) Monitor:

A monitor, situated in front of the participant, displayed the computer graphic representation generated by the XVR framework. The participants’ vision was limited

to the monitor, and a physical barrier prevented them from seeing their hand or the thimble haptic actuators during the experiment.

The program is further detailed in appendix B.

Methods:

To assess the effectiveness of the transitory feedback we use a within-subject experiment design where we compare the performance of the participants in a grasp and hold task using 4 different sensory feedback modalities.

I. Visual only (No Feedback): No sensory feedback is delivered to the user.

II. Transitory feedback: The continuous feedback is gradually removed after a specific time window. (the end of transient phase)

III. DESC: Vibrational cues are given upon initial contact, lift off, turn back, and release.

Figure 10 Experimental setup. the user picks the virtual box showed on the screen and holds it. Different sensory feedbacks are delivered to his fingers.

(26)

25 IV. Continuous Feedback: A continuous haptic feedback proportional to how much pressure is

applied to the virtual box.

The transient phase is defined as the window from the initial contact with the virtual box to the 1 second mark after crossing the horizontal line.

There was an accommodation phase before the start of each feedback modality (>1 min). The participant attempted to perform 20 trials using each feedback modality (total 80 executions). To limit the training effects and trends, the order of introduction was unique for each participant.

During the experiment, the participants were asked to sit in front of the monitor and wear the two haptic thimble devices on their index and thumb digits, grasp the virtual box and hold it above the horizontal line shown on the monitor without breaking it. The box color will turn red if broken, and it will turn green upon successful execution of the task (>6 seconds above the line), and will start slipping if the applied grasp was not strong enough.

Each trial could terminate with one of three different results:

I. success: the cube was correctly picked from the starting platform and lifted above the line for the required amount of time;

II. dropped: the cube was lifted, but the contact force decreased, and the cube was dropped. III. broken: the cube was grasped with a peak force exceeding a fixed threshold value

representing the breaking force threshold of the cube.

records of the markers’ position in space, box position in the virtual environment, and the type of sensory feedback used were saved for later analysis. During the experiment, for each feedback the participants were asked to rate their experience using the following questions:

Q1. How much did you rely on the tactile feedback?

Q2. How much did you rely on the visual feedback?

Q3. How realistic was the haptic feedback? Q4. How much were the haptic feedback and visual feedback congruent? Q5. How easy was it to fine manipulate the contact force [approaching grasp]? Q6. Rate you experience in general.

Q7. Rate your confidence level with the tactile feedback? Q8. How easy was it to accomplish the task?

With multiple choice answers where -3 is negative response 0 is neutral and +3 is positive response.

Data collection and Statistical analysis:

The data outcomes are later organized and separated into groups according to the experiment outcome (success, broken during transient phase, broken after the transient phase, and slipped before transient phase and slipped after transient phase). The data of interest are the number of experiments

(27)

26 for each outcome, the distance between the two markers after the transient phase, and finally the box position on screen. for each reported output in the results the average is taken for each participant across all their trials for each feedback modality.

All tests for statistical significance were carried using a non-parametric Friedman test (α = 0.05) followed by a post-hoc test adjusted with Bonferroni correction.

Results:

The percentage of broken boxes when using transitory feedback and continuous feedback is significantly lower than DESC and Only visual feedback (χ2_{(3) = 34.28, P < 0.01). while the percentage}

of broken boxes after the transient phase is only significantly different for the DESC feedback (χ2_{(3) =}

17.86, P < 0.01), as demonstrated in figure 11.

Figure 11 The rate of broken boxes for each participant (blue) and mean across participants (red). (**: P<0.01, *:P <0.05)

There is no significant difference between the feedback modalities in the percentage of boxes slipping and falling neither before (χ2_{(3) = 3.09, P = 0.378) nor after the transitory phase (χ}2_{(3) = 3.64, P =}

0.303). Losing contact and dropping the box is relatively rare in all participants with all the feedback policies, As shown in figure 12.

Figure 12 The rate of slipped boxes for each participant (blue) and mean across participants (red).

(28)

27 The percentage of successful executions using the visual feedback only or DESC is significantly lower than continuous and transitory feedback. (χ2_{(3)=39.26, P < 0.01), as show in figure 13. All the feedback}

statistics are reported in table 5.

the fingers are closing or opening when the feedback is removed. We can see a small dent immediately after the start of pressure removal in the case of transitory feedback. To study if this dent is statistically significant, we calculate the maximum opening during the first half second after the end of the transitory phase and compare it across different feedback modalities as shown in figure 14. We find the opening in transitory sensory feedback is significantly higher in absolute terms χ2_{(3) = 12.55, (P <}

0.01) with significance only between the Transitory feedback and DESC and continuous feedbacks. In relative terms, the maximum aperture in the first half second (M = 0.47 mm, SD = 0.26) is significantly higher in the in Transitory sensory feedback compared to all other feedbacks χ2_{(3) = 30.75, (P < 0.01).}

Figure 14 The maximum distance between the markers immediately after start of sensory removal for each participant (blue) and mean across participants (red).(**: P<0.01, *:P <0.05)

Finally, the answers collected from the participants are shown on figure 16. For each feedback modality. No significant difference is found between the answers for the continuous and transitory feedbacks.

The distance between the two markers on the fingers is plotted in figure 15 for different feedback policies starting from the moment the box reaches above the target horizontal line. The gradual removal of the sensory feedback in case of transitory feedback starts at the 1 second mark. The mean distance between the fingers during the first second above the line is taken as a reference point from which we can compare if

Figure 13 Percentage of successful trials for each participant (blue) and mean across participants (red). **: P < 0.01

(29)

28

Performance metric Feedback policy In transient phase After transient phase

M [%] SD [%] M [%] SD [%]

Boxes Broken Visual only 31.43 16.27 21.66 11.72

DESC 33.37 18.57 25.64 13.23

Continuous feedback 12.92 12.71 15.76 11.21

Transitory feedback 14.73 17.80 13.16 9.86

Boxes Slipped Visual only 7.95 8.19 2.96 4.48

DESC 6.13 6.20 1.29 2.77

Continuous feedback 15.76 11.21 3.13 3.85

Transitory feedback 5.69 5.88 4.43 5.78

Success rate Visual only - - 36.01 18.03

DESC - - 33.56 17.95

Continuous feedback - - 62.51 18.90

Transitory feedback - - 59.06 17.35

(30)

29 Fi g u re 15 R ela ti ve d ist a n ce b et w een th e tw o m a rke rs (th e me a n o f th e f ir st seco n d a b o ve th e li n e is ta ken a s ref er en ce p o in t fo r e a ch p a rti ci p a n ts ) th e g ra d u a l r emo va l o f sen so ry fe ed b a ck st a rts 1 seco n d a fter r ea ch in g th e li n e in ca se o f tr a n si to ry fe ed b a ck (th e en d o f tr a n si to ry p h a se) .

(31)

30 Fi g u re 16 Qu esti o n n a ir e resp o n se m ea n a n d st a n d a rd d ev ia ti o n s a cro ss p a rti ci p a n ts ( *: P < 0.05, * *: P < 0.01)

(32)

31

Discussion:

The transitory feedback showed to be as effective as continuous feedback. The percentage of broken boxes in transient phase shows no significant difference between the continuous and transitory feedback, which is expected since they are the same feedback in the transient phase, the continuation of non-significant difference after the transient phase indicates the effectiveness of the transitory feedback. Again, in figure 13 the indifference in the success rates further emphasizes its effectiveness.

Upon the removal of transitory sensory feedback, the grasp was observed to widen a little. This could be indicating that participants are more likely to lose contact with the box and drop it when using transitory feedback. Figure 12 shows that contact loss is rare across all feedbacks and there is no significant increase in the case of transitory sensory feedback. To study if this widening is relevant and only occurs on the transitory feedback case, we studied the maximum aperture with respect to the mean aperture before the removal of the sensory feedback and report its significant increase form baseline. To our knowledge, this phenomenon has not been reported before in literature and it could be due to a neural reflex at the motor neuron level where it is interpreted as the box broke or yielded to pressure, hence the body tries to limit the grasp force. Another

explanation could be that the PNS interpreted the decrease in sensory feedback as if it was opening the hand and releasing the grasp, the muscles try to realize this interpretation by releasing the grasp before the CNS intervenes and stops it.

In this study the removal of the sensory feedback and limiting it to only the first second will achieve energy savings in the range of 50%.

In contradiction with previous studies [2], the DESC strategy in this study didn’t differ from no sensory feedback. This is because the DESC is effective if the CNS effectively learned the internal model of the task, in this study the virtual environment doesn’t completely represent the physical properties of the task specially the incidental feedback (i.e., inertial properties, deformation of the body.) hence, the predictive forward sensory control couldn’t take over the task and the grasp relied solely on the sensory feedback instead. Hence, it is showing the same performance as no feedback in general.

The questionnaire answers show that participants in general were satisfied the proposed feedback as continuous feedback and found it to be helpful in achieving the task.

Conclusions:

In this study we demonstrated the effectiveness of the transitory sensory feedback using haptic thimbles to perform a grasp and hold task in virtual environment. Transitory sensory feedback is equivalent to continuous feedback and better than no feedback and DESC conditions in the number of successful task executions and number of broken boxes. We also report a reflex upon the start of

(33)

32 removal of sensory feedback. The main advantages of the Transitory sensory feedback are power savings and reduced operational time while providing rich sensory information of the continuous feedback specially in the transitory phase reflected in the general satisfaction of the participants using it. The transitory sensory feedback is open for further studies regarding the definition the end of transitory phase and force removal profile (the sensory removal here was strictly linear), also the cognitive loading compared to other feedback policies.

(34)

33

Chpater 5: conclusions

Myoelectric prostheses are the pinnacle of technological advancement in prosthetics. Yet, they suffer from the limited sensory feedback, impeding their market penetration and popularity between patients with upper limb amputations. In this thesis we study the principles used to deliver non-invasive mechanical sensory feedback and try to find an optimal solution to deliver a modality-matched sensory feedback. Our review of currently available solutions nominated voice-coil based actuation as the best solution to deliver sensory feedback, with its main disadvantages of power consumption and heat generation (Chapter 3). We tried to overcome these disadvantages using a new feedback policy where we limit the continuous analogous feedback to the phase where the grasp force changes frequently and fade out the feedback as it arrives to a stable level. To further enhance this strategy, we attempted to couple it with neural adaptation mechanisms, this way the fading out will not be noticeable by the user. We could not find a threshold of pressure removal rate not noticeable by the users (section 4.1). To study if the sensory fade out (Transitory feedback) is effective, we run a study in virtual reality where participants grasp and hold a fragile virtual box for specific time. The transitory feedback showed to be equivalent in performance with continuous feedback and better than the no-feedback condition with respect to broken boxes and successful trials and participant received as good as continuous feedback based on a questionnaire response. Finally, a small involuntary reflex was detected within 500 msec of the sensory feedback removal. (section 4.2)

In summary, we explored different solutions to deliver sensory feedback and found the best solution to use voice-coil actuators to deliver a matched-modality temporary feedback. We ran an experiment in virtual reality and demonstrated it did not affect the participants performance. This promising result can help in further developing a more discrete, energy saving, and more feasible matched-modality sensory feedback systems in the future.

(35)

34

Chpater 6: References and appendices

6.1 References

[1] T. DILLINGHAM, L. PEZZIN, and E. MACKENZIE, “Limb amputation and limb deficiency: Epidemiology and recent trends in the United States,” Southren Med. jounral, vol. 95, no. 8, pp. 875–884, 2002.

[2] F. Clemente, M. D’Alonzo, M. Controzzi, B. B. Edin, and C. Cipriani, “Non-Invasive, Temporally Discrete Feedback of Object Contact and Release Improves Grasp Control of Closed-Loop Myoelectric Transradial Prostheses,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 24, no. 12, pp. 1314–1322, Dec. 2016.

[3] Keehoon Kim, J. E. Colgate, J. J. Santos-Munne, A. Makhlin, and M. A. Peshkin, “On the Design of Miniature Haptic Devices for Upper Extremity Prosthetics,” IEEE/ASME Trans.

Mechatronics, vol. 15, no. 1, pp. 27–39, Feb. 2010.

[4] C. Antfolk, M. D’Alonzo, B. Rosén, G. Lundborg, F. Sebelius, and C. Cipriani, “Sensory feedback in upper limb prosthetics,” Expert Rev. Med. Devices, vol. 10, no. 1, pp. 45–54, Jan. 2013. [5] A. C. Guyton and J. E. Hall, Textbook of Medical physiology. 2006.

[6] “Medical gallery of Blausen Medical 2014,” WikiJournal Med., vol. 1, no. 2, 2014.

[7] D. M. Taylor, “Direct Cortical Control of 3D Neuroprosthetic Devices,” Science (80-. )., vol. 296, no. 5574, pp. 1829–1832, Jun. 2002.

[8] L. R. Hochberg et al., “Neuronal ensemble control of prosthetic devices by a human with tetraplegia,” Nature, vol. 442, no. 7099, pp. 164–171, Jul. 2006.

[9] M. Velliste, S. Perel, M. C. Spalding, A. S. Whitford, and A. B. Schwartz, “Cortical control of a prosthetic arm for self-feeding,” Nature, vol. 453, no. 7198, pp. 1098–1101, Jun. 2008. [10] A. Kubler et al., “Patients with ALS can use sensorimotor rhythms to operate a

brain-computer interface,” Neurology, vol. 64, no. 10, pp. 1775–1777, May 2005.

[11] L. R. Hochberg et al., “Reach and grasp by people with tetraplegia using a neurally controlled robotic arm,” Nature, vol. 485, no. 7398, pp. 372–375, May 2012.

[12] R. Romo, A. Hernández, A. Zainos, C. D. Brody, and L. Lemus, “Sensing without Touching,”

Neuron, 2004.

[13] X. Navarro, T. B. Krueger, N. Lago, S. Micera, T. Stieglitz, and P. Dario, “A critical review of interfaces with the peripheral nervous system for the control of neuroprostheses and hybrid bionic systems,” Journal of the Peripheral Nervous System. 2005.

[14] G. S. Dhillon and K. W. Horch, “Direct Neural Sensory Feedback and Control of a Prosthetic Arm,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 13, no. 4, pp. 468–472, Dec. 2005.