CONTROL AND HAPTIC FEEDBACK INTERFACES FOR PROSTHETIC USE

(1)

UNIVERSITÁ DIPISA

DOTTORATO DI RICERCA ININGEGNERIA DELL’INFORMAZIONE

C

ONTROL AND HAPTIC FEEDBACK INTERFACES FOR

PROSTHETIC USE

DOCTORALTHESIS

Author

Matteo Rossi

Supervisor

Prof. Antonio Bicchi

Reviewers

Prof. Dario Farina Prof. Levi J. Hargrove

Prof. Katherine J. Kuchenbecker Dr. Claudio Pacchierotti

The Coordinator of the PhD Program

Prof. Marco Luise

Pisa, June 2018 PhD Cycle XXX

(2)

(3)

to Elena, my bright sun to my family, solid ground on which I stand and to me, not as important as the ground and the sun... but I did the actual work for this

(4)

(5)

Acknowledgements

F

IRST and foremost, I would like to thank Professor Antonio Bicchi for his guid-ance throughout the course of my PhD studies. Besides being my supervisor, he has also been a great mentor, encouraging me to constantly improve both as a researcher and as a person. I also owe my most sincere gratitude to Professor Levi Hargrove, who gave me the opportunity to work with him at the Shirley Ryan Ability Lab. It was an honor for me to spend six months under his wise guidance.

I would like to express my gratitude to Alperen, Andrea, Lucia, Sara, Zhuoqi, and the other fellow labmates at the Advanced Robotics Department of the Italian Institute of Technology; I will not forget our stimulating discussions, the time spent working side by side and all of the fun we have had. I also want to thank my friends and coauthors Edoardo, Alessandro, Cristina, Cosimo, Matteo, Sasha, Manuel, Arash and Giorgio.

Needless to say, this whole work would not have been possible without the help of current and former members of my Institute and hosting organizations, and without the extensive support offered by Fabio, Alberto, Alessandro and all of the wonderful people working at QB Robotics. I would also like to thank the reviewers for their careful reading of the manuscript; their comments helped me improve the presentation in terms of both clarity and scientific rigor.

Finally, a very special thank you goes to Roza, Clint, and all of the people that volunteered as subjects in my research. Their great enthusiasm and generosity have been an incredible source of inspiration.

(6)

(7)

Summary

T

HEhand is a very complex organ that possesses an incredible versatility. Besides its grasping and manipulation functions, the hand can be considered a sensory organ and it also plays a role in social interactions. The loss of a hand is there-fore a major traumatic event that can have great impact in the life of an individual. Prostheses have been developed to help persons with amputation to restore the lost functionality. In particular, body powered and cosmetic prostheses have been used for centuries and are still commonly prescribed today. More recently, a new generation of prostheses has seen the incorporation of electrically-powered actuators controlled via electromyographic (EMG) signals from the muscles in the residual limb. Despite the advances made in the design of these dexterous anthropomorphic hand prostheses, roughly one third of these prescribed prostheses are completely rejected by their users. The factors that lead to the abandonment of these technologically advanced myoelec-tric prostheses are mainly two: the difficulty in control and the lack of adequate sensory feedback. This thesis aims to provide a contribution in this context, both for the con-trol of powered prostheses and for the restitution of haptic feedback. Taking a holistic approach to deal with the design possibilities concerning the next generation of pros-thetic interfaces, three main topics are investigated. First, an analysis on hand stiffness modulation strategies is presented; the relationship of finger stiffness with the EMG activity of the forearm’s muscles is investigated and the possible implications for pros-thetic design are explored. Second, a device for proprioceptive feedback in upper-limb prosthetics is presented. The haptic feedback device, which uses a rolling contact to convey information on the opening of an artificial hand, was developed and experimen-tally evaluated as part of this thesis. Finally, two different paradigms for the control of upper-limb prostheses are explored: (i) the use of a cable-driven interface, in which the interface was used for the control of a robotic hand prosthesis and was tested on person with amputation; (ii) the use of algorithms for the simultaneous and proportional EMG control of multiple joints, presenting experimental results from tests with able-bodied subjects and people with amputation.

(8)

(9)

List of publications

International Journals

1. Piazza, C., Catalano, M.G., Rossi, M., Grioli, G. Bianchi, M., Zhao, K., Bicchi, A. (2017, December). The SoftHand Pro-H: a Hybrid Body-Controlled, Electrically-Powered Hand Prosthesis for Daily Living and Working. In IEEE Robotics & Automation Magazine24(4), 87-101.

2. Godfrey, S.B., Rossi, M., Piazza, C., Catalano, M.G., Grioli, G. Bianchi, M., Zhao, K., Bicchi, A. (2017, December). Softhand at the Cybathlon: A user’s experience. In Journal of NeuroEngineering and Rehabilitation, 14(1), 124. 3. Rossi, M., Bianchi, M., Battaglia, E., Catalano, M.G., Bicchi, A. (2018, May).

Hap-Pro: a wearable haptic device for proprioceptive feedback. In IEEE Transac-tions on Biomedical Engineering[In press].

4. Rossi, M., Catalano, M.G., Bicchi, A., L. J. Hargrove: Simultaneous and Propor-tional Myoelectric Control of a Multi-DOF Transradial Prosthesis. [Submitted].

International Conferences/Workshops with Peer Review

1. Rossi, M., Altobelli, A., Godfrey, S. B., Ajoudani, A., Bicchi, A. (2015, Au-gust). Electromyographic mapping of finger stiffness in tripod grasp: a proof of concept. In Proceedings of IEEE International Conference on Rehabilitation Robotics (ICORR)(pp. 181-186).

2. Godfrey, S. B., Altobelli, A., Rossi, M., Bicchi, A. (2015, August). Effect of ho-mogenous object stiffness on tri-digit grasp properties. In Proceedings of IEEE In-ternational Conference on Engineering in Medicine and Biology Society (EMBC) (pp. 6704-6707).

3. Ajoudani, A., Hocaoglu, E., Altobelli, A., Rossi, M., Battaglia, E., Tsagarakis, N., Bicchi, A. (2016, May). Reflex control of the Pisa/IIT SoftHand during ob-ject slippage. In Proceedings of IEEE International Conference on Robotics and Automation (ICRA)(pp. 1972-1979).

(10)

4. Rossi, M., Della Santina, C., Piazza, C., Grioli, G., Catalano, M., Bicchi, A. (2017, July). Preliminary results toward a naturally controlled multi-synergistic prosthetic hand. In Proceedings of IEEE International Conference on Rehabilita-tion Robotics (ICORR)(pp. 1356-1363).

Others

1. Godfrey, S.B., Piazza, C., Catalano, M.G., Rossi, M., Bicchi, A. (2016, October). The SoftHand Pro-H: a shoulder-driven, motor-actuated prosthetic solution. In Cybathlon Symposium, SWISS Arena.

2. Godfrey, S.B., Piazza, C., Catalano, M.G., Rossi, M., Bianchi, M., Grioli, G., Bicchi, A. (2017, August). The Softhand Pro-H: A Prosthetic Platform For Work-Oriented Applications. In Myoelectric Controls Symposium.

(11)

Introduction

The loss of a limb is a personally and socially dramatic event, especially for upper limb, due to the fine motor tasks carried out by the hand and arm. For millennia, people have tried to use the most advanced technologies available at their time to build sophisticated upper-limb prostheses that could help with such a physical loss and partially restore the lost functionality [1]. In his Naturalis Historia, Pliny the Elder wrote about Marcus Sergius, a Roman general who lost his right hand while fighting in the Second Punic War (218 to 201 BC) and was able to return to battle after receiving an iron prosthesis.

M. Sergius in his second campaign lost his right hand; in two campaigns he was wounded twenty-three times, with the result that he was crippled in both hands and both feet, only his spirit being intact; yet although disabled, he served in numerous subsequent campaigns. [. . . ] He had a right hand of iron made for him and going into action with it tied to his arm, raised the siege of Cremona, saved Piacenza, captured twelve enemy camps in Gaul: all of which exploits are testified by his speech delivered during his praetorship when his colleagues wanted to debar him from the sacrifices as infirm - a man who with a different foe would have accumulated what piles of wreaths! [2] Another famous example of historic upper-limb prosthesis is given by the iron hand of Götz Von Berlichingen. Götz was a German knight who had lost his right hand in battle, during the siege of Landshut in 1508, and became famous thanks to the poet Johann Wolfgang von Goethe, who wrote a play based on the life of the German knight. Like most artificial limbs of the 16th and 17th centuries [3], Götz’s prosthesis was a heavy and cumbersome iron hand, crafted with the same technology and knowledge that was used for the manufacturing of armors. With his iron hand, Götz’s was able to grip weapons, hold reins and, ultimately, return to battle.

In 1818, Peter Baliff, a German dentist, developed what is now considered the first example of body-powered prosthesis [1, 4]. In Baliff’s design, the terminal device of the upper-limb prosthesis could be actuated by the user through a movement of the contralateral shoulder; the movement of the shoulder was transmitted to the terminal device by means of leather straps. Thanks to the use of lightweight materials and ingenious mechanisms, at the end of the 19th century the field of prosthetics had made

(15)

significant progress, and along with progress came, for some, the overly optimistic perception that upper-limb prostheses were already adequate substitutes for the human arm. In the chapter on upper-limb prostheses of his treatise on artificial limbs [5], George Edwin Marks wrote:

We are frequently amused by reading newspaper articles of artificial arms, made by forgotten mechanics, “that are fully as good as natural arms.” We frequently have to listen to the narration of some magical performances of men who wear artificial arms. [. . . ] We have read a great many newspaper articles equally as absurd as the above, and, being acquainted with every method used throughout the world for the construction of artificial limbs, we brand all such stories as fabrications of poorly informed but highly imagi-native newspaper correspondents.

As pointed out by Marks, despite the enthusiasm of some newspaper articles, a device capable of replicating the dexterity and functionality of the human arm did not exist. Even in the present days, although significant progress has been made in the design of dexterous anthropomorphic prostheses, building an adequate substitute for the human arm remains an open problem.

Currently available upper-limb prostheses can be divided into three categories: cos-metic, body-powered, and externally-powered (typically myoelectric) prostheses. Cos-metic prostheses do not usually have moving joints, their functionality is limited and their aim is primarily aesthetic. This aesthetic function can be a very important factor in psychological well-being, but the lack of adequate functionality may not be suffi-cient for all users [6]. In contrast, body-powered prostheses, which are the result of the evolution of Baliff’s invention, are operated by means of a cable control system and offer a more functional replacement. These prostheses are durable and allow per-formance of heavy work in punishing environments that include exposure to dirt or liquids [7]. An important disadvantage of this type of prostheses is that wearers may need to make abnormal movements of the shoulder or wrist in order to operate them. These movements, called compensatory motion [8], [9], and the discomfort they cause have been cited among the main factors influencing prostheses abandonment [10]. Fur-thermore, a person that is unable to generate sufficient force may not be able to operate a body-powered prosthesis. This consideration is particularly true for individuals with limb-loss that prefer to have an anthropomorphic terminal device: body-powered hands require much higher force at the shoulder to activate the prosthesis in comparison to body-powered hooks. Because of the difficulty of use and weak grip, many individu-als with amputation reject body-powered hands [11]. Body-powered hooks, however, may be aesthetically objectionable to some users, particularly while adjusting to loss of limb, but are ultimately preferred over body-powered hands because they are lighter and easier to use [12].

In the 20th century, a new generation of prostheses has seen the incorporation of electrically powered actuators controlled via electromyographic (EMG) signals from the muscles of the residual limb. The aim of these devices, called myoelectric prosthe-ses, is to provide increasing functionality without sacrificing appearance. Myoelectric prostheses can be operated with minimal effort form the user with respect to body-powered prostheses. Though compensatory motion is still seen in users of myoelectric prostheses, it is often less pronounced because the control is provided by the ipsilateral

(16)

arm rather than involving the contralateral side. Thanks to the advancements in the robotic field, today’s myoelectric prostheses are compact, lightweight and very dexter-ous. However, myoelectric prostheses are not considerably better their older counter-parts [13]; these sophisticated devices tend in fact to disappoint the high expectations of their potential users, who ideally want a prosthesis able to replicate the dexterity, sen-sitivity, and strength of a human hand, while being at the same time easy to use. This discrepancy between the users’ expectations and the products that are currently avail-able on the market is evident when looking at the rejection rate of upper-limb prosthe-ses: roughly, one third of these prescribed prostheses are completely rejected [14]. But which are the factors that lead to the abandonment of these technologically advanced myoelectric prostheses? The problem seems to lie in the interfaces with the users. Dif-ficulty in the control and absence of haptic feedback are cited among the major reasons for prosthetic abandonment in numerous studies [15–17]. Modern myoelectric hands are in fact designed to match many of the functions of human hands through complex design, and are capable of performing a large span of grasp shapes, but users struggle to control them [18]. Also, although sensory feedback is of primary importance for prosthesis users [19], commercial devices do not generally provide haptic feedback, and myoelectric users must therefore rely heavily on visual feedback when operating their prostheses.

This thesis aims at addressing the shortcomings of today’s upper-limb prostheses by dealing with the design possibilities concerning the next generation of prosthetic inter-faces from a user-centered perspective. In particular:

Chapter 1 analyzes the role that co-contractions of the extrinsic hand muscles play during grasp, and gives insight into how myoelectric interfaces should be designed in order to better translate the intentions of users into the desired movements. The chapter is articulated in three main sections that describe three different studies: (I) a mapping between the finger stiffness and the EMG activity of the forearm muscles is extracted, (II) experimental findings on how humans modulate their muscle activity while grasp-ing objects of varygrasp-ing levels of compliance are presented, (III) the stiffness modulation is replicated in a robotic hand for prosthetic use.

Chapter 2 explores the various sensory feedback methods that have been proposed in literature and deals with the specific problem of conveying information on the open-ing of a hand prosthesis to its user in a non-invasive way. A device for proprioceptive feedback in upper-limb prosthetics is presented: the HapPro, which uses a rolling con-tact to convey information on the opening of an artificial hand and was developed as part of this thesis. Effectiveness of its use to convey proprioception with a compliant prosthetic hand (Pisa SoftHand Pro) is investigated, both from a quantitative point of view through perceptual experiments and from a qualitative point of view through a Likert-style survey. Preliminary tests with an amputee are also presented.

Chapter 3 explores the possibility of developing a new set of prosthetic configura-tions suitable for transradial amputaconfigura-tions, mainly designed for work-oriented, home chore and hobby use. The goal is to merge the simplicity of a prosthesis operated via a body-powered cable with the versatility, appearance and the reduced physical demand of robotic devices. Among the analyzed configurations, one solution has been imple-mented in a functional prototype and used by the team SoftHand Pro in the Cybathlon 2016 - Powered Arm Prosthesis Race.

(17)

Chapter 4 still addresses the problem related to the control of robotic prostheses, but with a focus on poliarticulated prostheses. While natural movements result from smooth and fluid coordination of multiple joints, robotic prostheses are in fact still limited to sequential control of multiple degrees of freedom (DOFs), or, at best, are constrained to move along predefined patterns. This chapter takes a step forward to-wards the development of more intuitive and advanced control interfaces for upper-limb prostheses by evaluating the use of algorithms that allow the simultaneous and propor-tional control of multiple joints. The experiments presented were performed in realistic conditions, both with healthy individuals and persons with amputation.

(18)

CHAPTER

1

Insights from Human Studies on Stiffness

Modulation Strategies for Prosthetic Control

Design

The most common strategy for controlling myoelectric prostheses is known as “di-rect control”. As the name suggests, di“di-rect control uses the electromyographic (EMG) signals from a pair of agonist/antagonist muscles to directly control a single degree of freedom (DOF). Different implementations of direct control can make slightly different decisions on whether to move a joint in one direction or another, depending for instance on which muscle of the pair was activated first, or which EMG signal has the highest amplitude. Also, when dealing with multiple DOFs, a short burst in the activation of both muscles can be used to switch the DOF that has to be controlled. However, it is generally true that in direct control attention is focused upon single flexion/extension movements: when controlling a DOF, co-contractions are considered as a disturbance and are either ignored (first come first served implementation) or simply taken into account just to reduce the amplitude/speed of the final movement (proportional imple-mentation). This approach does not take into account the role that co-contractions play in normal motor activities, a role that has been observed as early as 1925 [20]. An-tagonist co-contraction can in fact serve several purposes, including monitoring limb position, especially when learning a new task, decelerating the limb in ballistic move-ments, and increasing stiffness [21]. Stiffening behavior can be realized to stabilize movement or to fix posture in isometric tasks [22].

This first chapter analyzes the role that co-contractions of the extrinsic hand mus-cles play during grasp. The purpose is twofold: on the one hand, understanding the role of co-contractions during grasp can lead to the design of interfaces for myoelec-tric prostheses that are able to better exploit the information that is already present in

(19)

the EMG signals used for the control. On the other hand, the observation of stiffness modulation strategies of the hand during grasp can bring new insights on how to build new bio-inspired control strategies for smart artificial hands. Three studies are pre-sented. The first section investigates the possibility of creating a mapping between the endpoint stiffness of the fingers and the EMG activity of a pair of agonist/antagonist extrinsic muscles of the hand; the second section analyzes how hand muscle activity is modulated when grasping objects of varying level of compliance; finally, the third sec-tion presents an example of bio-inspired control for robotic hands that exploit stiffness modulation to guarantee robustness against slippage during grasp.

(20)

1.1. Inferring Finger Stiffness from EMG Signals

Figure 1.1: Experimental setup used for the trials. The KUKA robot applies planar perturbations to the fingers of the subject and the resulting forces are measured at the fingertips along with the surface EMG signals from the flexor digitorum superficialis and extensor digitorum communis muscles.

1.1

Inferring Finger Stiffness from EMG Signals

Previous work examining finger and hand stiffness has explored various topics includ-ing the mechanical impedance of the finclud-ingers [23] or at the finclud-ingertip [24], pinch grasp stiffness during an isometric grasp task [25], or variance of stiffness depending on fin-ger force or posture [26]. The estimation of the impedance parameters in these studies is mainly achieved by an off-line post-processing phase, imposing severe limitations in real-time applications such as tele-impedance control of the prosthetic or robotic hands. At first, this might imply that a complete model and thus control of the finger motion and stiffness trajectories are required to perform a target manipulation task. However, observations in human control suggest that the central nervous system solves for this complexity in an elegant and coordinated manner which has been well-recognized with the concept of hand synergies [27–29]. While the exploitation of this concept in kine-matic coordinates has lead to the development of several simple, effective and adaptive robotic designs and control strategies (e.g., see [29, 30]), its extension to dynamic co-ordinates, such as coordinated stiffening of the fingers, remains to be investigated.

(21)

Toward the purpose of investigating the presence of coordinated regulations of the finger stiffness in human hand and establishing a real-time technique in modelling and identification of the finger stiffness while grasping, this study explores the relation-ship between the fingertip stiffness and the EMG activity of the antagonist muscles contributing to this profile. To achieve this, the experiments are performed using a sen-sorized object, called Tripod Device, that was designed to be grasped by using specific contact points. A 6-DOF force/torque sensor is positioned at each contact point and a 6-DOF force/torque sensor is mounted at the base of the device. While constrained in a tripod posture in the Tripod Device, subjects were asked to hold a stable level of stiffness without applying grasping forces and experienced a series of perturbations pro-vided by the KUKA lightweight robot arm. EMG was recorded alongside force/torque measurements. Consequently, the mapping between the EMG data and the fingertip stiffness profiles, as calculated from the force/torque measurements, was established. 1.1.1 Materials and Methods

Study Design

Five subjects participated in the experiment, 3 males and 2 females aged 28 ± 3 years. Before participating, subjects signed an informed consent form approved by the local ethical committee. Subjects placed their fingers in a Tripod Device and were asked to maintain a steady level of hand stiffness, as measured by surface EMG, while experi-encing perturbations provided by the KUKA lightweight robot (Figure 1.1). Subjects completed the first trial while relaxed and then increased stiffness to low, medium, and finally high levels in subsequent trials. The block of four trials was repeated three times for a total of 12 trials. Each perturbation trial lasted 35 seconds and the experiment lasted less than an hour.

Tripod Device and Experimental Setup

The Tripod Device is an instrumented manipulandum that can be grasped with three fingers and includes three individual contact surfaces. Each contact surface consists of a contact module rigidly attached to the structure of the manipulandum, through an interface engineered in Acrilonitrile-Butadiene-Stirene (ABS) rapid prototyping mate-rial. Every contact module consists of a cylindrical base in ABS (rigid case, Young’s Modulus = 1.4 GPa). To minimize structural deformations, the core frame of the Tripod Device was built in aluminum using a CNC (Computer Numerical Control) machine.

A force/torque sensor (Series Nano 17 by ATI, Apex, NC, USA) was positioned below the interface where each contact module was attached to measure the force and torque components applied by each finger. A finger-slot was designed and fixed to each contact surface to minimize the relative movements between the finger and the Tripod Device. A fourth F/T sensor (Series Nano 45 by ATI, Apex, NC, USA) placed at the base of the structure provided an independent measure of the external wrench. An exploded drawing view of the manipulandum with dimensions is reported in Figure 1.2.

The Tripod Device was mounted on the end-effector of a 7-DOF robot arm: the KUKA lightweight robot (KUKA/DLR). All force and displacement measures were reported in the base reference frame of the KUKA. The KUKA, which has a positioning

(22)

frame

F/T Sensor

kuka interface

finger-slot

contact module

(a)

104

12 0,

7

(b)

40

(c)

Figure 1.2: (a) Exploded view, (b) lateral view, and (c) top view of the Tripod Device and its main features, with dimensions in [mm].

repeatability of ±0.05 mm, was programmed to follow a planar random trajectory, keeping constant the orientation angles of the end-effector (roll, pitch, yaw), so that the Tripod Device remained parallel to the ground and maintained the same orientation with respect to the fixed frame (Figure 1.1). The subjects adopted a tripod posture and inserted the index finger, the middle finger, and the thumb in the dedicated finger-slots of the Tripod Device. At each finger, the forces in response to the position perturbation were measured by the contact point F/T sensors described above. Surface EMG signals on the forearm were measured and amplified with a Delsys-Bagnoli 16 (Delsys Inc.). The data acquisition and synchronization interface between the KUKA controller, the four F/T sensors, and the EMG acquisition board were developed in Microsoft Visual C++ environment.

(23)

Protocol

Subjects were seated for the duration of the experiment. Surface EMG electrodes were placed on the FDS and EDC muscles. To minimize cross-talk from neighboring mus-cles, the electrodes were positioned following the methods described by Perrotto et al. [31].

Each subject placed his or her thumb, index, and middle fingertips in the finger-slots of the Tripod Device; the subject’s arm and wrist were immobilized against a board us-ing an orthosis, at an angle that allowed the subject to comfortably maintain the tripod grasp parallel to the ground, see Figure 1.1. In a pre-trial, the KUKA did not perturb the subject, and the subject was instructed to produce maximum hand stiffness without applying any force to the F/T sensors. The level of co-contraction produced in this trial was used as an upper-bound for the co-contraction level in subsequent trials. In the first trial, the subject remained relaxed while the KUKA perturbed the subject fol-lowing the trajectory described above. In subsequent trials, while the KUKA perturbed the hand, subjects were asked to produce respectively a “low”, “medium” and “high” level of stiffness without squeezing the Tripod Device; to help them maintaining con-stant and coherent co-contraction levels, subjects were provided visual feedback on the co-contraction level and the visualized targets for the three conditions corresponded roughly to 20%, 40%, or 60% of the maximum level of co-contraction previously ob-tained. Subjects were also instructed to prioritize stability of co-contraction level over accuracy of targeted level; that is, subjects aimed to keep a low standard deviation over producing a particular mean co-contraction level. The block of four trials was repeated three times for a total of 12 trials.

Data Analysis

To estimate the endpoint stiffness at each of the three fingers, the same techniques used in [32] for the arm were adopted for this study. Following Perrault et al. [33], 35 seconds of continuous stochastic perturbations in x and y directions were applied to the subject’s fingers through the Tripod Device. The perturbations had a peak-to-peak value of 10 mm in each direction and a frequency spectrum that was flat in the range of 0 to 6 Hz and null elsewhere. The first 5 seconds of data were discarded to allow the subject to reach the required stiffness level.

For each finger, the multiple-input, multiple-output (MIMO) dynamics of the end-point stiffness were decomposed into four linear single-input, single output (SISO) subsystems; the identification of each SISO subsystem was performed in the frequency domain using a nonparametric algorithm [34]. The endpoint inertia, viscosity and stiff-ness matrices, I, B and K, where found by comparing each SISO transfer function with a second order linear model of the type:

Gi, j(s) = Ii, js+ Bi, js+ Ki, js, i, j = x, y.

The external wrench measured at the base of the Tripod Device with the ATI Series Nano 45 force/torque sensor was compared with the external wrench derived from the three force/torque sensors placed under the fingers to verify that the measurements were correct. The surface EMG signals were acquired with a Delsys-Bagnoli 16 apparatus, sampled at 750 Hz, high-pass filtered at a cut-off frequency of 4 Hz with a 4th order

(24)

Figure 1.3: x components of a typical endpoint displacement d(t) and the resulting force F(t). F(t) and d(t) were used to estimate the endpoint stiffness of the fingers by means of a nonparametric algorithm in the frequency domain [34].

Butterworth filter and then rectified. Finally, each rectified signal was low-pass filtered in order to obtain its envelope. The average values of the EMG signals relative to the FDS and EDC muscles, respectively p0 and p00, were calculated at each trial. The resulting level of co-contraction (Lcc) was computed as:

Lccs,t = 1 2( p0_s,t p0_s,max+ p00_s,t p00_s,max)

Where s indicates the subject, t the trial number and p0_s,max, p00_s,maxthe maximum EMG values recorded respectively at channel 1 and channel 2 of subject s.

1.1.2 Results

Figure 1.3 shows the x component of a typical endpoint displacement d(t) along with the x component of the resulting force F(t) measured at the index fingertip. To evaluate the linear dependency of each output (forces) to all system inputs (displacements), the multiple coherence indices were computed on the obtained measurements. A strong lin-ear dependency of the inputs and the outputs was found in the frequency range 0-6 Hz, as shown in the Figure 1.4; for this reason the parameter estimation was performed in the same range.

After estimating each stiffness matrix K, its symmetric Ks and asymmetric Kaparts were extracted: Ks= 1 2(K + K T₎ _K a= 1 2(K − K T₎ The error of approximation was computed as:

e= ||Ka||2 ||Ks||2 , obtaining a mean value ¯e≈ 0.07.

(25)

0 1 0 0.5 1 Multipl e Coheren ce Frequency [Hz] Thumb F x 0 1 0 0.5 1 Multipl e Coheren ce Frequency [Hz] Thumb F y 0 1 0 0.5 1 Multipl e Coheren ce Frequency [Hz] Index F x 0 1 0 0.5 1 Multipl e Coheren ce Frequency [Hz] Index F y 0 1 0 0.5 1 Multipl e Coheren ce Frequency [Hz] Middle F x 0 1 0 0.5 1 Multipl e Coheren ce Frequency [Hz] Middle F y Subj. 1 Subj. 2 Subj. 3 Subj. 4 Subj. 5 Subj. 1 Subj. 2 Subj. 3 Subj. 4 Subj. 5 Subj. 1 Subj. 2 Subj. 3 Subj. 4 Subj. 5 Subj. 1 Subj. 2 Subj. 3 Subj. 4 Subj. 5 Subj. 1 Subj. 2 Subj. 3 Subj. 4 Subj. 5 Subj. 1 Subj. 2 Subj. 3 Subj. 4 Subj. 5 0 0 0 0 0 0

Figure 1.4: Example of Multiple Coherence function values for the five subjects. A value close to 1 indicates linear dependency of the inputs and the outputs.

250 N/m Stiffness Levels Rest Low Medium High

Index Finger Middle Finger

Thumb

X y

z

Figure 1.5: Endpoint stiffness ellipses generated by Subject 1 during four consecutive trials. Each ellipse represents the planar stiffness measured for a particular finger (index finger, middle finger or thumb) and during a specific trial (rest, low stiffness, medium stiffness, and high stiffness conditions).

The eigenvalues λ1and λ2(with λ1< λ2) of Ks and the corresponding eigenvectors, v1and v2, were computed. In all of the examined cases, Ks was found to be positive definite, with λ1and λ2real and greater than 0.

Figure 1.5 presents the endpoint stiffness ellipses that were generated by one of the subjects during four consecutive trials (with four different indications on the stiff-ness set-point level). Stiffstiff-ness ellipses are a consolidated method of representing the

(26)

Table 1.1: Average ellipse orientations for the four targeted stiffness levels.

θ (◦) Finger Rest Low St. Medium St. High St. Subject 1 I 8.1 10.3 0.2 5.7 M 12.1 14.6 15.5 15.6 T 10.6 14.0 14.2 19.1 Subject 2 I 20.8 22.4 24.1 −15.0 M 24.8 25.9 32.2 22.8 T 12.3 1.5 −6.1 0.5 Subject 3 I −1.6 3.1 4.1 4.1 M 28.2 7.5 2.3 3.2 T 5.4 11.4 2.5 −2.0 Subject 4 I 34.1 8.9 6.7 −11.6 M 27.3 15.6 20.5 24.7 T 27.3 23.5 22.0 22.9 Subject 5 I 15.3 13.5 15.5 16.7 M 15.2 18.1 −10.7 −25.7 T 13.7 20.3 23.3 −10.2

endpoint stiffness. In the 2D case, the major and minor axes of the ellipse represent respectively λ2 and λ1, while the orientation θ of the ellipse is given by the angle be-tween v2 and the x axis. As expected, the stiffness ellipse area (A = πλ1λ2) increases with increasing targeted stiffness levels. On the other hand, the orientation θ , as well as the shape (here quantified with the ratio λ1

λ2), of the stiffness ellipse does not appear

to be correlated with the targeted stiffness levels. Table 1.1 reports the mean values of the stiffness ellipses’ orientations corresponding to different targeted stiffness levels. For each subject, the values are distributed in three different rows I, M and T that cor-respond respectively to the index finger, middle finger and thumb. Following the same structure of Table 1.1, Table 1.2 presents the average ratio λ1

λ2 with respect to different

targeted stiffness levels.

To further investigate the behavior of the stiffness components λ1 and λ2 at each finger, the measured surface EMG signals from the FDS (channel 1) and EDC (channel 2) muscles were used as an indicator of the global stiffness of the hand. Figure 1.6 shows the filtered EMG signals acquired from one of the subjects during three different trials. The three trials corresponded to three increasing targeted stiffness levels. Each signal was normalized to the maximum value produced by the subject during the pre-trial phase for the corresponding EMG channel.

The acquired EMG signals were used to compute the level of co-contraction (Lcc) index for each trial. To measure the correlation between the stiffness components at each finger and the Lcc index, Pearson’s Correlation Coefficient (PCC) was used. PCC produces a measure of the linear correlation between two measures; it can range from −1 (total negative correlation) to 1 (total positive correlation), with 0 indicating absence of correlation. The average correlation coefficient obtained between λ and Lcc was:

(27)

Table 1.2: Average ratioλ1

λ2 of the ellipse axes for the four targeted stiffness levels.

λ1

λ2 Finger Rest Low St. Medium St. High St. Subject 1 I 0.13 0.09 0.19 0.25 M 0.27 0.31 0.27 0.23 T 0.19 0.14 0.16 0.17 Subject 2 I 0.41 0.34 0.31 0.30 M 0.37 0.26 0.22 0.19 T 0.22 0.17 0.20 0.30 Subject 3 I 0.22 0.19 0.17 0.16 M 0.51 0.29 0.26 0.32 T 0.48 0.44 0.32 0.20 Subject 4 I 0.61 0.43 0.61 0.58 M 0.07 0.32 0.42 0.40 T 0.23 0.31 0.40 0.46 Subject 5 I 0.14 0.14 0.18 0.21 M 0.18 0.25 0.33 0.19 T 0.17 0.23 0.26 0.40

Figure 1.6: Typical normalized EMG signals for three different targeted stiffness levels. The data are relative to Subject 2.

For every subject, the normalized values of λ1and λ2were linearly fitted with respect to Lcc. A total of six fittings were performed for each subject (two per finger) by imple-menting least-squares regression to find the coefficients mi, f and qi, f in the following equation:

λi, f λ_{i, f}MAX

≈ mi, fLcc+ qi, f i= 1, 2

where f = T, I, M indicates thumb, index or middle finger, respectively, and λ_{i, f}MAX is the maximum value of λi, f among the 12 trials. The results for one of the subjects are presented in Figure 1.7. The slopes of the fitting lines for each subject along with the

(28)

Figure 1.7: Linear regression of normalized stiffness axes with respect to the level of co-contraction (Lcc). The graphs are relative to Subject 3.

Table 1.3: Angular coefficients mi, fof the fitting lines along with the average coefficient of determination

R2for each subject.

m1,I m2,I m1,M m2,M m1,T m2,T R2 Sub ject1 1.20 1.25 1.13 1.17 1.25 0.71 0.73 Sub ject2 1.19 1.18 0.93 0.96 1.11 0.61 0.86 Sub ject3 1.31 1.33 1.14 1.34 0.91 0.53 0.72 Sub ject4 0.88 0.74 0.45 0.67 0.70 0.50 0.42 Sub ject5 1.68 2.60 2.13 1.67 1.95 1.13 0.60

mean coefficient of determination R2are given in Table 1.3. 1.1.3 Discussion

The experimental results corroborate the feasibility of a generalized mapping between the EMGs recorded from the forearm and the hand stiffness. Since the aim of this work was not the generation of an accurate model of hand stiffness control but instead to move a step towards the design of more natural and better performing control inter-faces for hand prostheses and teleoperation, the focus was primarily on the relationship between the hand stiffness and the FDS and EDC muscles. In particular, the endpoint stiffness of the fingers was represented with stiffness ellipses and it was found that their area increased with respect to increasing levels of co-contraction Lcc, while their ori-entation and shape did not seem to be related to Lcc. In fact, the variation of θ and λ1

λ2 across trials as seen in Table 1.1 and 1.2 did not appear to depend on the requested

stiffness level; this conjecture is supported by the studies on the human limb stiffness, which state that the orientation and shape of the endpoint stiffness ellipses is mainly

(29)

determined by posture [35].

In general, the endpoint stiffness of the thumb, index and middle fingers was found to increase when the level of co-contraction Lcc increased. Furthermore, the results of the linear fittings (Table 1.3) show that, not only the relationship between stiffness and Lccwas nearly linear, but also that, with the exception of m2,T, the slopes of the fitting lines tended to maintain a similar value within the same subject. This result highlights a tendency of the fingers to stiffen in a coordinated way that is also proportional to the level of co-contraction of the EDC and FDS muscles.

One limitation of this work was that grip force was not taken into account. The subjects were in fact asked not to produce any grip force and the request was fulfilled in all the trials with the exception of Subject 4 and Subject 5 during the high stiff-ness condition. This simplification was made in order to better understand the role of co-contraction in the control of hand stiffness, however, in order to produce a simple but efficient generalized mapping between hand stiffness and co-contraction, further studies should be conducted.

(30)

1.2. Effect of Object’s Stiffness on Grasp Properties

1.2

Effect of Object’s Stiffness on Grasp Properties

Multi-finger grasp is a complex task that is normally performed by humans without any effort. When grasping and lifting an object, the constraint on the grip force (normal to the surface of the object) is unilateral: it must exceed a value that depends on the weight of the object (load force) and on the friction coefficient between the fingertips and the object. Due to the minimal constraints and redundancy within the human motor system, the problem of choosing the correct grip posture and force can potentially have an infinite number of solutions. Understanding the way in which the central nervous system (CNS) regulates the force during the grasping and lifting phase of an object is therefore not simple but of primary importance. Several studies [36], [37], [38] have shown that the CNS makes use of specific patterns of co-activation of muscles to grasp an object. He et al. [39] for example suggested that the coordination of multiple hand muscles seems to be invariant across different grasp forces and different contraction history profiles.

Undoubtedly, human motor control could be affected by several factors as it per-forms a grasping task. In [40], [41], [42] the authors investigated the effects due to the geometry, the friction, and the weight of the grasped object on the value of grip forces and load forces. Another important property that can influence the contact force distri-bution is the object compliance: compared with rigid objects, compliant objects present additional uncertainties and Winges et al. [43] showed that, during a grasp, when one or two contact points are compliant, the activation patterns of finger muscles are differ-ent with respect to the case where the contact points are rigid. Besides analyzing the grip forces, to fully understand the control of hand grasping by the CNS, it is important to study how the hand stiffness is regulated during a grasp: stiffening behavior is com-monly realized to stabilize movement or to fix posture in isometric tasks [22] and recent findings suggest that, to some extent, grip stiffness is independent from grip force [44]. This study investigates the relationship between object compliance and grasping stiffness of the hand. To achieve this goal, experiments with 11 subjects were conducted using a modified version of the Tripod Device presented in section 1.1.1. For this study, the rigid contact modules were in fact substituted with contact modules characterized by a certain level of stiffness: rigid, high, medium, or low stiffness. The experiment consisted of four blocks of trials, corresponding to the four different levels of stiffness; in each trial the subject grasped and lifted the Tripod Device 25 times while EMG was recorded from the flexor digitorum superficialis and extensor digitorum communis. These two muscles are the main finger antagonist pair and thus can be used to monitor the EMG activity resulting in the production of grasp force as well as overall hand stiffness; this assumption is in agreement with the capability of the human control system to increase hand stiffness exploiting the co-contraction of antagonist muscles [21] and with the study presented in the previous section.

1.2.1 Materials and Methods

Study Design

Eleven healthy volunteers participated in this study (5 males and 6 females, mean age 28 ± 3 years, 10 right-handed). Before starting the study, all participants signed an informed consent document that was previously approved by the regional ethics

(31)

com-Figure 1.8: Experimental setup used for the trials. A subject is grasping the Tripod Device using his or her index finger, middle finger and thumb. EMG activity of FDS and EDC muscles is recorded by the electrodes positioned on the forearm.

mittee. The study consisted of four blocks of 25 trials each in which the subject grasped and lifted the Tripod Device while EMG was recorded from the main finger flexor and extensor muscles (Figure 1.8). The device was held steady for a brief period and placed back on a table. At the contact points for the thumb, index, and middle fingers, an in-terface of varying rigidity was placed. Three silicone inin-terfaces of compliant, medium, and high stiffness were used as well as a rigid ABS plastic interface covered in a thin film of silicone, to match haptic conditions. The order of the four blocks was random-ized to reduce order and learning effects.

Tripod Device and Experimental Setup

In this study, a modified version of the Tripod Device with respect to the one presented in Section 1.1.1 was used. An exploded drawing view of this version of the manipulan-dum is presented in Figure 1.9. Instead of the contact modules with finger-slots showed in Figure 1.2, for this experiment, new cylindrical contact modules were designed. The

(32)

1.2. Effect of Object’s Stiffness on Grasp Properties contact module contact module contact module F/T Sensor F/T Sensor F/T Sensor F/T Sensor frame base weight 78 145,9 40 (b) (c) (a)

Figure 1.9: (a) Exploded view, (b) lateral view, and (c) top view of the Tripod Device and its main features, with dimensions in [mm].

new contact modules were composed of a rigid interface with the aluminum frame made in Acrilonitrile-Butadiene-Stirene, and an external cylinder made in silicone or ABS. The silicone was obtained by mixing a given quantity of a commercial bicompo-nent, room temperature-curing silicone (BJB TC-5005A/B), with different percentage of plasticizer (BJB TC-5005C), acting as a softener. Softener was mixed at a percent-age of 45%, 20%, 0% as shown in [45] to obtain three different stiffness levels. A fourth specimen was made only with ABS. The four different contact surfaces have a Young’s Modulus of 200 kPa, 500 kPa, 750 kPa and 1.4 GPa and are referred to as low-, medium-, and high-stiffness silicone and rigid ABS conditions, respectively.

As in the previous study, the force and torque components applied by each finger were measured using three force/torque sensors (Series Nano 17 by ATI, Apex, NC, USA) fixed below each contact module. The effect of cables and the external wrench are monitored by a fourth F/T sensor (Series Nano 17 by ATI, Apex, NC, USA) placed at the base of the structure. The total weight of the manipulandum, including the sensor cables was 300 g. The Tripod Device was built to allow an additional component to be attached at the base to easily change the weight of the device; in this experiment, an additional 100 g was used for a total device weight of 400 g. Surface EMG signals

(33)

on the forearm were measured and amplified with a Delsys-Bagnoli 16 channel system (Delsys Inc.). The data acquisition and synchronization were performed in Simulink (Matlab R2012a) software exploiting the Data Acquisition Toolbox, Instrument Control Toolbox, and Simulink Block for Real Time Execution. Force and torque data from the Tripod Device were collected at 100 Hz, and EMG data at 1 kHz.

Protocol

Surface EMG sensors were placed on the main muscle belly of the Flexor Digitorum Superficialis(FDS) and Extensor Digitorum Communis (EDC) muscles following the identification and verification procedures outlined in [31] to minimize cross-talk. Max-imum voluntary contractions (MVC) were then collected for each muscle by asking subjects to contract against resistance provided by the experimenter. Subjects were seated in front of the Tripod Device, which was placed on a table. The device was equipped with the appropriate contact stiffness interface, according to the randomiza-tion table. Subjects were instructed to lift the object vertically, avoiding object tilt as much as possible. (Note: in order to encourage as natural a grasp as possible, subjects were given no instruction as to grasp force, e.g., to use the minimum force necessary to lift the device.) After a brief (1-2 second) pause, subjects then placed the Tripod Device back on the table. This procedure was repeated for 25 total lifts. Subjects proceeded at their own pace and were allowed to pause as needed to avoid fatigue both during the block of 25 trials and between blocks. After each block, the interface was changed to a new stiffness condition and a new block of 25 trials was conducted until subjects had completed all four conditions.

Data Analysis

The first five trials in each block were discarded to avoid learning or crossover effects. The vertical axis of the FT sensor at the base of the manipulandum was used to segment the data into lift, hold, and place phases of the subsequent 20 trials. The mean was subtracted from the EMG data to remove the DC offset before rectifying the data. EMG data was then normalized to the maximum contraction collected prior to the trials. The average value of the hold phase of the EMG and the normal force exerted by each finger were then calculated. The data were further synthesized into an average value for each condition for each subject. To perform group analysis, a repeated measures analysis of variance (RM ANOVA) was used. The RM ANOVA was performed on four sets of data: FDS and EDC EMG levels, co-contraction levels, and the sum of the index and middle finger contact point forces. When a main effect of stiffness was found, the data was then subjected to a post-hoc analysis using Bonferroni corrections for multiple comparisons.

1.2.2 Results

All subjects tolerated the protocol well, and each session lasted approximately an hour, including set-up and self-timed breaks. Subjects occasionally reported low levels of fatigue and were encouraged to break as needed to minimize fatigue effects. Measure-ments from a sample trial are shown in Figure 1.10: the normal forces and the weight

(34)

1.2. Effect of Object’s Stiffness on Grasp Properties 0 2 4 6 8 −12 −10 −8 −6 −4 −2 0 2 Time [s] Normal Forces [N] I M T (a) 0 2 4 6 8 −0.2 0 0.2 0.4 0.6 Time [s] Weight [Kg] (b) (c) (d)

Figure 1.10: An example of the measurements collected in one trial. (a) shows the normal forces exerted by thumb, index and middle fingers; (b) displays the weight of the object measured by the FT sensor. Examples of normalized FDS and EDC EMG signals during one trial are presented respectively in (c) and (d).

in Figures 1.10(a) and 1.10(b); the values of the EMG signals in the same time range in Figures 1.10(c) and 1.10(d).

A summary of the FDS, EDC, and co-contraction EMG data can be found in Figure 1.11. The group data was analyzed using RM ANOVA, as detailed in the preceding section. The FDS data violated the assumption of sphericity (using Mauchly’s test, p0.01), therefore the Greenhouse-Geisser correction was applied (ε = 0.446). No ef-fect of stiffness condition on FDS contraction levels was found (F=3.592, p=0.071). In contrast, there was a main effect of stiffness condition on EDC contraction (F=9.942, p0.01). This analysis was thus followed by post-hoc tests with Bonferroni correc-tions: EDC activity during the low-stiffness silicone condition was found to be sig-nificantly different from the high-stiffness silicone and rigid ABS conditions (p=0.021 and 0.001, respectively). Finally, the co-contraction values were analyzed: they vi-olated Mauchly’s test of sphericity (p=0.006), therefore the Greenhouse-Geisser cor-rection was again applied (ε = 0.583). There was a main effect of stiffness condition (F=6.280, p=0.011) and post-hoc tests showed a significant difference between rigid ABS conditions and low-stiffness as well as medium-stiffness silicone (p=0.045, 0.015, respectively).

(35)

sub-Figure 1.11: Average values of FDS, EDC, and co-contraction normalized to MVC, with standard error bars.

tracted from the sum at the index and middle finger contact points. The resulting dif-ference was found to be near zero, as expected (data not shown). The average of the sum of the index and middle finger contact forces is plotted in Figure 1.12. The index and middle finger contact force was analyzed using RM ANOVA as before and a main effect of condition was found (F=4.984, p=0.006). However, post-hoc analysis did not find any significant difference between condition pairs, possibly due to the conservative nature of the Bonferroni correction.

1.2.3 Discussion

Although subjects generally seemed to exhibit higher FDS activity for more compliant conditions than more rigid conditions, results also showed large intra- and inter-subject variability. There was no significant difference due to stiffness conditions in the FDS activity. The EDC results, however, showed a clearer trend of increasing activity with decreasing stiffness with a significant effect of stiffness. This trend was visible in the post-hoc results showing that grasping and lifting the low-stiffness silicone resulted in significantly higher EDC EMG activity than the two highest stiffness conditions. Finally, there was a main effect of stiffness on co-contraction levels, with post-hoc tests showing EMG co-contraction was significantly lower when grasping and lifting the rigid ABS compared to both the low- and medium-stiffness silicone.

For what concerns the force produced during each condition, though there appears to be a trend toward increasing force with decreasing stiffness, and indeed a main effect of stiffness on force levels; post-hoc testing did not reveal any significant differences between specific condition pairs. It is possible that this effect is masked by the conser-vative nature of the Bonferroni correction. Taken in combination, these results suggest that the motor system responds to the increase in compliance by increasing the activity of the antagonist muscle, ultimately resulting in higher co-contraction levels from the antagonist pair and an overall stiffening of the hand. This increased stiffness would thus serve to counterbalance the decrease in stability of the grasp caused by the increased

(36)

1.2. Effect of Object’s Stiffness on Grasp Properties

Figure 1.12: Average force at the index and middle finger contact points, normalized to MVC, with standard error bars.

compliance at the contact points.

The results shown here suggest a decoupling of flexor and extensor activity with changing object compliance, despite relatively stable grasp posture. Further, it is worth noting that because the Tripod Device was grasped from above with only fingertip contact, subjects may have been more likely to increase stiffness to produce a more secure grasp, especially as the contact compliance increased. In a power or conformal grasp, this stiffness effect may thus be decreased due to the increased positional stability and thus reduced reliance on grasp force and/or stiffness.

(37)

1.3

Stiffness Modulation to Prevent Slippage: What Robots Can Learn from Humans

Sections 1.1 and 1.2 of this chapter presented two studies that investigate the role that co-contractions of the extrinsic hand muscles play during grasp. These studies were performed with human subjects and aimed at gaining a better understanding of the strategies of co-contraction and stiffness modulation that humans adopt during grasp. This section describes the transition from some of the insights gained from these stud-ies, to the implementation of a control strategy that could be applied to modern robotic prostheses.

Two experiments are described in this section. Prior to the design of the robotic hand’s reflex controller, a preliminary human grasping experiment was carried out to understand the underlying biomechanical principles that result in stable and reliable grasping of various objects1. Such principles can potentially be a source of inspiration to achieve an appropriate mechanical interface between the fingertips of the robotic hand and the object. After this preliminary experiment, as a primary step towards the accomplishment of a reliable grasp in an underactuated anthropomorphic prosthetic hand, different reflex control strategies, namely Current, Pose and Impedance, were implemented and experimentally evaluated. Given the adaptive and synergy-driven hand functionalities, the design of the Impedance hand reflex controller was inspired by the observations in human grasping experiments described in Sections 1.1, 1.2 and 1.3.2.

1.3.1 Background

The human central nervous system steadily regulates the grip forces exerted by the hand in order to avoid slippage of the object out of the grasp with a safety margin of 10 − 40% [48]. At the very beginning of the grasp, gripping force is generated depending on the estimation of the load and altered based on the feedback provided by the hand receptors. Such behavior is dominated by the spinal cord and recognized as reflex control.

The most basic way to achieve a similar performance in prosthetic or robotic hands is to exert high level grip forces to prevent slippage of the grasped object. This method, however, poses some difficulties while dealing with fragile or deformable objects. As a consequence, toward the twofold purpose of improving the grasp robustness against slippage and avoiding the generation of unnecessarily high interaction forces, some robotic/prosthetic hands replicate a human-like reflexive behavior and regulate grip forces once a slippage is detected [49, 50].

Slip detection has been realized with the aid of various techniques, e.g., vibration, optical tracking, pressure, and force vector control [51–55]. Based on the studies in neurophysiology, humans perceive slip with the occurrence of firing activity generated in Pacinian corpuscles, which are nerve endings in the skin, sensitive to high frequency vibrations [56]. In the light of this information, some researchers focused on the vi-bration based techniques to prevent object slippage during the grasp. Cutkosky et al.

1_{It is worth mentioning that even though several research groups investigated the feasibility of achieving a similar human-like}

grasping performance in robotic hands, only few studies consider the contribution of the mechanical impedance of the fingers in grasp robustness to disturbances such as slippage (e.g., see [46, 47]).

(38)

1.3. Stiffness Modulation to Prevent Slippage: What Robots Can Learn from Humans

proposed a method for incipient slippage detection by sensing vibrations generated by the spread of slip zone inside the contact area while the tangential force increases [51]. A couple of accelerometers were placed near and on top of the contact region to detect vibrations due to the slippage and noise, respectively. As a commercial example, Bio-Tac fingertip sensors that are capable of sensing force, vibration and temperature, detect slippage by means of vibration. Results of this study suggest that the high-frequency spectral power from 30-200 Hz can be a reliable indicator of sliding [57]. Even though vibration based techniques are commonly used to warn the system for incipient slip-page, they are not very robust in the presence of disturbances; in particular, unexpected changes during loading or manipulation phases, may produce undesired vibrations that can compromise the efficacy of the slip detection [51].

Optical sensing has also been employed in closed loop control systems to regulate the grasping force of hand prostheses by detecting the amount of object slippage. For instance, authors in [52] utilized an optical tracking sensor together with an i-Limb prosthetic hand [50]. Experimental results showed the efficacy of the optical tracking sensor while reducing the slippage of the cylindrical object under different loading conditions. Due to its size, the sensor was placed on the palm of the hand prosthesis; in such a system the slip detection feature can only be used if the objects to be grasp are big enough and therefore can be grasped by the fingers while remaining in the visual field of the sensor. Besides, it is known that such a system does not work well for glossy and smooth surfaces.

Other avenues of research seek for appropriate reflexive strategies once the slippage is detected. In particular, Mouri et al. proposed a human-like reflexive control strat-egy while grasping objects with unknown shapes. In their study, to control the 16 DoF anthropomorphic hand, the grasp was divided in two sub-phases: grasping and with-holding. During the grasping phase, the hand was controlled with a velocity control if no contact was detected, and force control in the presence of a contact. During the withholding phase, force control was used and the Euclidean norm of the contact point velocity was added to the desired contact force to avoid slippage [58]. Authors in [59] used a prototype optical three axis tactile sensor for the analysis and detection of the normal and tangential forces in a two-finger robotic grasp. The rigid fingers’ re-pushing velocity was controlled based on the object’s classified stiffness to stabilize the grasp. In [49] an underactuated myoelectric hand prosthesis equipped with the position and force sensors was exploited to realize successful grasps for different weight of objects with various grasp types (e.g., power and pinch grasp). To achieve this, three different control strategies based on the shared autonomy between the low-level and high-level (user intention) control were proposed. Results suggested that the incorporation of the human intention into the control loop can increase the success rate of the grasping ac-tion. Alternatively, a research group [60] explored a useful relationship between the reflexive responses of humans in various conditions and features of the measured EMG signals in order to functionalize hand prostheses against unexpected disturbances. The proposed technique, however, requires feature extraction analysis and decision algo-rithm to detect the grasp reflex time, which introduce more computational load to the system.

In this study, as previously mentioned, three different reflex control strategies were implemented and experimentally evaluated. In the experimental setup (see Figure 1.13),

(39)

(a) (b)

Figure 1.13: (a) ThimbleSense addon for the Pisa/IIT SoftHand, and (b) two ThimbleSenses attached to the thumb and middle fingers of the Pisa/IIT SoftHand. The hand is mounted on the end-effector of the KUKA lightweight robotic arm.

due to the limited space in the fingertip area, small and lightweight tactile sensors were required for real-time detection of the object slippage. To achieve this, a force/torque sensor based tactile sensory system was designed and integrated into the fingertips of the robotic hand for the estimation of the tangential and normal forces and torques at the corresponding point of contact.

1.3.2 Biomechanical Control Principles in Grasping

Five naïve subjects participated in the experiment. Each subject wore two Thimble-Sense tactile sensors on the index and thumb fingertips for real-time measurements of the grasping forces during the experiments (see Figure 1.14). ThimbleSense [61] is a sensorized system that provides full-fledged measurement of force (normal and tangential at the contact point) and torque components, together with the location of contacts, and can be placed on fingertips for analysis of unconstrained grasping tasks. This is obtained by assembling a Force/Torque sensor (ATI Nano 17) between an inner and an outer shell separated by a gap. Due to the rigid coupling between the sensor’s

(40)

1.3. Stiffness Modulation to Prevent Slippage: What Robots Can Learn from Humans FDS-EDC EMG electrodes Human-finger adapter TimbleSense 0 12 F o rce [N ] 4 8 12 16 20 24 28 32 36 40 44 48 0.05 0.1 0.15 0.2 0.25 Time [sec] FD S + ED C F_t F_n

Figure 1.14: Two ThimbeSenses are worn by the human while grasping the test object. The tangential (blue, solid) and normal (red, dotted) forces at the contact point of the thumb, and the summed EMG values of the FDS and EDC muscles of a typical experiment for two repetitions of a complete sequence (grasping, lifting and putting down the experimental object) are illustrated in the top and bottom plots, respectively.

outer shell and the human finger, meaningful information can still be transmitted to the fingertip receptors.

Due to the known geometry of the external support, it is possible to obtain the po-sition of the contact centroid of the loading force, through the intrinsic tactile sensing algorithm defined by Bicchi et al. in [62]. More in general, given a surface S with an outward normal defined everywhere, and a distribution of compressive tractions applied on it, the contact centroid is defined as a point c such that a wrench exists and consists of a force directed into S applied to c and a pure torque about the contact normal. The intrinsic tactile sensing algorithm can be used to identify the contact centroid on a general surface, as long as it can be represented by a NURB (Non Uniform Rational B-Splines) parametrization: an application was provided in the Tactile Toolbox [63].

An object (Figure 1.14, on the right), which consisted of three wooden blocks that are interconnected by strings, was designed to emulate sudden grasp force variations

(41)

60mm 130mm 80mm 60mm 270mm wood wood 295 g each 640 g

Figure 1.15: Objects used in the grasping experiment. Most left object emulates unexpected mass vari-ations by interconnecting three blocks of wood using strings. The three objects on the right have different friction properties while being similar in shape and weight.

induced by the gravitational loading while lifting. Realization of a robust and reliable grasp while picking such an object requires that the grasping forces are appropriately regulated. In addition, similar objects with different surface properties (Figure 1.15, three objects on the right) were taken into account to investigate the role of surface texture (friction properties) in grasp force modulations.

Each subject was asked to pick up the object naturally and place it back ten times, using only the thumb and index finger to perform the grasp. To minimize the effect of learning, objects were sorted in a random order. During the experiments, two surface EMG sensors collected the muscle activities of a dominant finger flexor/extensor an-tagonistic pair: flexor digitorum superficialis and extensor digitorum communis. Cor-responding EMG signals were then processed (full rectified, filtered and normalized) for further analysis.

Typical results of the experiment with the interconnected three wooden blocks are illustrated in Figure 1.14. The tangential (blue, solid) and normal (red, dotted) forces at the contact point of the thumb, and the summed EMG values of the FDS and EDC muscles for two repetitions of a complete sequence (grasping, lifting and putting down the object) are illustrated in the top and the bottom plots, respectively. As observed in the plots, once a change in the gravitational loading of the object is detected, the normal forces are effectively regulated to achieve a reliable grasp. As expected, when lifting the three objects that had the same weight and shape but different surface type, the normal force exerted was highly correlated with the surface type; in particular, the highest normal forces were exerted when lifting the object that had lowest surface friction and vice versa. In accordance with [48], it was found that the safety margin in the regulation of the normal forces was kept between 16% and 42%. Such interaction forces ensure the task execution and its robustness against disturbances (e.g. object slippage) that might be caused by sudden mass variations.

On the other hand, an increase in the FDS and EDC EMG signal amplitudes can be observed in correspondence of the lifting phase (see a similar trend for all subjects in Figure 1.16). Section 1.1 showed how an increased activity of the antagonistic pair can account for an increase in the grasp stiffness. Since the limb stiffness and pose can be regulated in a decoupled way in humans [64], simultaneous increase of the interaction

(42)

1.3. Stiffness Modulation to Prevent Slippage: What Robots Can Learn from Humans 0 2 4 6 8 10 12 14 0.51 1.5 FDS+EDC Subj. 1 0 2 4 6 8 10 12 14 0 5 Force [N] Time 0 1 2 3 4 5 6 7 8 9 0.51 1.5 FDS+EDC Subj. 2 0 1 2 3 4 5 6 7 8 9 0 5 Force [N] Time 0 1 2 3 4 5 6 7 8 9 10 0.51 1.5 FDS+EDC Subj. 3 0 1 2 3 4 5 6 7 8 9 10 0 5 Force [N] Time 0 1 2 3 4 5 6 7 8 9 10 0.51 1.5 FDS+EDC Subj. 4 0 1 2 3 4 5 6 7 8 9 10 0 5 Force [N] Time 0 1 2 3 4 5 6 0.51 1.5 FDS+EDC Subj. 5 0 1 2 3 4 5 6 0 5 Force [N] Time

Figure 1.16: Example of tangential forces and summed EMG values of the FDS and EDC muscles are plotted for the five subjects. These values are recorded during the grasp of the interconnected wooden blocks.

forces as a consequence to the stiffening of the human fingers suggest that the virtual pose (equilibrium position, which is not measurable) of the finger is moved into the object. Considering a simple and static formulation of the fingertip impedance, i.e. F_n= kn∆xn, with Fn, kn and ∆xn being the normal component of the fingertip force, stiffness and displacement, if the error between the virtual and the actual fingertip pose remains zero, Fnwill not increase regardless of the change in kn.

The above observations suggest that the implementation of a similar principle in a robotic or prosthetic hand to simultaneously and instantly regulate the virtual pose and the rigidity (stiffness) of the grasp can lead to a human-like grasping performance that ensures the grasp robustness against disturbances such as slippage.

1.3.3 Robotic Implementation

The Pisa/IIT SoftHand

The Pisa/IIT SoftHand design [30] takes inspiration from neuroscience. It is well known that humans control their hands not merely by acting on each of its numer-ous degrees of freedom, but rather by coordinating and co-activating them in organized motions called synergies ( [28, 65]). Soft synergies, introduced in [29], are a more

CONTROL AND HAPTIC FEEDBACK INTERFACES FOR PROSTHETIC USE

C

ONTROL AND HAPTIC FEEDBACK INTERFACES FOR

PROSTHETIC USE

Acknowledgements

F

Summary

T

List of publications

Contents

Introduction

CHAPTER

1

Insights from Human Studies on Stiffness

Modulation Strategies for Prosthetic Control

Design

1.1

frame

F/T Sensor

F/T Sensor

F/T Sensor

F/T Sensor

kuka interface

finger-slot

finger-slot

finger-slot

contact module

contact module

contact module

104

12

0,

7

40

1.2

1.3