Designing an AR interface to improve trust in Human-Robots collaboration

(1)

DIPARTIMENTO DI INGEGNERIA DELL’ENERGIA DEI SISTEMI,

DEL TERRITORIO E DELLE COSTRUZIONI

RELAZIONE PER IL CONSEGUIMENTO DELLA

LAUREA MAGISTRALE IN INGEGNERIA GESTIONALE

Designing an Augmented Reality interface to improve trust in

Human-Robots Collaboration

RELATORI IL CANDIDATO

Prof. Ing. Gino Dini Guglielmo Bertolino

Dr. John Ahmet Erkoyuncu

Sessione di Laurea del 03/10/2018 Anno Accademico 2018 - 2019

(2)

Summary

ABSTRACT ... 4 List of Figures ... 5 Acknowledgment ... 6 Chapter 1 ... 7 1.1 Introduction ... 7 1.2 Background ... 8 Chapter 2 ... 9

2.1 What is Augmented Reality ... 9

2.2 Adoption of Augmented Reality Applications ... 10

2.2.1 Manufacturing ... 10

2.2.2 Maintenance ... 11

2.2.3 Assembly ... 12

2.2.4 Logistics ... 13

2.2.5 Retail and Marketing ... 14

2.2.6 Training and Education ... 15

2.3 How does Augmented Realty works? ... 16

Chapter 3 Augmented Reality-Human Robot Collaboration ... 17

3.1 Introduction Human-Robot Collaboration in Industry ... 17

3.2 Kind of Human-Robot Collaboration ... 18

3.2.1 POWERMATE ... 18

3.2.2 Flexible Assembly Systems through Workplace-Sharing and Time-Sharing Human-Machine Cooperation (PISA) ... 19

3.2.3 Rob@work projects ... 20

3.2.4 Co-operative Robot Assistant (CORA) ... 21

3.2.5 Lightweight robotic arm ... 22

3.2.6 Baxter ... 23

3.2.7 HRC in cellular manufacturing ... 23

3.2.8 Turtlebot ... 24

3.3 Understanding the AR Context Operating Principle about ... 25

Chapter 4. Augmented Reality for Human-Robot Collaboration ... 26

4.1 Introduction ... 26

4.2 Literature review AR – HRC ... 27

Chapter 5 Designing an AR interface to improve trust in Human-Robots collaboration ... 28

(3)

5.2.1 Psychological aspect-“Voice of Users” ... 29

5.2.2 Technical Descriptors - "Voice of the Engineer"... 30

5.3 Design of the AR-HRC architecture ... 31

Chapter 6 Validation test ... 33

6.1 Results ... 34

6.2 Discussion ... 37

6.3 Conclusion ... 38

REFERENCIES... 40

(4)

ABSTRACT

In a global, e-commerce marketplace, product customization is driven towards manufacturing flexibility. Conventional caged robots are designed for high volume and low mix production cannot always comply with the increasing low volume and high customization requirements. In this scenario, the interest in collaborative robots is growing. A critical aspect of Human-Robot Collaboration (HRC) is human trust in robots. This research focuses on increasing the human confidence and trust in robots by designing an Augmented Reality (AR) interface for HRC. The variable affecting the trust involved in HRC have been estimated. These have been utilised for designing the AR-HRC. The proposed design aims to provide situational awareness and spatial dialog. The AR-HRC developed has been tested on 15 participants which have performed a “pick-and-place” task. The results show that the utilization of AR in the proposed scenario positively affects the human trust in robot. The human-robot collaboration enhanced by AR are more natural and effective. The trust has been measured through an empirical psychometric method also presented in this thesis.

(5)

List of Figures

FIGURE 1 ”Design a new product with Augmented Reality” FIGURE 2 ”Support to Maintenance of BMW motor”

FIGURE 3 ”Digital information to Assembly an component electronic” FIGURE 4 ”indoor Navigation”

FIGURE 5 ”Interaction with virtual information in the store” FIGURE 6 ”Digital information of Anatomy ”

FIGURE 7 ”Overlay the digital information based on marker” FIGURE 8 ”Digital information retrieved by check point ” FIGURE 9 “POWERMATE”

FIGURE 10 “PISA”

FIGURE 11 “ROB@WORKS” FIGURE 12 “CORA”

FIGURE 13 “LIGHTWEIGHT ARM” FIGURE 14 “BAXTER”

FIGURE 15 “HUMAN ROBOT COLLABORATION IN THE CAGE” FIGURE 16 “TURTLEBOT”

FIGURE 17 “METHODOLOGY”

FIGURE 18 “PARTECIPANT INTERACTING WIHT THE ROBOT”

FIGURE 19 “PARTECIPANT INTERACTING WIHT THE ROBOT TROGHT AUGMENTED REALITY” FIGURE 20 “ AR-HRC trust questionnaire results“

(6)

Acknowledgment

My research is the result of an unbelievable adventure in a great team in Cranfield University. I want to say thank to my Professor Gino Dini to give me the opportunity to go and write in Cranfield University.

I would like to say thank you also to Dottor John to welcome me in their team and give me costant support to find the right direction.

The life is like a game of centimetre, I did the best and give all myself everyday to achieve one centimetre.

(7)

Chapter 1 1.1 Introduction

We are entering a new era of manufacturing; one in which mass production has given way to mass customization. A global marketplace, built on rich e-commerce platforms and make-to-order product configurations, has made it a great time to be a consumer but a challenging one to be a manufacturer. The fact is, conventional robots were designed for speed, precision and payload. They are built to do one thing at a time, in one place, repeatedly and for high volume, low mix production. Nowadays industries have the need of being more flexible [1]. Collaborative robots (cobots) could be developed specifically to address the lower volume, higher mix applications that have historically been impractical to automate. Cobots have been defined by Akella [2] as a sub-set of Intelligent Assist Devices (IADs) for “direct, physical interaction with a human operator in a shared workspace”. They are well suited for flexible manufacturing environments because they are designed to be safe to deploy around people with no guarding. They also tend to be less expensive to purchase and deploy than their caged counterparts, which means a faster return on investment for their owners. Given the growing trend, one might think that the use of robots in industry becomes increasingly popular and widespread. This brings with it the need for a careful analysis of the issues related to the safety of the workplace. The latter, in fact, will see human operators and robots working always more and more closely to each other’s [3]. Trust has been identified as a key element for successful Human-Robot collaboration (HRC) [4]. To appropriately understand the development of trust between human and robots, it is vital to effectively quantify the level of trust. Such a measurement tool would offer the opportunity for system designers to identify the key aspects to increase trust in HRC. Moreover, it is the authors’ belief that HRC system could benefit from the use of AR. The latter, in fact, can be designed to provide digital information for increased situational awareness, enable the use of natural spatial dialog, allow for multiple collaborative partners and enable local and remote collaboration [5]. Using AR to display information, such as robot state, progress and even intent, will enhance understanding, grounding, and thus collaboration. This research focuses on understanding the trust development in HRC and design and develop an AR solution for increasing it.

(8)

1.2 Background

A significant amount of assembly tasks in various manufacturing processes still require the flexibility and adaptability of the human operator [6]. In such processes, it is neither feasible nor cost-effective to introduce full automation. The manufacturing industry has shown growing interest in the concept of industrial robots working as teammates alongside human operators [7]–[10]. Considering recent technological developments, health and safety regulations have been updated to reflect that in some circumstances it is safe and viable for humans to work more closely with industrial robots [11]. Industrial HRC can enhance manufacturing efficiency and productivity since the weakness of one partner can be complemented by the strengths of the other [12]. However, the integration of humans and robots within the same workspace can be a challenge for the human factors community. For example, the installation of large assemblies requires operators to cooperate with large and high payload robots under minimized physical safeguarding [13]. One key aspect that can determine the success of a HRC system is the degree of trust of the human operator in the robotic teammate [4], [14], [15]. With the concept of industrial HRC being embraced further, trust needs to be explored in depth in order to achieve successful acceptance and use of industrial robotic teammates

The continuous increase of robot installations across different manufacturing disciplines is expected to increase the need for human and robot co-existence and collaboration. Historically, industrial robots have been used in factories as a standalone system and operating autonomously [16]. Most of the time, where robots were implemented, they were surrounded by fences and guards for safety purposes. Essentially this allowed no room for real time interaction. The increasing need for flexibility and adaptability along with the prohibitive cost for implementing full automation, the manufacturing industry has shown growing interest in the development of collaborative robots able to work alongside human operators [7], [9]. The rationale of HRC is that the weaknesses of the human operator can be complemented by the strengths of the robot and vice versa [12]. As described earlier certain manufacturing processes require the sensory skills and ability of the human worker to react to external influences, such as tolerances or process variations. Thus, the application of full automation in these types of processes is not a viable solution allowing the human operator to retain a key role. However, human operators lack accuracy, repeatability, speed and strength. Industrial robots on the other hand are very accurate and do not suffer from fatigue. Furthermore, industrial HRC can enhance employee working conditions by delegating heavy,

(9)

repetitive and sometimes dangerous tasks to the robots. Examples include instances where workers are required to perform a task within a confined space or carry out tasks which pose very high physical load.

Chapter 2 2.1 What is Augmented Reality

Augmented Reality systems combine real and virtual objects in a real. The aim of this technology is to enhance human performance providing relevant information for any specific task. Augmented Reality (AR) is a variation of Virtual Environments (VE), or Virtual Reality as it is more commonly called. The ability of Augmented Reality (AR) to overlay virtual information on top of the real world can directly impact maintainers performance through this support[0]. From helping maintainers to enhance their performance to retrieving valuable feedback, AR can close the information loop between information systems and the operations these support. Nevertheless, AR implementation has been restricted owing to the great amount of resources needed for developing software solutions. VE technologies completely immerse a user inside a synthetic environment. While immersed, the user cannot see the real world around him. In contrast, AR allows the user to see the real world, with virtual objects superimposed upon or composited with the real world. Therefore, AR supplements reality, rather than completely replacing it. Ideally, it would appear to the user that the virtual and real objects coexisted in the same space. Augmented Reality can fulfil this vision by creating an augmented workspace through inserting digital contents into the physical space where operators work. Such augmented workspace is realized by integrating the power and flexibility of computing environments with the comfort and familiarity of the traditional workspace . Augmented Reality (AR) opens a promising gate for integrating designs into their to-be-built real-world context.

(10)

2.2 Adoption of Augmented Reality Applications

There are more challenges and growing trend about AR. This technology did not find a target audience, more applications and different field could be use AR. The most important are manufacturing, maintenance, remote maintenance, assembly, training, education, logistics, game, fashion, military, tourism experience, retail and more.

2.2.1 Manufacturing

In the last decade the competitive business and manufacturing environmental, manufacturing industry is facing the constant challenge of producing innovative products at reduced time to design and time to market figure 1. The growing trend of globalized manufacturing environments requires real-time information exchanges between the various nodes in a product development life cycle, e.g., design, setup planning production scheduling, machining, assembly. Augmented Reality technology could be an innovative solution to avoid these problem to simulate and improve these manufacturing process before they are carried out[26]. Augmented Reality technology helps to fill the gap between digital product development and the operations of the equipment as information such as simulations.

(11)

2.2.2 Maintenance

The application of Augmented Reality technology in maintenance activities is increasing, showing advantages in terms of efficiency, costs and safety figure 2. It allows the reduction of intervention time and material consumption, with the consequent minimization of the maintenance costs. In addition, it has a crucial impact on the safety for the technicians while they are performing a task. The maintenance process is a sequence of elementary operations that ensure the equipment functionality (longer life cycle), the prevention of failures and the diagnosis and/or repair of equipment parts, in order to realise products with high quality and avoid any critical or unexpected situation, e.g. breakdown, where is required a downtime of the production process. In terms of manufacturing, an appropriate maintenance planning is required in order to achieve several goals, such as, quality, availability and reliability of production systems, safety on the work environment, production maximization and minimization of the maintenance costs [27].

(12)

2.2.3 Assembly

Assembly information, such as textual instruction manuals figure 3, drawings or schematics, is often detached from the equipments. The operators would usually need to alternate their attention between the assembly instructions and the parts being assembled, where the assembly instructions could be in the form of paper manual or electronic manual in computers. AR makes it possible to display assembly information in the operator’s field of views according to the situation figure 3. The operator can concentrate on the tasks at hand without having to change their heads or body positions to receive the next set of instructions [28]. The main challenge in AR-assisted assembly system is to determine when, where, and what virtual information to display in the augmented world during the assembly process.

(13)

2.2.4 Logistics

The benefits of Augmented Reality technology in the logistics field is relatively new trend. Augmented Reality could use for give information about the stock location or the better way to find the items the warehouse environmental figure 4. This is very important for the prospective and exact planning and operation of tasks such a load optimization and delivery, and is critical to providing higher levels of customer service. AR could be implemented in different front such as warehouse planning, Warehousing Operations, Transportation Optimization, Last-mile Delivery, Enhanced value-added Services[29].

(14)

2.2.5 Retail and Marketing

In the retail industry, AR can bring major benefits both in the online and brick-and-mortar sectors by enabling the interaction with virtual objects and enhancing the shopping experience with capabilities offered by the Internet, respectively. There are many advantages in using AR in the retail industry [30]: It can improve the conversion rates and reduces returns for clothing stores via the use of virtual fitting rooms figure 5. Such rooms allow customers to sample products online as clothes are automatically overlaid on the consumer`s real-time video image through their webcam. In addition, the clothing retailer, American Apparel, is also adopting AR technology with the aim of bringing the online experience offline. AR allows customers to try a product before they buy it with the use of a 3D preview. With the help of AR technology, additional information can be displayed about products in order to enrich the shopping experience, enable customers to search for nearby deals and attract them inside a store.

(15)

2.2.6 Training and Education

Augmented Reality is also a highly interactive technology which makes it suitable for the concept of "learning by doing" figure 6 . The interaction possibilities range from the basic interaction with virtual objects tocomplex interactive features. ]). Augmented Reality is not only a means to educate and train people, but also to entertain while acquiring new knowledge. Students usually find concept acquisition more interesting when using this kind of systems [31]. AR technology has a fast learning curve, which means that users can start utilizing the applications with very little prior information.

(16)

2.3 How does Augmented Realty works?

There are two primary types of AR implementations:

 Marker Based

 Marker-Less

Marker-based implementation figure 7 utilizes some type of image such as a QR/2D code to produce a result when it is sensed by a reader, typically a camera on a cell phone.

Figure 7 ”Overlay the digital information based on marker” Figure 8 ”Digital information retrieved by check point”

Marker-less AR figure 8 is often more reliant on the capabilities of the device being used such as the GPS location, velocity meter, etc. It may also be referred to as Location-based or Position-based AR. Both Marker-based and Marker-less AR require ” AR specific software or browsers” to function. Marker-based AR is currently the most prevalent and easiest to accomplish. While Marker-less AR is emerging, it is currently rather limited due to sensor accuracy (i.e. GPS accuracy anywhere between 10 – 50 meters), service limits (i.e. indoors vs outdoors), bandwidth requirements (4G is not a reality in all places nor can the devices currently in existence handle it), and power pulls on the devices.

(17)

Chapter 3 Augmented Reality-Human Robot Collaboration

3.1 Introduction Human-Robot Collaboration in Industry

The most of organization are in need to achieve superior product performance, enhance the production rates while reducing costs to stay competitive and meet the markets demand. The automated e smart system will be a great solution. We are entering a new era of manufacturing – one in which mass production has given way to mass customization. A global marketplace, built on rich e-commerce platforms and make-to-order product configurations, has made it a great time to be a consumer – but a challenging one to be a manufacturer. As global wages rise, companies are realizing that the financial value of off-shoring their labour has diminished greatly, while the inherent risks to product quality and supply chain disruptions have remained very much intact. Manufacturers have sought to automate their high mix operations – but traditional industrial robots are poorly suited for this task. The fact is, conventional robots were designed for speed, precision and payload. Not flexibility. They were built to do one thing at a time, in one place, repeatedly. And for high volume, low mix production, they have excelled at this for decades – and still do. Automation has not always been able to successfully replace the human input needed for many complex tasks. A possible solution is the implementation of closer Human-Robot Collaborative working.

(18)

3.2 Kind of Human-Robot Collaboration

There are many type of Robot Industrial Robot Collaborative and Robot Humanoid. The operator could interact with the robot in online mode and offline mode. Generally, if the interaction is offline mode means the operator cannot stay near the robot because it is not safe. The robot does a specific task inside a protection like a cage. Otherwise, If the interaction is online mode the operator can communicate with the robot when it works.

3.2.1 POWERMATE

PowerMate figure 9 is an intuitive robotic assistant utilised to assist operators in assembly and handling tasks. This is an stationary Robot and it has physical contact with the human operator. The interaction occurs through a force-torque-sensor enabling the robot to move when the operator applies force. The main purpose of this system is to assist the assembly of heavy parts. Initially the robot has to grip the heavy component and bring it to the human worker. During the collaborative part the human worker can ensure the final component has been precisely assembled. When the collaborative task is finished, the robot moves the completed item to a separate area and the next cycle begins [32].

(19)

3.2.2 Flexible Assembly Systems through Workplace-Sharing and Time-Sharing

Human-Machine Cooperation (PISA)

The aim of this project is to support human workers with powerful tools in order to complete a task as well as keep in the 23 loop. The focus of the project is to develop novel intelligent assist systems, provide planning tools for their integration and to achieve reusability of assembly equipment. One of the sub-projects of this initiation is the development of a humanoid service robot figure 10 to be used in human workplaces [33].

Figure 10 “PISA”

(20)

3.2.3 Rob@work projects

The aim of rob@work figure 11 is to assist production workers in fetch and carry tasks, assembly and tool handling tasks as well as participating in manual arc welding. It has the ability to navigate autonomously while the human worker commands and supervises the robot. Further on this work, a second variant of rob@work was developed in 2008. Rob@work 2, on the other hand, is developed as static machining equipment which can be positioned in a variety of workplaces [34].

(21)

3.2.4 Co-operative Robot Assistant (CORA)

CORA figure 12 can be fixed on a table, and its purpose is to physically interact with a human worker standing across the table. CORA consists of a seven DOFs manipulator arm in combination with a two DOF stereo camera mounted on its head. In addition, it includes an interface for the human worker to interact and provide corrections to its end-effector according to the needs of the task [35].

(22)

3.2.5 Lightweight robotic arm

This humanoid arm figure 13 is designed for co-operation with human workers in unstructured environments. In addition, the humanoid construction of this arm, when compared to an industrial robotic arm, offers intrinsic safety due to its light-weight structure. Potential industrial applications can be assembly processes where accuracy is not of prime importance, applications where the robot operates within the immediate workspace of the human worker and possibly in direct physical co-operation with them and mobile service robotics applications [36]

(23)

3.2.6 Baxter

Baxter figure 14 is an easy to use interactive robot and was designed to handle light payloads and operate alongside human operators without being physically safeguarded. Baxter was designed to execute a variety of manufacturing and productions tasks, while at the same time it can be aware of its environment allowing it automatically adjust to changes. Furthermore, It features advanced force sensing technology, back-drivable motors, and a moderate velocity that aim to reduce the likelihood and impact of a collision [37]

Figure 14 “BAXTER”

3.2.7 HRC in cellular manufacturing

This type of environmental was designed to execute repetitive task. The operator cannot stay close the robot. That means the operator must to work on offline mode figure 15. Every time open the cage the Robot will stop their task [38]

(24)

3.2.8 Turtlebot

Turtlebot figure 16 was designed to assist the operator in a common assembly task. It is able to move around the ground floor. It can recognize the environment through the odometry map. Generally that collaborative robot could be used for the pick and place task [39]

(25)

3.3 Understanding the AR Context

Operating Principle about

AR is made possible by performing four basic and distinct tasks, and combining the output in a useful way.

1. Scene capture: First, the reality that should be augmented is captured using either a video-capture device such as a camera, or a see-through device such as a head-mounted display.

2. Scene identification: Secondly, the captured reality must be scanned to define the exact position where the virtual content should be embedded. This position could be identified either by markers (visual tags) or by tracking technologies such as GPS, sensors, infrared, or laser.

3. Scene processing: As the scene becomes clearly recognized and identified, the corresponding virtual content is requested, typically from the Internet or from any kind of database.

4. Scene visualization: Finally, the AR system produces a mixed image of the real space as well as the virtual content.

(26)

Chapter 4. Augmented Reality for Human-Robot Collaboration

4.1 Introduction

Human-Robot Collaboration system would benefit from the use of Augmented Reality. It can provide digital information for increased situation awareness, enable the use of natural spatial dialog, allow for multiple collaborative partners and enable local and remote collaboration. The result would be a system that allows natural and effective communication and collaboration. AR is an optimal method of displaying information. AR could be showed through user tests that spatial displays in a wearable computing environment were more intuitive and resulted in significantly increased performance. Using AR to display information, such as robot state, progress and even intent, will enhance understanding, grounding, and thus collaboration. AR is an optimal method of displaying information. AR could be showed through user tests that spatial displays in a wearable computing environment were more intuitive and resulted in significantly increased performance. Using AR to display information, such as robot state, progress and even intent, will enhance understanding, grounding, and thus collaboration.

(27)

4.2 Literature review AR – HRC

The application of Augmented reality in industry has seen a rapid increase in the last decade. It has been applied in different maintenance and manufacturing scenarios: the aviation industry, plant maintenance, mechanical maintenance, customer technology, nuclear industry and remote applications [18], [19]. Both the hardware and the development platform utilised in the development and the utilisation of AR can vary widely. The main hardware utilised are: Head Mounted Displays (HMD), Hand Held Displays (HHD), desktop pc, projectors, haptic devices and other sensors. The main development platforms are: mid/low-level languages, libraries of functions, Software Development Kits (SDK), game engines and 3D cad modelling platforms. Other features that characterise an AR system are, among the others: the object tracking method, the user interface and the authoring solutions [18]. It is not always easy to identify the requirements of the AR system that has to be designed, for this reason Palmarini et al [20] proposes a survey based method to drive the designers in choosing the right AR features for a specific application. Several studies have explored the utilisation of AR in HRC. Bischoff [21] research in 2004, proposed AR for overlaying the robot coordinates system over the real environment in order to de-skill the robot programming and operation. Fang [22], in 2009, developed RPAR-II (Robot Programming AR) for assisting users in robot programming both on-site and off-site. The virtual robot is overlaid on the real one allowing interaction and path planning on the real working environment. More recently Andresson [23] developed the AR-Enhanced Multimodal Robot-Programming Toolbox (AR-EMRPT) for programming industrial robots by demonstration, instructions, observation and context techniques. The goal was to improve training, programming, maintenance and monitoring of robots in both the training facility with a physical robot and within a complete virtual environment. The common driver in all the AR-HRC designs for supporting industrial applications is always to de-skill the Human-Robot operations while improving the situational awareness. Humans, in fact, determine their trust in cobots by observing their characteristics, performance and ways of accomplishing a task [4].

(28)

Chapter 5 Designing an AR interface to improve trust in

Human-Robots collaboration

5.1 METHODOLOGY

The methodology figure 17 utilised for assessing the trust involved in the HRC and designing the AR application for enhancing trust in HRC. Firstly, the variables affecting the trust involved in the HRC have been estimated and weighted. It has been done through literature and industrial experts’ knowledge. These have been used to build two rating scale tables named: “Voice of the Users” (Table 2) and “Voice of Engineers” (Table 3). Then was designed an “Experimental Design” to met the Requirements and Need of Users to designed an AR Interface to improve the trust in the HRC. After has been a data collection and data analysis for the validation.

(29)

5.2.1 Psychological aspect-“Voice of Users”

It considers the main aspects in AR-HRC in terms of Human aspect, Robot aspect and Augmented Reality information aspect. A scale from 1 to 5 (low relevance – high relevance) has been used for rating the relevance of the “elements” (robot, human and AR) on their respective “trust related themes” (Performance, safety, digital information). The “Voice of the Users” table aims to find which one is the most relevant component in the AR-HRC collaboration. It has been filled independently by 15 participants between students and technicians.

Element Trust-related themes importance of trust-related themes

Robot Performance 1-5

Human Safety 1-5

Augmented Reality Digital information in advance 1-5

(30)

5.2.2 Technical Descriptors - "Voice of the Engineer"

The second table has been named “Voice of Engineers”. It is a relationship matrix for determining the relationship between trust-related themes and the technical HRC aspects. Relationships can either be weak, moderate, or strong or respectively carry a numeric value of 1, 3 or 9. The participants were 5 engineers with heterogeneous background. These have been reunited and after an open discussion on the HRC topic, have filled the proposed rating Table 3 together.

Percen tal of im po rta nc e tr us t-rel at ed Mo tion ar m wi th ou t cage Pa th wi th ou t cage Gr_ipp er AR -H RC C on tex t-aware Tas k safety Mo tion ar m Pa_th Sou_nd Inform at io n Man age the r ob ot wi th d ev ic e Performance 0.29 3 9 3 9 9 Safety 0.37 9 9 9 9 9 9 9 9 3 3 Digital information in advance (AR) 0.34 9 9 9 9 9 9 9 Importance of the relation 4.20 5.34 4.20 9.00 9.00 7.26 6.39 6.39 4.17 4.17 Percental importance 6.67% 3.44% 6.67% 14.30% 14.30% 11.53% 10.15% 10.15% 6.63% 6.87% Ranking 4 6 4 1 1 2 3 3 5 3

Table 3 "Voice of the Engineer"

Based on these results, it has been possible to design the test. The test consisted of carrying out a maintenance operation utilising the AR-HRC system designed and developed and answering the Likert scaled questionnaire (Table 1) designed for assessing the trust involved in the operation. Participants were informed regarding their right to withdraw and anonymity. For the validation on AR-HRC scenario they have been requested to share the same workspace and collaborate with an industrial robot to complete a task. The operation training could be used as a strategy to raise operators’ awareness regarding the ability and limitations of the robot and assist matching operators’ perceptions with the system’s actual capabilities.

(31)

5.3 Design of the AR-HRC architecture

This section describes the design of AR in the HRC. The system has been developed with Unity3D and Vuforia SDK. The robot utilised is the Turtelbot (programmed in ROS). The requirements of the system have been captured from the “Voice of the Users” and “Voice of Engineers” tables described in the previous section. The AR-HRC has been developed to provide accurate context awareness to the technician by giving him the information about the robot movements in advance. The system utilised Vuforia SDK for recognising the marker attached to the robot hence align the virtual robot with the real one. The movements of the robot were captured from the ROS and transferred to the hardware application in advance. These movements have been applied to the virtual robot aligned to the real one therefore providing the animation of the full robot plan on the real working environment. In Figure 18 it is possible to see one of the participants interacting with the robot. The marker utilised for recognising the robot and aligning the virtual robot real time is placed on the basement of the Turtlebot. The product to be maintained is on the table. Once the participant is introduced to the test, he is provided with the tablet for experiencing the AR-HRC.

(32)

In Figure 19, is shown the AR visualisation of the robot designed for improving the trust in HRC. The virtual Turtlebot is overlaid over the real one through a hand-held device. The electric board that is pick-and-placed by the robot is placed on the top of the black toolbox. The product where the electric board has to be assembled is on the table on the right. The user can start the animation of the cobot to understand what movements it is going to do once start. In this case, the arm would pick up the electric board and place it in the product.

Figure 19 “PARTECIPANT INTERACTING WITH THE ROBOT TROUGHT AUGMENTED REALITY”

In summary, the design of the AR-HRC developed utilised: • Hardware: TABLET

• Development Platform: Unity 3D + Vuforia SDK • Tracking method: Marker-based

• Interaction method: Dynamic 2D/3D

(33)

Chapter 6 Validation test

During the validation test, the participant would receive digital information about the task involving the robot. More specifically, the information is overlaid on the robot and it consists of a virtual animation of the robot movements. These are proposed before the robot starts moving. It aims to give an accurate idea of the robot spatial movements. The task chosen for the test was a “pick and place” of an electronic card. The robot was very close to operator and close to work station. Once the maintenance task is completed, the participants were asked to complete the questionnaire. This has been used for measuring the participants trust in the AR-HRC system developed.

Nr.Questions

1 I felt safe interacting with the robot

2 I trusted that the robot was safe to cooperate with me

3 The way the robot moved made me uncomfortable

4 Knowing the spatial movements in advance made me

feel more comfortable

5 I felt more safe because I knew in advance the path

movements

6 I felt more safe because I knew in advance the arm

movements

7 I prefer to know the movements of the robot in

advance

8 I prefer to see the complete animation of the robot

and then start the procedure

9 I prefer to see the animation of the robot 5 seconds in

advance

Table.” Questionnaire utilised for understanding trust involved in AR-HRC. The answer has to be as Likert Scale: from strongly disagree to strongly agree (5 levels in total).”

(34)

6.1 Results

The first result is the estimation of the variables affecting the trust in HRC. Tese avo teen estimate by

1. Asking to 15 participants which aspects among the one listed in Table 2 (“Voice of the Users”) were more important for them. Table 2 reports the average results of the Likert scaled tables filled by the 15 participants. A scale from 1 to 5 (low relevance – high relevance) has been used for rating the relevance of the “elements” (robot, human and AR) on their respective “trust related themes” (Performance, safety, digital information). Table 2 “Percentages of

relevance” are reported on the first row of Table 3 and have been used for weighting the results of the “Voice of the Engineers” table.

.

Element (1) Trust-related themes (2) Average relevance (2 on 1)

Percentage of relevance (2 on 1)

Robot Performance 3.40 0.29

Human Safety 4.30 0.37

Augmented Reality Digital Information in advance

3.93 0.34

(35)

2. Asking to 5 engineers with exposure to HRC which of the aspects listed in Table 3 (“Voice of Engineers”) were more relevant for them. Table 3 is the relationship matrix filled by 5 Engineers together after an open discussion on HRC. The trust-related themes relationship with the HRC technical aspects have been scored as follows: 1 for weak relationship, 3 for moderate relationship and 9 for strong relationship. The outcome has been that AR needed to provide context awareness in order to improve the human perception of safety enhancing his trust in HRC.

Per fo rman ce Saf ety A R info rmatio n Relev an ce o f _Relatio n % o f Relev an ce Ran ki ng Percentage relevance 0.29 0.37 0.34

Arm Movement (no cage)

3 9 1 4.2 6.67% 4

Path Movement (no cage) 9 9 1 5.34 3.44% 6 Gripper 3 9 1 4.2 6.67% 4 Context-awareness 9 9 9 9.0 14.30% 1 Task Safety 1 9 9 7.26 11.53% 2 Arm Movement 1 9 9 6.39 10.15% 3 Path Movement 1 9 9 6.39 10.15% 3 Audial information 1 3 9 4.17 6.63% 5

Robot Tablet Controls 1 3 9 4.17 6.87% 3

(36)

The second result is the trust involved in the designed and tested AR-HRC scenario.

The AR-HRC system design and development has been described before. The pick-and place test has been performed and the participants have been asked to answer the Likert scaled questionnaire in Table 1. A total of 15 participants between engineering students and university staff members (10 male/5 female) took part in the study. The average age was 27.2 (min 22, max 33). Results are shown in Figure 20.

Figure 20 “ AR-HRC trust questionnaire results“

The results show that in average the trust in the AR-HRC system developed is above 3. This means that the utilization of the proposed AR design for understanding the cobot movements in the real environment has positively affected the trust of the participants in collaborating with the cobot.

(37)

6.2 Discussion

The first result estimated that human-safety and context-awareness to be the key variables affecting trust in HRC. The results from this study are in agreement with the literature [5][3]. The context-awareness which in this case is the workspace awareness defined as “the up-to-the-minute knowledge of other participants interactions with the shaped workspace” [24], is essential for providing the human technician with the confidence necessary for working in safety. The second result showed an average trust in the AR-HRC scenario proposed always above 3 on a 1 to 5 scale. Even if it has not been compared with the same scenario without the utilization of AR, it still provides a valuable indication of the trust involved. This can be used for future comparison with different AR support designs for the same pick-and-place task even the same AR support design for a different task. This study has been carried out utilizing turtlebot, an intrinsically safe robot due to its low weight and strength. Anyway, I emphasized its dangerousness to the participants. Therefore, confide I was confident that the same study done utilizing an industrial cobot (heavy and strong) would lead to similar results. Furthermore, the proposed design for AR used in HRC has demonstrated to be suitable to raise trust. It will be interesting to explore further studies on how the trust level is different among different backgrounds and levels of experience. Further research could help with understanding how AR can help to grow or reduce the trust over time.

(38)

6.3 Conclusion

This project focused on understanding the variables affecting the trust involved in HRC and designing an AR-HRC system for increasing the trust of the human operators in collaborating with robots within an industrial environment. Trust plays a key role in human and robot interactions, particularly when the decisions we make depend on the people when they work very close with the robot. However, little research has been done at understanding trust development in industrial human-robot collaboration (HRC). With industrial robots becoming increasingly integrated into production lines as a means for enhancing productivity and quality, it will not be long before close proximity industrial HRC becomes a viable concept. Since trust is a multidimensional construct and heavily dependent on the context, it is vital to understand how trust develops when shop floor workers interact with industrial robots. These provide a flexible solution for the increasing industrial customization needs. Their implementation is still detained by the lack of human trust in collaborating with robots. Context-awareness and human safety have been found to be one of the key variables affecting the trust in HRC. An AR-HRC system designed to provide context-awareness for improving human safety has been developed and tested on a pick-and-place task. The AR-HRC shows the robot movements by overlaying a virtual animation of the cobot on the real environment, real time. The operator can see the operations that the cobot will carry out once started, in advance. This solution has been tested and has shown a great potential in enhancing the participants trust in the turtlebot utilized for the test. Future studies should compare the results of this study with different scenarios and AR-HRC systems. The scenario should include a real industrial cobot which, would be more effective in understanding the human trust in cobots. The AR system for supporting HRC should take advantage of recent advancements in head-mounted displays and provide a more immersive virtual scenario for improving the workspace awareness. The main aspect to should be considered is the flexibility, a successful collaboration with human and robot could be developed specifically to address the lower volume. The high mix that have been impracticable to automate. The successful collaboration means also move the robot without move any cage. The environmental will be more flexible and the operator feel more safe when they works.

(39)

The architecture to improve trust on Human Robot Collaboration has been validate but Augmented Reality is still a prototype. This technology is not ready to be implemented as a technology solution for the industry. The main problem are light condition to recognize the marker and relative position to detected that. However, big company invested in Augmented Reality Technology as GOOGLE, APPLE, DHL etc.… but the application is still in a prototype environmental.

(40)

REFERENCIES

[0]

(Ronald T. Azuma “A Survey of Augmented Reality”) [1]

J. Krüger, T. K. Lien, and A. Verl, “Cooperation of human and machines in assembly lines,” CIRP Ann. - Manuf. Technol., vol. 58 no. 2, pp. 628–646, 2009.

[2]

P. Akella, M. Peshkin, E. Colgate, W. Wannasuphoprasit, N. Nagesht, J. Wells ’, S. Holland ’, T. Pearson ’, and B. Peacockt, “Cobots for the automobile assembly line,” Proc. 1999 IEEE Int. Conf. Robot. Autom., no. May, pp. 728–733, 1999.

[3]

J. Fryman and B. Matthias, “Safety of Industrial Robots: From Conventional to Collaborative Applications,” Robot. Proc. Robot. 2012; 7th Ger. Conf., pp. 1–5, 2012.

[4]

A. Freedy, E. DeVisser, G. Weltman, and N. Coeyman,

“Measurement of trust in human-robot collaboration,” in 2007 International Symposium on Collaborative Technologies and Systems, 2007, pp. 106–114.

[5]

S. Green, M. Billinghurst, X. Chen, and G. Chase, “Human-Robot Collaboration: A Literature Review and Augmented Reality Approach in Design,” Int. J. Adv. Robot. Syst., vol. 5, no. 1, pp. 1– 18, 2007.

[6]

Z. Ding and B. Hon, “Constraints analysis and evaluation of

manual assembly,” CIRP Ann. - Manuf. Technol., vol. 62, no. 1, pp. 1–4, 2013.

(41)

M. Hägele, W. Schaaf, and E. Helms, “Robot assistants at manual workplaces - Effective co-operation and safety aspects,” Proc. 33rd Int. Symp. Robot., p. 6, 2002.

[8]

R. D. Schraft, C. Meyer, C. Parlitz, and E. Helms, “PowerMate - A Safe and Intuitive Robot Assistant for Handling and Assembly Tasks,” in ICRA, 2005.

[9]

A. De Santis, B. Siciliano, A. De Luca, and A. Bicchi, “An atlas of physical human-robot interaction,” Mech. Mach. Theory, vol. 43, no. 3, pp. 253–270, 2008.

[10]

V. V. Unhelkar, H. C. Siu, and J. a. Shah, “Comparative performance of human and mobile robotic assistants in

collaborative fetch-and-deliver tasks,” Proc. 2014 ACM/IEEE Int. Conf. Human-robot Interact. - HRI ’14, pp. 82–89, 2014.

[11] I

SO, “Robots and robotic devices-safety requirements for industrial robots,” IEEE Trans. Human-Machine Syst., vol. 1, 2011.

[12]

D. Bortot, M. Born, and K. Bengler, “Directly or on Detours?: How Should Industrial Robots Approximate Humans?,” in Proceedings of the 8th ACM/IEEE International Conference on Human-robot Interaction, 2013, pp. 89–90.

[13]

M. Walton, P. Webb, and M. Poad, “Applying a Concept for Robot-Human Cooperation to Aerospace Equipping Processes,” in SAE Technical Paper, 2011.

[14]

R. Parasuraman and V. Riley, “Humans and Automation: Use, Misuse, Disuse, Abuse,” Hum. Factors, vol. 39, no. 2, pp. 230–253,

(42)

1997. [15]

J. Y. C. Chen and M. J. Barnes, “Human #x2013;Agent Teaming for Multirobot Control: A Review of Human Factors Issues,” IEEE Trans. Human-Machine Syst., vol. 44, no. 1, pp. 13–29, Feb. 2014. [16]

A. M. Zanchettin, L. Bascetta, and P. Rocco, “Acceptability of robotic manipulators in shared working environments through human-like redundancy resolution,” Appl. Ergon., vol. 44, no. 6, pp. 982–989, 2013.

[17]

J. E. Colgate, J. Edward, M. A. Peshkin, and W. Wannasuphoprasit, “Cobots: Robots For Collaboration With Human Operators.” 1996. [18]

R. Palmarini, J. A. Erkoyuncu, R. Roy, and H. Torabmostaedi, “A systematic review of augmented reality applications in

maintenance,” Robotics and Computer-Integrated Manufacturing. 2018.

[19]

S. K. Ong, M. L. Yuan, and A. Y. C. Nee, “Augmented reality

applications in manufacturing: a survey,” Int. J. Prod. Res., vol. 46, no. 10, pp. 2707–2742, 2008.

[20]

R. Palmarini, J. A. Erkoyuncu, and R. Roy, “An innovative process to select Augmented Reality ( AR ) technology for maintenance,” 2016.

[21]

R. Bischoff and A. Kazi, “Perspectives on augmented reality based human-robot interaction with industrial robots,” 2004 IEEE/RSJ Int. Conf. Intell. Robot. Syst. (IEEE Cat. No.04CH37566), vol. 4, pp. 3226–3231, 2004.

(43)

[22]

H. Fang, S. K. Ong, and A. Y.-C. Nee, “Robot Programming Using Augmented Reality,” 2009 Int. Conf. CyberWorlds, pp. 13–20, 2009.

[23]

N. Andersson, A. Argyrou, F. Nägele, F. Ubis, U. E. Campos, M. O. De Zarate, and R. Wilterdink, “AR-Enhanced Human-Robot- Interaction - Methodologies, Algorithms, Tools,” Procedia CIRP, vol. 44, pp. 193–198, 2016.

[24]

J. Drury, J. Scholtz, and H. Yanco, “Awareness in human-robot interactions,” Proc. IEEE Conf. Syst. Man Cybern., 2003. [25] (Wang & Dunston 2006a)

[26] ( B. Morkos, J. Taiber, J. Summers, L. Mears, G. Fadel and T. Rilka, “Mobile devices within manufacturing environments)

[27] (S. Enderson S. FEINER “Exploring the benefits of Augmented Reality documentation for maintenance and repaire ”)

[28] (Hou, X. Wang, L. Bernold and P. E. Love, “Using Animated Augmented Reality to Cognitively Guide Assembly”)

[29] (L. . Kückelhaus, M., Terhoeven, M. Key Logistics Trends in Life Sciences 2020+. A DHL perspective on how to .prepare for future growth)

[30] (D. Gaioshko, “10 ways how augmented reality can help retailers) [31] ( C.DEDE “Immersive interface for engagement and learning” )

[32] (PowerMate-A Safe and Intuitive Robot Assistant for Handling and Assembly Tasks)

[33] (FLEXIBLE ASSEMBLY SYSTEMS THROUGH WORKPLACE-SHARING AND TIME-SHARING HUMAN-MACHINE COOPERATION – PISA)

[34] (https://www.care-o-bot.de/en/rob-work.html)

[35] (Cooperative Robot Assistant (CoRA) For Human Environments)

[36] (https://www.kinovarobotics.com/en/products/robotic-arm-series/ultra-lightweight-robotic-arm)

[37] (https://www.rethinkrobotics.com/baxter/)

[38] (Human-robot collaboration in cellular manufacturing: Design and development) [39] (https://en.wikipedia.org/wiki/TurtleBot)

(44)

APPENDICE “QUESTIONNAIRE”

PART A-1

Please describe the task in terms of Augmented Reality in Human Robot Collaboration: It took a board and piched up it

PART B

Participant age: ( 23 ) Participant gender: ( F )

Previous experience with robot: NONE

Previous experience with AR: NONE

I felt safe interacting with the robot

1

χ Strongly

disagree Disagree neutral Agree Strongly agree

I trusted that the robot was safe to cooperate with

2

χ Strongly

The way the robot moved made me uncomfortable

3

χ Strongly

Are you more comfortable if you knew in advance digital information?

4 χ

(45)

disagree

I felt more safe because I knew in advance the paths

5

χ Strongly

I felt more safe because I knew in advance the arm movements

6

χ Strongly

I prefer knew animation and robot movement in same time

7

χ Strongly

I prefer knew animation and then the robot start

8

χ Strongly

I prefer knew animation to give me 5 second in advance and then the robot start

9

χ Strongly

Use the scale from 1-5 from low importance to high importance

Element Trust-related themes Score (1-5)

Robot Performance 3

Human Perception Personal Safety 5

Augmented Reality Digital information in

(46)

PART A-2

Please describe the task in terms of Augmented Reality in Human Robot Collaboration: THE TASK CONSISTED IN PICKING UP A BOARD FROM THE ROBOT

PART B

1

χ Strongly

2

χ Strongly

3

χ Strongly

4 χ

(47)

disagree

5

χ Strongly

6

χ Strongly

7

χ χ

Strongly

8

χ Strongly

9

χ Strongly

Robot Performance

4

Human Perception Personal Safety

5

(48)

PART A-3

Please describe the task in terms of Augmented Reality in Human Robot Collaboration: COLLABORATIVE ROBOT

PART B

Participant age: ( 26 ) Participant gender: ( M ) Previous experience with robot: NONE

1

χ Strongly

2

χ Strongly

3

χ Strongly

(49)

Strongly

5

χ Strongly

6

χ Strongly

7

χ Strongly

8

χ Strongly

9

χ Strongly

Robot Performance 4

(50)

PART A-4

Please describe the task in terms of Augmented Reality in Human Robot Collaboration:

PART B

1

χ Strongly

2

χ Strongly

3

χ Strongly

(51)

4

χ Strongly

5

χ Strongly

6

χ Strongly

7

χ Strongly

8

χ Strongly

9

χ Strongly

Robot Performance 3

(52)

advance 2

PART A-5

Please describe the task in terms of Augmented Reality in Human Robot Collaboration: PICKUP WITH COLLABORATIVE ROBOT AND AUGMENTED REALITY TECH

PART B

1

χ Strongly

2

χ Strongly

3

χ Strongly

(53)

4

χ Strongly

5

χ Strongly

6

χ Strongly

7

χ Strongly

8

χ Strongly

9

χ Strongly

Robot Performance 4

(54)

advance 5

PART A-6

THE TASK IS AN ASSEMBLY OPERATION WITH THE SUPPORT THE COLLABARATIVE ROBOT AND AUGMENTES REALITY

PART B

Previous experience with AR: RESEARCHER

1

χ Strongly

2

χ Strongly

(55)

3

χ Strongly

4

χ Strongly

5

χ Strongly

6

χ Strongly

7

χ Strongly

8

χ Strongly

9

χ Strongly

Robot Performance 2

(56)

advance 4

PART A-7

Please describe the task in terms of Augmented Reality in Human Robot Collaboration: THE ROBOT MOVED AND PICKED AN ITEM WICH EAS MOVED TO ANOTHER LOCATION

PART B

1

χ Strongly

2

χ Strongly

(57)

3

χ Strongly

4

χ Strongly

5

χ Strongly

6

χ Strongly

7

χ Strongly

8

χ Strongly

9

χ Strongly

Robot Performance 3

(58)

advance 4

PART A-8

USE THE ROBOT TO RICEVE INFORMATION ABOUT THE ROBOT’S TASK AND MAKE HIM DO SOME COLLABORATIVE TASK

PART B

Participant age: ( 26) Participant gender: ( M )

Previous experience with robot: PROGRAMMING/ AUTOMATION

1

χ Strongly

2

χ Strongly

(59)

3

χ Strongly

4

χ Strongly

5

χ Strongly

6

χ Strongly

7

χ Strongly

8

χ Strongly

9

χ Strongly

(60)

advance 4

PART A-9

Please describe the task in terms of Augmented Reality in Human Robot Collaboration: IT TOOK A BOARD AND PICKUP WITH COLLABORATIVE ROBOT

PART B

1

χ Strongly

2 χ

(61)

disagree

3

χ Strongly

4

χ Strongly

5

χ Strongly

6

χ Strongly

7

χ Strongly

8

χ Strongly

9

χ Strongly

(62)

advance 3

PART A-10

Please describe the task in terms of Augmented Reality in Human Robot Collaboration: COLLABORATIVE HUMAN-ROBOT

PART B

1

χ Strongly

2 χ