DEVELOPING AN AUGMENTED REALITY BASED TRAINING TOOLKIT FOR MANUFACTURING CHERRY PICKERS

(1)

i

D

IPARTIMENTO DI

I

NGEGNERIA DELL

’E

NERGIA DEI

S

ISTEMI

,

DEL

T

ERRITORIO E DELLE

C

OSTRUZIONI

RELAZIONE PER IL CONSEGUIMENTO DELLA LAUREA MAGISTRALE IN INGEGNERIA GESTIONALE

Developing an Augmented Reality based Training

Toolkit for Manufacturing Cherry Pickers

RELATORI IL CANDIDATO

Prof. Ing. Gino Dini Francesca Ferrati

Dipartimento di Ingegneria Civile e Industriale ferrati.francesca@gmail.com

Dr. John Erkoyuncu

Through-Life Engineering Services Centre, Cranfield

Dr. Samuel Court

Through-Life Engineering Services Centre, Cranfield

Sessione di Laurea del 03/10/2018 Anno Accademico 2017/2018

(2)

SOMMARIO

Il progetto è consistito nello sviluppo e nella validazione di un dimostratore di Realtà Aumentata per testare la fattibilità di introdurre tale tecnologia presso lo Sponsor di progetto, Niftylift, per migliorarne il processo di formazione.

Il progetto è stato dapprima definito con il cliente; in seguito, la rassegna bibliografica, le osservazioni ed i colloqui hanno portato all’identificazione dei requisiti. Il dimostratore è stato quindi sviluppato, validato e i suoi benefici valutati.

Il progetto si è focalizzato sul processo di assemblaggio di tubi idraulici alla relativa valvola. L’hardware selezionato è stato Microsoft HoloLens, e i software Unity e Vuforia. Lo strumento fornisce istruzioni sequenziali; immagini e marcatori permettono il riconoscimento degli oggetti e l’associazione delle informazioni.

I benefici sono stati valutati in termini di tempo e tasso di errore, paragonando due gruppi di utilizzatori: il primo ha eseguito il processo in maniera tradizionale, il secondo tramite AR. Sono stati rilevati miglioramenti a seguito dell’introduzione dell’AR, concludendo la validità della soluzione per soddisfare i bisogni di Niftylift.

ABSTRACT

The project was carried out in collaboration with Niftylift. It aimed at developing and validating an Augmented Reality demonstrator to test the feasibility of introducing the technology at Niftylift and to enhance its training process.

First, the project was defined in collaboration with the Client. Then, literature review was combined with observations and interviews to identify requirements. The tool was developed and validated, and its benefits were assessed.

The tool focused on covering the assembly of hydraulic hoses to the relative valve. Microsoft HoloLens was the hardware, while Unity and Vuforia were the software. The tool provides sequential augmented instructions. Images and markers are used to recognise elements and to overlay information on them.

Benefits were assessed comparing time and error rate performances among two groups: the first performed the process following SOPs, while the second group used the AR tool. Results showed improvements when introducing AR. It was concluded that AR can be a feasible and valid solution to address Niftylift needs.

(3)

iii Keywords:

(4)

ACKNOWLEDGEMENTS

I would like to thank my supervisors, John Erkoyuncu, Gino Dini and Samuel Court for their support and guidance during the development of the project. I would also like to thank Niftylift for giving me the opportunity of working with them and my industrial supervisors: Steve Beckwith and Andrew Sharman.

A special thank goes to my flatmates and friends, Federica, Arianna and Valentina, for sharing with me this year and for growing up together.

I thank Giulio, who has believed in me and has always been on my side.

A special thank goes to my parents and my sisters, Ilaria and Arianna, which encouraged and supported me during these years from far away.

(5)

v

LIST OF FIGURES

Figure 1 Cherry picker ... 1

Figure 2 Project scope ... 3

Figure 3 Training's key elements ... 6

Figure 4 Methodology ... 11

Figure 5 Training-bay ... 18

Figure 6 Valve assembled ... 18

Figure 7 Hydraulic valve to be assembled ... 19

Figure 8 Process map ... 20

Figure 9 Application conceptual map ... 26

Figure 10 Process breakdown ... 28

Figure 11 Solution logic ... 29

Figure 12 "Main interface" - example ... 30

Figure 13 "Instructions interface” - example ... 31

Figure 14 "Side interface" - example ... 31

Figure 15 AR preliminary experiments - Time results ... 34

Figure 16 Valve stand ... 36

Figure 18 Experiments with SOPs - time results ... 37

Figure 17 Experiments with AR - time results ... 37

Figure 19 Average SOPs' and AR's time performances ... 39

Figure 20 Cylinder target in Vuforia ... 45

Figure A-1 Hydraulic valve v Figure B-1 Tool's board pointers - example ... ix

Figure B-2 Expected outcome of the task - example ... x

Figure B-3 Cap indication - example ... x

Figure B-4 Incorrect hose label example ... xi

Figure B-5 Correct hose label - example ... xi

(8)

Figure B-7 Instructions - example ... xii

Figure B-8 Red mark on conjunction ... xii

Figure B-9 Final acknowledgement - example ... xiii

(9)

ix

LIST OF TABLES

Table 1 Requirements identification ... 13

Table 2: Training bay and production lines influential factors ... 17

Table 3 Solution requirements ... 21

Table 4 Hardware selection criteria ... 24

Table 5 Preliminary tests' results - example ... 32

Table 6 Preliminary tests' errors - example ... 33

Table 7 Preliminary tests' help requests - example ... 33

Table 8 Company preliminary tests feedback ... 33

Table 9 Experiments with AR ... 36

Table 10 Experiments with SOPs ... 36

Table 11 Time results benefits assessment ... 38

Table 12 KPIs results and benefits assessment ... 38

Table 14 AR and assembly experiences compared to first hose fitting times ... 39

Table 15 Performance's comparison among AR groups ... 40

Table 16 AR experience questionnaire results ... 41

Table D-1 Preliminary test's results xv Table D-2 Preliminary experiments results ... xvii

(10)

LIST OF ABBREVIATIONS

2D Two-dimensional

3D Three-dimensional

AR Augmented Reality

FOV Field of View HHD Hand Held Devices HMD Head Mounted Displays

IMA Industrial Maintenance and Assembly KPIs Key Performance Indicators

KSA Knowledge, Skills and Attitudes QR Code Quick Response Code

SOPs Standard Operational Procedures TNA Training Need Analysis

UK United Kingdom

(11)

1

1 Introduction

1.1 Background

Niftylift is one of the world's leading manufacturers of mobile elevated work platforms, also known as cherry pickers, shown in Figure 1.

The Company was founded in 1985, and since then continuously increased its production and its network, being now present in over 40 countries worldwide.

Niftylift places innovation at the hearth of its core beliefs. Creative passion, research & development investments and optimised design processes are key Company drivers, which lead Niftylift to research regularly new and innovative solutions to improve business performances.

The project was developed in collaboration with the Milton Keynes manufacturing plant. There, high-value products are assembled, taking up to 48 hours to complete each machine. Low performances in the training process were previously identified, raising the possibility of introducing a technologically advanced solution to improve its efficiency. Critical elements are a significant time that expert technicians spend training a trainee and relative costs. Costs are directly linked to the amount of time required to assemble cherry pickers.

Among possible solutions, a valid option to meet Niftylift needs was identified in Augmented Reality (AR). AR is a powerful technology that allows human-computer

(12)

interaction, displaying holograms on the real world. It finds several applications within the industrial and manufacturing fields, such as supporting training processes thanks to the possibility of overlaying information directly on the machine on which operators work. (Webel, Bockholt, & Engelke, 2011)(Nee, Ong, Chryssolouris, & Mourtzis, 2012) Full guidance can be provided independently from the trainer, and operators can learn while performing operations. Since AR leaves the world intact and does not prevent the user to see it, risks related to the manufacturing environment can be significantly reduced compared to Virtual Reality (VR) solutions.

1.2 Research problem definition

The research project intends to develop an AR demonstrator that can test the applicability of AR within Niftylift context and its effectiveness in addressing training issues.

The work intends to study the Company context and environment, to identify its specific requirements and to develop a demonstrator suited for their manufacturing processes. Developing a customised solution is needed, to assess the applicability of AR within Niftylift industrial environment, which cannot be evinced solely from previous applications. The focus, when developing the tool, is on speeding up Niftylift training process, without affecting its quality.

1.3 Project aim, objectives and scope

1.3.1 Aim

The project aims at developing and validating an Augmented Reality demonstrator, to test the feasibility of introducing the technology at Niftylift and to enhance its training process on assembly operations, reducing employees’ training time without affecting the learning level.

1.3.2 Objectives

The objectives are:

▪ Conduct a literature review of existing AR training applications; ▪ Study Niftylift context to identify application’s requirements;

(13)

3

▪ Development of the AR demonstrator to assess the applicability of AR at Niftylift; ▪ Validate the tool and assess benefits;

1.3.3 Scope

The project scope, shown in Figure 2, was restricted to the following elements:

▪ Replicating the current training process in the AR tool, without re-designing it, to obtain comparable performances’ results;

▪ Considering a sub-part of the process, due to the length of the whole assembly process;

▪ Assessing benefits from time, error rate and trainees’ learning level point of view, according to the key performance indicators (KPIs) defined by the Company. Costs and other indicators are therefore not in scope;

▪ The application environment is Niftylift Milton Keynes site;

▪ Including images and visual representations, explicitly required to support not native-born English people;

▪ Not including vocal aids, due to the noisy environment;

▪ Considering assembly operations and not disassembling operations;

(14)

2 Literature review

2.1 Augmented reality

Augmented Reality tools allow human-computer interaction, overlaying information, such as images, videos or audios on the real world. Information is computer-generated and can include more senses than visual, providing touch or haptic sensations. (Nee et al., 2012) (Syberfeldt, Holm, Danielsson, Wang, & Brewster, 2016).

AR is a variation of Virtual Reality (VR); VR technologies completely immerse a user in an artificial environment, preventing him/her from seeing the real world. On the contrary, AR superimposes virtual objects, enhancing users’ perception and displaying additional information. Users can combine augmented information with practice, a key element to enhance learning. (Pathomaree & Charoenseang, 2005) (Kipper, G. & Rampolla, J., 2012) AR offers advantages compared to VR, which requires modelling the whole working environment, a complex and time-consuming activity. (Ong, Yuan, & Nee, 2008)

AR finds several applications in the manufacturing field, providing context-based information. It can support industrial training, projecting assembling or disassembling instructions directly on to the machines on which operators work. (Webel et al., 2011)(Nee et al., 2012)

2.1.1 Hardware devices, visual presentation and tracking systems

There is three major hardware to implement AR technologies: Head-Mounted displays (HMD), Hand-Held devices (HHD) and Projectors, each having advantages and drawbacks. (Syberfeldt et al., 2016) The device’s choice depends on the context of the application (working environment, users and assembly processes), and relative requirements. As examples, HHD present the inconvenience of keeping hands busy, while some HMD might cause dizziness, especially after long usage; projectors usually have a fixed installation, making them less flexible. (Chimienti, Iliano, Dassisti, Dini, & Failli, 2010)(Nee et al., 2012)

(15)

5

There are three ways in which AR can be visually displayed: video see-through, optical

see-through and projective displays. With the video see-through technique, a video

replaces reality and AR is overlaid upon it: the user’s view is completely digital. Conversely, the real world is left untouched with the optical see-through and AR is overlaid on it. With projective displays, AR is cast directly on real objects. (Krevelen & Poelman, 2010)

Visual information displayed can be texts, virtual elements (such as arrows, images, videos, icons or symbols) or 3D object’s models. (Dalle Mura, Dini, & Failli, 2016) The tracking and registering systems enable the alignment between the user view and the world. Marker-based and marker-less solutions are available, potentially making it possible to introduce AR within shop-floor contexts. Since AR systems are capable of recognising objects and components (either through feature-based or model-based tracking methods), users can potentially move in the AR environment and manipulate objects intuitively and naturally. Marker-less solutions are suitable for non-prepared environments when the work is performed on a new machine each time. (Wang, Ong, & Nee, 2016)(Nee et al., 2012)(Nee & Ong, 2013)

2.2 Training and tacit knowledge transfer

Training can be defined as the process that allows beginners or learners to acquire skills, through a well-defined procedure, becoming experts. It aims at providing a set of specific knowledge, skills and attitudes (KSAs) and at allowing trainees to generalize these skills. (Borsci, Lawson, & Broome, 2015)(Salas, Tannenbaum, Kraiger, & Smith-Jentsch, 2012)

Industrial maintenance and assembly (IMA) training combine two elements: the basic understanding of how the machines to be assembled work, and the learning of the sequence of steps to perform the job. Procedural skills represent the capacity of a person to form a good mental representation of how to “perform each of the steps of a task and the correct order to perform them, which is reflected in their hierarchic organisation”. (Gavish, Gutierrez, Webel, & Rodriguez, 2011) They depend on the ability

(16)

of a person to memorise procedures (procedural memory) and on his/her ability to create a correct mental model of the machine, of its part and of the equipment required to perform the tasks. (Gavish, Gutierrez, Webel, & Rodriguez, 2011)

Four elements should be considered to provide effective training, displayed in Figure 3.

First, trainers should convey information (instructions and examples). Then, a demonstration of the desired behaviour should be provided. Trainees should then have the opportunity to perform what they observed and, last, timely diagnostic feedback and the possibility to recover from errors should be provided. (Salas et al., 2012) Actions before, during and after a training process influence the training outcome. At first, a Training Need Analysis (TNA) need to be developed. It allows to define what needs to be trained (Job-task analysis), the type of organisation that is delivering the

(17)

7

training (Organizational analysis) and for whom the training is (Person analysis). During the delivery of the training, an appropriate instructional strategy needs to be followed, according to trainees’ characteristics and the nature of the work. Finally, debriefing and training evaluation has to be performed, to define opportunities for improvement. (Salas et al., 2012)

During the training delivery, both technical and tacit knowledge are transferred. Tacit knowledge was defined by Howells as: “Non-codified, disembodied know-how that is acquired via the informal take-up of learned behaviour and procedures.” (Howells, 1996)

Literature studies argue the importance of transmitting tacit knowledge to new employees to obtain a competitive advantage. Investments in IT are risky since they only allow communicating explicit knowledge, often available or replicable by competitors. Experiences, practical and action-oriented knowledge is harder to identify and therefore imitate. (Johannessen, Olaisen, & Olsen, 2001)

2.3 Augmented Reality enabled training

AR has been proven a valid solution for training within certain manufacturing environments: it is highly effective, time-efficient and assures trainees learn the required capacities. Furthermore, it allows providing feedback, a key factor in the learning process. (Wang et al., 2016)

Compared to traditional training, AR avoids the inconvenience of information detached from the equipment, which forces trainees to switch their attention between the instructions and the subassembly. (Ong, Yuan, & Nee, 2008)

AR-based training systems enable the combination of real experiences with virtual instructions and guidance. Directions are provided through interactive checklists, to assure each step is followed. The system can potentially recognise steps carried out incorrectly and can alert the user, preventing him/her from continuing the procedure unless the error is fixed. (Tatić & Tešić, 2017) Suggestions and further instructions can

(18)

be provided if the tasks are performed in a wrong way, for instance depicting the goal position of the component. (Pathomaree & Charoenseang, 2005)

Different levels of guidance, according to the trainees’ level, can be provided using

Indirect Visual Aids. The user can decide whether to see further information, whose

availability is represented by an annotation or an icon. This choice is not available when using Direct Visual Aids: 3D animations or other information are directly superimposed on the product. (Webel et al., 2013)

2.3.1 Benefits and positive effects

AR systems advantages include the possibility of freeing-up expert technicians, thanks to an autonomous training system; variations in the input (teachings) are reduced, consequently diminishing variations in learnt procedures. The possibility of customising the system according to different starting levels and personal characteristics can positively affect motivation and therefore performances; learning by doing approach is enabled, increasing the quality of skills acquired thanks to interactions. The overall understanding of tasks and procedures raises, compared to traditional training. Real-time data on trainees’ performances can be collected, taking immediate and long-term corrective actions; paperwork can be eliminated, and updates can be easily made.

(Borsci et al., 2015)

Previous applications showed that completion times could be reduced up to 85%, comparing 2D assembly task with and without AR guidance and 96.2% for 3D assembly tasks. (Pathomaree & Charoenseang, 2005) On the contrary, other experiments found no statistically relevant difference comparing the two form of training delivery: traditional and AR training brought to the same learning curve for the users. (Peniche, Diaz, Trefftz, & Paramo, 2012)

2.3.2 Guidelines

(19)

9

Enriched information has a positive influence on trainees and their understanding of tasks: providing explanations at a sub-task level, instead of only at an activity level, reduces the number of non-solved errors and should be encouraged. (Gavish et al., 2011) Thus, the trainee can create a mental model of the task, thanks to the visualisation of context. Pre and post- conditions of the sub-assembly help operators to understand and visualise the job, better than instructions. (Webel et al., 2011)

Observational learning was proved to play an important role: learning time reduction was observed, while no difference was registered on trainee’s performances and should, therefore, be properly integrated into the AR training. (Gavish et al., 2011)

According to the training phase of the user, different levels of guidance should be provided to decrease the potential danger of users becoming dependent on virtual information. Diminishing visual aids increases the difficulties trainees face and decreases the possibility of users not being able of performing tasks autonomously. In some occasions, providing an excessive amount of information can be frustrating for the operator. (Webel et al., 2013)(Syberfeldt et al., 2016) Using too many guidance aids can negatively affect performances when users replicate tasks independently: additional guidance should be pondered carefully. Providing them in a controlled way, nonetheless, do not impede learning. (Gavish et al., 2011)

Information can be displayed to the users in two different ways: through cognitive

fidelity or physical fidelity. The former focuses more on the conceptual model of the

task, while the latter is more specific to the singular machine or tasks. These two techniques should be merged to maximise procedural skills acquisition. (Gavish et al., 2011)

2.4 Research gap identification

Literature widely covers previous AR applications for training processes. Nonetheless, any publication was found on applications developed for manufacturers of mobile elevated work platforms. The assembly processes and components are specific to the field, and they need to be studied and explored to understand if AR can support and

(20)

improve such processes. Furthermore, training depends significantly on the human factor and the context of application; a research study is required to develop an effective AR solution for the manual assembly operations performed at Niftylift.

(21)

11

3 Methodology

3.1 Overview

The following methodology was applied to achieve the aim of the study. (Standard, 2011) The project was divided into five phases, shown in Figure 4:

▪ Project and problem definition ▪ Solution requirements identification ▪ Solution design

▪ Solution development

▪ Tool’s validation and benefits assessment

Figure 4 Methodology

3.2 Phases

3.2.1 Project and problem definition

Aim and objectives of the research project were defined in collaboration with the Company through a meeting with the Operations Director and the Production Manager. During the meeting, training key performance indicators were decided and communicated by the Sponsor, according to their priorities. The scope was agreed according to time constraints through a second meeting with the Production Manager. Concurrently, a literature review was conducted on Augmented Reality and existing AR applications for training purposes. Previous implementations allowed understanding

(22)

the potentiality of the technology for training and different approaches. Hardware and software solutions and their advantages and disadvantages were studied.

Researches were also conducted on training and its best practices, due to the absence of structured procedures at Niftylift, to define what needed to be included in the AR tool.

3.2.2 Solution requirements identification

Interviews and observations at the Company shop-floor allowed performing a context analysis and gathering data on training delivery methods. Three days’ visits were carried out, interviewing the Production Manager, two Line Managers, four expert technicians and two new employees. Four meetings were carried out with the Production Manager, while with the other subjects one meeting was made. Environment, processes and trainers were analysed. Interviews were conducted orally and then written on paper. Visits’ minutes can be found in Appendix A.

Questions to the management allowed understanding how and where training is delivered, and which knowledge the Company transfers to new employees. Questions to expert technicians allowed understanding their experience as trainers, the procedures they followed and what they believed was important to transmit to new employees.

Observations concerned training on the assembly lines to document technical and tacit knowledge transfer, and Company conventions.

Possible environments for the solution implementation were identified and compared using Company’s and training’s needs as drivers: not affecting production rates and long-term applicability.

In relation to the aim and scope, a sub-process of the assembly procedure was considered. The criteria were to choose a process that could teach procedures applicable to more production lines, to maximise the usefulness of the demonstrator, and whose components could be brought to Cranfield.

Standard Operational Procedures (SOPs) documents were collected, and a video of the assembly process was recorded to draw the process map and to reproduce each step and relative guidance in the tool.

(23)

13

The tool’s requirements, that are independent of the technology used, were identified from the interviews with the management and literature review’s guidelines. Functional requirements stated what the tool had to do, in relation with the purpose of the application itself, while other requirements were defined according to the final users’ needs.

Table 1provides an overview of the requirements and relative measurement criteria.

Table 1 Requirements identification

Requirements Way of measurement

Functional requirements:

Complete and trainer independent guidance to operators.

Number of help request during the process.

Feedback provision. Number of errors detected after the process completion.

Knowledge transfer. Assessment questionnaire results. Faster learning process. Time to perform the assembly process. Other requirements:

Intuitive and usable tool. Number of help requests related to the tool.

Precise and accurate information displayed.

Number of times operators ask for clarifications.

Possibility to work comfortably. Qualitative questionnaire on AR experience.

3.2.3 Solution design

Technological tools suitable to implement the requirements were identified. The choice was driven by three criteria: tools’ advantages and drawbacks, integration between hardware and software and tools’ availability at Cranfield.

(24)

The conceptual map of the system was drawn, detailing its inputs and outputs, its elements and their interaction. The application’s logic design aimed at maximising the learning for users and reducing the learning time.

The process was broken down into sub-processes, which, in turn, were broken down into steps. For each step, guidance and relative supporting elements were identified (instructions, texts, images and animations). Guidance is provided in sequential order: information is conveyed through text, and the desired behaviour is displayed through animations or images. Time to practice the operation is then allowed, and the user can decide whether to see the instructions again or to proceed forward. Finally, feedback is provided through the representation of the process completed, allowing users to check the work.

Augmented Reality features were introduced for critical steps, where the technology could add value: when the user needed to be guided and when checks had to be performed.

Data collection for the KPIs were included through timers and other counters. Three user interfaces, used along the whole process, were also designed.

3.2.4 Solution development

The solution was developed by transferring the conceptual elements and the logical flow into Unity, using C# as a programming language.

AR functionalities were introduced through Vuforia. During the tool development, trials were made for the objects, images and targets recognition. Working with elements that were new to Vuforia, it was not possible to know prior which solution would have worked better.

The application development required progressive improvements. It was first deployed in the hardware device and tested at Cranfield laboratory with students, who provided feedback. A preliminary validation session was carried out at the Company too. Users first followed the tutorial on the HoloLens that teaches gestures and then performed the process with the AR application. Feedback was gathered to improve the application before proceeding with the final validation.

(25)

15

3.2.5 Tool validation and benefits assessment

Final validation experiments were run at Cranfield Laboratory, comparing the performances of the groups. A stand that could hold the component in the right position was designed.

The first group was asked to perform the process following the SOPs, while the second one the AR instructions. The groups were defined according to the following criteria, to avoid the influence of other variables on the outcome of the analysis: experience with assembly operations, experience with AR and age.

Quantitative data relative to the KPIs were collected through observations and the AR tool. A test to assess the learning level and a questionnaire to gather qualitative feedback on the AR experience were developed.

(26)

4 Solution requirements identification

4.1 Context analysis

Interviews’ questions concerned training procedures in place and training locations. It was asked how new employees are introduced to the job, for how long they are trained, how the training level is assessed, the criteria to decide where they are trained and so on. Documents on visits’ minutes can be found in Appendix A.

Answers lead to the following understandings. Training, for each working station, lasts approximately 40 hours on the shop-floor: one expert technician teaches procedures, technical skills, how to use SOPs books and work autonomously.

New operators can be trained in three locations: on a stand-alone working station (training-bay), if they have little experience, on the Sub-MEC station, if they have partial experience or directly on the production lines if they have experience.

Guidance is provided by the SOPs, while the trainer is available to answer questions. Trainers might not have experience as trainers, and they do not follow written procedures. They explain how to work, rather than the sequence of steps detailed on the SOPs, according to their own experience. Working techniques and Company practices are shared in an informal way through trainers, and they might vary from operator to operator. Once the training is completed, a test takes place to assess the learning level.

4.1.1 Environment and sub-process selection

The Sub-MEC station was not a suitable environment to introduce the tool since operators do not familiarize with the machines’ assembly operations.

Table 2 summarises the characteristics of the training bay and production lines considered to decide where to implement the tool.

(27)

17

(28)

It was agreed with the Production Manager that the training-bay, shown in Figure 5, best suited the AR introduction.

The sub-process to include in the demonstrator was the assembly of hydraulic hoses to the relative valve. Pictures of the valve are displayed in Figure 6 and Figure 7.

Figure 5 Training-bay

(29)

19

Figure 7 Hydraulic valve to be assembled

The choice was driven by two factors: the possibility of teaching a procedure applicable to all the production lines and the possibility of bringing the components to Cranfield Laboratory.

Hydraulic valves are present on each machine, and they only vary in size and number of hoses, but the principle to fit hoses remains the same. The components can be assembled also without the whole chassis, which was critical to run experiments at the Laboratory.

It was agreed with the production manager that this step was a critical one to be thought to operators and therefore was agreed to be suitable for the tool development.

4.2 Process map and description

(30)

(31)

21

First, the operator picks the tools on the shadow board. Each shadow board within the factory has the same tools’ disposition. After, the operator brings the tools nearby the working area and starts the assembly. 11 hoses have to be fitted to the valve and to be tied together. The hoses’ fitting order is important, to avoid difficulties in accessing at the back joints later. The operator removes a cap from the joint and picks the hose to fit. Before fitting the first hose, operators need to calibrate the torque wrench. He/she then half-screws the hose manually and then torques it. The operator must assure that the tool clicks; he/she can help himself/herself with a spanner. He/she then red marks the conjunction according to the Company’s colour code.

The same operations are performed iteratively for each hose. Hoses are then tied: two of them on the left side and all the other ones on the right side. The torque wrench needs to be wind back to zero and put back to the shadow board with all the other tools. 4.3 Requirements

AR tool requirements and relative ways of measurement are displayed in Table 3.

Table 3 Solution requirements

General requirements: Detailed requirements: Way of

measurement: Functional requirements

Complete and trainer-independent guidance to assemble hydraulic hoses to the relative valve.

Provision of text instructions. Number of help requests during the process due to lack of instructions.

Provision of visual aids.

Provision of how to perform operations (examples).

Provision of Company conventions information.

Provision of explanations of the actions.

(32)

Feedback provision. Checks on the hoses choice to avoid wrong hose fitting.

Number of errors detected after the

process was

completed. Provision of representation of the

final state of the assembly to allow users’ check.

Knowledge transfer. How to calibrate a torque wrench. Questionnaire results. How to fit hoses.

How to torque wrench hoses. Tools’ location on the shadow board.

Practices to be respected for company conventions (e.g., colour codes).

Providing suggestions for better work (fit hoses at the back first). Teaching the importance of preventive and final checks. Precise and accurate

information display.

No possibility to misunderstand where to pick tools.

Number of help request during the process due to unclear instructions. No possibility to misunderstand

where to fit the hose.

No possibility to misunderstand if the hose was the correct one. Non-functional requirements

(33)

23

Faster learning process. Provision of visual representations to enhance the understanding of the task.

Time to perform the assembly process.

Reduction of the number of instructions for similar operations.

Possibility to resume instructions. Intuitive and easy-to-use

application.

Short initial training. Number of help request related to the AR tool’s use.

Intuitive user interface.

Possibility to work comfortably.

No discomfort (e.g., dizziness). Qualitative questionnaire. No interferences with the work.

5 AR-based training solution

5.1 Hardware and software selection

5.1.1 Hardware selection

Both HMDs and HHDs satisfied functional requirements. Criteria to choose the hardware device were driven by KPIs and literature. (Palmarini, Erkoyuncu, & Roy, 2017) They are summarized in Table 4, were the symbol ✓ signifies that the device can satisfy the requirements, while the symbol X that it cannot meet them.

(34)

Table 4 Hardware selection criteria

HMDs provide information in front of the user, who does not have to switch between the instructions and the working area, reducing the time and work interferences. Training for HMDs is performed only once, while switching between the working area and the tablet affects the whole process time; Head-Mounted-Displays were therefore selected, because more suitable.

The choice among HMDs solutions was constrained by the time-boundaries of the project. Hardware available at Cranfield University was considered, selecting Microsoft HoloLens. It is the most technologically advanced device, providing the highest

(35)

25

resolution for information display. Their battery-life (5 hours) allows their usage for a whole shift, if recharged, for instance during the lunch-break. The tracking system is integrated into the HoloLens and does not need initial calibrations. The heaviness of the tool is of slightly more than 0.5 kg, such that it can be used for more hours consecutive. HoloLens is perceived as a comfortable tool by some users, which could wear them for many hours without experiencing any headache or discomfort. On the contrary, some other users find it difficult to use it and work with it for a long usage.

5.1.2 Software selection

Unity and Vuforia were chosen as software since they provide all the required functionalities and they are fully integrated with HoloLens. Unity allows to develop the User Interface and to code the behaviour of the system. Vuforia is the market leader for objects, images and targets recognition, enabling the Augmented Reality functionalities, displaying context-based information.

5.2 Solution design

5.2.1 System conceptual map

The map in Figure 9details the elements of the AR application (hardware and software) and their interactions. Inputs and outputs are also represented.

(36)

(37)

27 The inputs of the system are:

• Work instructions and relative guidance;

• Software and hardware to develop AR applications; • The external world, to trigger context-based information. Within the system, there are three main elements:

• Unity, for tool development;

• Vuforia, for the external world recognition;

• HoloLens, for the two-way communication with the external world and the user. The outputs of the system are:

• Step-by-step guidance to the user; • Feedback on operations;

• Data collection.

Unity can use Vuforia functionalities to determine the application’s behaviour; projects developed in Unity can be deployed directly on HoloLens through Visual Studio solutions. HoloLens allows interactions with the external world and the user: the cameras acquire triggers such as gestures, gazes or markers, and projectors show holograms to the user, such as panels, 3D models or animations.

5.2.2 The tool

As shown in Figure 10, the process was divided into 15 sub-processes: tools’ picking, each hose fitting (11), hoses tie and tools’ replacement on the shadow board. For each step, instructions’ images and information were defined.

(38)

Figure 10 Process breakdown

Figure 11 shows the instructions’ sequence logic.

First, a picture of the outcome is displayed. When the operator is ready to work, he can press the button and instructions are provided.

There are three possible ways of moving forward: a timer, if no interaction with the external environment is required, the Vuforia recognition, when overlaid information is displayed and the “Resume Instructions” button. After, the user has the empty field of view (FOV) to work without visual interferences; two side buttons are available: “Tap for instructions”, to go back to the same set of instructions, and “Finished”, to proceed further. The next step’s instructions are then displayed. At the end of the sub-process, an image of the work completed is displayed, to allow user checking if the task was carried out correctly. When all the steps of a sub-process are completed, the system proceeds to the next sub-process.

Instructions are progressively reduced for similar or identical operations, to enhance the learning process. First, they are fully displayed, then direct information is reduced and from the third time on they are provided with fewer details. Full instructions are still always available under request.

(39)

29

Full process description can be found in Appendix B.

Future state representation Instructions Empty FOV Instructions Empty FOV Final acknowledgement

Sub-process 1

Iterations

Sub-process 2

Step 1 Step n

(40)

5.2.3 Vuforia functionalities

Vuforia is used in three phases:

• Tools’ shadow board: holograms point at the tools on the board, aiding users at finding them and increasing their capability of memorising the location.

• Hoses QRs: before fitting the hoses, the user checks if they are correct thanks to a virtual label that appears when showing the QR to the cameras. This functionality prevents from screwing the wrong hose and emphasizes the importance of preventive checks.

• Valve QR: it triggers animations showing how to perform the process.

5.2.4 User interface

In the application, there are three user interfaces:

• The “Main interface” is used at the beginning and at the end of each sub-process. It shows the sub-process completed and allows acknowledgement. An example is displayed in Figure 12.

Figure 12 "Main interface" - example

(41)

31

Figure 13 "Instructions interface” - example

• The “Side interface”, showed in Figure 14, leaves the Field of View empty.

(42)

Each of the interfaces has the “Time-out” button in the top left corner. When pressed, timers are stopped, and the “Restart” button appears, in the middle of the panel. This allows operators to have a break if they need, without affecting the performance time. The panel follows the user on its horizontal movements.

5.3 Tool development

5.3.1 Tool’s preliminary tests

Table 5 shows an example of the preliminary tests’ results and consequent improvements on the tool. Full tables can be found in Appendix D.

All users were Cranfield students aged between 20 and 30 years old. None of them had previous experience with AR or with assembly operations.

Table 5 Preliminary tests' results - example

Also, errors and the requests for help from users were used for the tool’s improvements. Table 6 and Table 7 show examples from one user.

(43)

33

Table 6 Preliminary tests' errors - example

Table 7 Preliminary tests' help requests - example

Company preliminary test results are shown in Table 8.

Table 8 Company preliminary tests feedback

Timing Feedback Improvements

40 minutes Improve the hoses recognition. Changed the cylinder QRs of the hoses with images QRs, so they are more easily recognisable. Did not appreciate the rotation of

the panel with the user.

Removed the panel rotation (kept the horizontal movement). Difficulties in wearing the

HoloLens and in familiarising with the tool.

Tutorial provided by Microsoft on wearing HoloLens was provided. Development of a pre-training tutorial.

(44)

Observations during the preliminary tests showed that people had difficulties in familiarizing with HoloLens and with the application, particularly for the interaction between the virtual and the physical world. This issue, probably due to the purely virtual “Learn gestures” tutorial, also arose at the Company.

Also, data, displayed in Figure 15, showed a significant time difference between the first hose fitting and the following ones, either due to lack of familiarisation or to the learning curve.

Therefore, a pre-training application was developed to allow users familiarize with the tool and the logic of the application.

Figure 15 AR preliminary experiments - Time results

0 2 4 6 8 10 12 14 16 Tim e [m in ]

AR preliminary experiments - Time results

(45)

35

6 Tool validation and benefits assessment

6.1 Tool validation

6.1.1 Experiments KPIs & groups’ definition

Data concerning KPIs were collected: the whole process time, the tools’ picking time, users’ errors rate and the number of help requests were counted.

Time for the “Learning gestures” tutorial (approximately 5 minutes) and for the pre-training application (3 minutes) were not considered. Their impact on time performances would not have provided a realistic scenario since they are undertaken only once and should be considered when comparing the whole training process. Questionnaires to assess the learning level and the AR experience can be found in Appendix C.

Table 9 and Table 10 show the two users’ groups. Factors influencing performances for assembly operations were considered, among which experience was the most relevant. Other factors, such as motivation, tiredness, stress and language difficulties were not considered, assuming that the students that volunteered were motivated, not in a stressful situation and no tired, since experiments were scheduled according to users’ preferences. Language barriers were avoided by explaining the specific terminologies to each user at the beginning and thanks to the visual aids specifically introduced into the tool. Allocations were made according to users’ preferences, when not contrasting with having evenly distributed groups. All users were Cranfield students with an engineering background.

Information about experience with AR was irrelevant for the second group, while it was valuable to compare performances within the group that performed the process with the AR.

(46)

Table 9 Experiments with AR

Table 10 Experiments with SOPs

Figure 16 shows the valve-stand designed to hold the valve, replacing the chassis.

(47)

37

6.1.2 Validation results and benefits assessment

6.1.2.1 Quantitative data analysis

Figure 18 and Figure 17 show the time performances of AR users and SOPs users.

0 1 2 3 4 5 6 7 8 9 Tim e [m in ]

User 1 User 2 User 3 User 4 User 5

Figure 17 Experiments with SOPs - time results Figure 18 Experiments with AR - time results

0 1 2 3 4 5 6 7 8 9 10 Tim e [m in ]

(48)

Table 11 summarises average time performances and improvements registered.

Table 11 Time results benefits assessment

Users following AR instructions, on average, performed the process with 22% of time less (8 minutes). Also, time to pick the tools was reduced, by 26 %. Thus, the tool brought an overall improvement on the time needed to perform the process and complete the tasks.

Table 12 shows other KPIs results and relative improvements.

Table 12 KPIs results and benefits assessment

What stood out from data is an improvement of average performances. The error rate and the number of help requests for the process decreased, while the questionnaire results improved. Looking at single data, it was noticed that SOPs users’ average questionnaire results were greatly influenced by one user that performed way worst than the others (3 correct answers out of 7), while other users answered correctly to either 6 or 7 questions.

Thus, due to the limited amount of data, it was not possible to obtain significant average values; more users should be tested to confirm the benefits found.

(49)

39

Figure 19 shows the average assembly time for each step of both groups.

The graph shows that AR required less assembly time for each fitting, except for the first and the second hose. Main differences were registered for the first hose and for the third hose fitting times.

The higher time for third hose assembly for SOPs’ users was explained through observations during the validation. Users had difficulties in fitting hoses at the back joint because of the sequence they followed when fitting them, not detailed in the SOPs. To understand the causes of the higher average time required to fit the first hose, information on AR experiences were analysed, as shown in Table 13, to check if there was any correlation between the two factors.

Table 13 AR and assembly experiences compared to first hose fitting times

User AR experience First hose assembly time

User 1 Partial 7.13 min

User 2 Yes 6.3 min

User 3 Partial 5.35 min

User 4 Yes 9.23 min

User 5 No 9.38 min

Figure 19 Average SOPs' and AR's time performances

00 01 02 03 04 05 06 07 08 Tim e [m in] SOPs AR

(50)

No relationship was noticed between users with AR experience and time to fit the first hose. The few data available do not allow to draw a full picture and to study deeper a possible correlation.

Reasons for the higher time are probably related to the AR tool itself since detailed information and step-by-step guidance are provided at the beginning. Next steps, on the contrary, are less time demanding thanks to grouped instructions.

Average performances were also compared between the preliminary tests’ group and the AR final validation group. Results are shown in Table 14.

Table 14 Performance's comparison among AR groups

Performances increased significantly. In the second group, more users were experienced with AR and those who didn't become familiar with it through the pre-tutorial. These elements, and the other improvements consequent to the feedback, were key factors to see benefits from the technology.

Detailed information on each user performance are provided in Appendix E. 6.1.2.2 Qualitative data analysis

Results of the questionnaire on the AR experience are displayed in Table 15. Detailed information can be found in Appendix E.

(51)

41

Table 15 AR experience questionnaire results

On average, people did not experience discomfort or a headache due to the HoloLens and the AR experience was evaluated positively. Also, the learning level was perceived to be high.

Other qualitative questions were asked, and on average, the worst thing was clicking on buttons, while the user interface (images and animations) and the hoses recognition feature were considered the best things. Most of the people (7 out of 10) believed that AR could enable a completely independent training process.

Requirements stated at the beginning of the project can therefore be considered satisfied.

(52)

7 Discussion

7.1 Results discussion

7.1.1 Validation’s results

Validation’s results showed improvements for training delivered through the AR tool compared to the traditional way. Questionnaire results showed that the learning level improved too. However, when evaluating the data, the following elements should be considered.

A limited amount of data was collected due to time constraints. The relevance of the improvements should be proved by a greater number of users, to obtain significant values in terms of average. Indeed, with such a small sample of data, the average can be greatly influenced also by one single value. Also, correlations among factors (such as previous experiences) and performances could be studied deeper, avoiding the influence of users’ personal characteristics on results.

Total times were considered to assess performances. However, the average time to assemble the first two hoses was shorter for SOPs users, which might be due to the number of instructions provided at first. It is worth considering improving the tool, reducing the detail level and the number of acknowledgement required at first. It was decided not to change this aspect to emphasize the importance of checks, but the learning level could be assessed and compared in the two scenarios and decisions could be taken accordingly.

It is worth noting that the chosen process was a simple one. From literature, main benefits are gained when processes are complex, and improvements are less enhanced on simple operations. For the purpose of the project, which was to develop a demonstrator to prove the feasibility of introducing AR on Niftylift shop-floor, the tool was applied to a relatively simple process, yet important for Company. Other processes, more complex, could bring further benefits and even more enhanced improvements. Benefits could have also increased by a proper visualisation of the complete task with the 3D model of the valve, which was not available. The understanding of the machine would have increased and consequently the time to understand the task would have decreased.

(53)

43

Finally, it must be considered that data were collected from experiments at Cranfield Laboratory. SOPs were provided by the Company but having data on the real training would have given the best comparison. Indeed, some elements varied when replicating the shop-floor. The shadow board was replicated through an image, since the real one was not available, so time to pick tools was counted as the time people took to find the tool on the board and indicate it. Also, hoses were not inserted in the chassis as it would be, so users had to spend more time looking for the correct one. These factors were the same for both groups performing operations but changing them with the real environment could impact performances. Also, users were engineering students rather than real operators, who might have reacted and performed differently.

7.1.2 Design for AR

Niftylift components were used during the demonstrator development phase, and they were new to Vuforia. The valve’s and the hoses’ recognitions were critical because of the potential benefits they could have introduced. Valve’s recognition for associating animations directly on the real valve, showing how to perform the work, and hoses’ recognition to check if they were correct, before fitting them. To enable these functionalities, trials were made. Results showed that elements’ shape, material and size greatly influence the extent to which Vuforia recognition systems works and can impede the possibility of triggering augmentations.

AR cannot be applied to any manufacturing environment in a straightforward way; prior considerations should be made to ensure that the components have adequate features allowing their recognition. Before applying AR to a whole training process, components’ design needs to be carefully considered. Companies should think about design from the very beginning, particularly if they intend to introduce the technology on new machines. Designing and developing components following guidelines for images and objects recognition can save a notable amount of time later and can provide better performances. The precision of where information is displayed and the effectiveness of knowledge transfer can be significantly emphasized.

(54)

In sections 7.1.2.1 and 7.1.2.2, details on the impact of components feature and consequent difficulties for the valve and the hoses recognitions will be detailed, and conclusions will be drawn in section 7.1.2.3.

7.1.2.1 Hydraulic valve

The 3D model of the valve was not available since it belongs to an old cherry picker model, whose design was not realized with 3D modelling software. This machine is the only one assembled at the training bay, selected to introduce AR.

Consequently, it was decided to 3D scan the valve with an Artec scanner, to use Vuforia model-recognition feature. 3D scanners can develop full models, including shape, material or dimension. However, the outcome was not satisfactory, since the complex shape, the shiny material and the black colour of the valve were not suitable to be recognized by the scanner.

It was then used the Vuforia object-scanner, which maps objects’ external surface. This option was less accurate, but it would have enabled using the object-recognition feature. Nonetheless, no satisfactory results were obtained. The valve was scanned through a tablet, which has a higher precision than the HoloLens. The recognition failed with the HoloLens, that recognised the valve only from certain angulations and positions. Different users, with different heights, would not have had a stable recognition.

A QR marker was used. This has as consequences a lower accuracy in the position of the overlaid information: slight inclinations of the paper, different users’ perspectives and other similar variables affect the orientation of the associated animations. Therefore, it was only possible to trigger their apparition with the QR.

Reasons that prevented a good and stable object recognition, obtainable in other situations with HoloLens, were studied. Vuforia application was designed to work with consumer products, with opaque and rigid parts. It works best when the surface of the objects has contrast-based features and when they are viewed indoors, under moderately bright and diffuse lighting.

(55)

45 7.1.2.2 Hoses’ cylinder targets

Hoses’ shape and features are very similar one to another. Therefore, Vuforia systems could not distinguish them according to their external characteristics. It was decided to introduce cylindrical markers, supported by Vuforia. The software takes in input the flat image, the cylinder diameter and the height of the image. From these elements, it creates the target. An example can be seen in Figure 20.

Figure 20 Cylinder target in Vuforia

Initially, it was decided to use QR codes, but the quality of the images, shrank to fit on the diameter of the hoses, was not good enough to be recognisable by the system. It was then decided to use images provided by Vuforia. The resolution of the images was kept high, cropping them to desired size thanks to images’ editing software. Nonetheless, the small size of the hoses’ diameter impeded the possibility of obtaining acceptable cylinder targets.

Therefore, image targets were developed, applied as labels to the hoses. 7.1.2.3 Conclusions on design for AR

Components’ characteristics can limit the possibility of applying AR in manufacturing environments. Objects rarely have numerous distinct surfaces features and lights condition are determined by the need of working in well-lit, rather than moderately

(56)

bright, environments. Limitations, nonetheless, might be restricted to the cases when 3D models are not available at first and need to be developed.

Despite the numerous attempts, it was not possible to achieve the initial desired outcomes. This implies that design for AR need to be considered since the characteristics of the components (for the valve) or the presence for very similar features (for hoses) can limit the possibility to use fully AR potentialities in manufacturing environments. Using markers have drawbacks, such as the need for preparing the environment prior to usage, which will be discussed in section 7.1.3.1.

7.1.3 AR limitations for training applications

7.1.3.1 Environment preparation

In the training bay, machines are always new and are part of the production schedule. This implies that the environment needs to be set-up for AR usage each time, with hoses labels, the valve QR and the shadow board, that needs to be kept in order, with tools always with the same disposition.

Being time the most relevant KPI for the Company, this time-consuming activity needs to be considered. Niftylift might think about a separate environment, purely dedicated to training. Other advantages of this solution will be discussed in 7.1.4.

7.1.3.2 Tacit knowledge transfer

During the observations at Niftylift, it was possible to witness tacit knowledge transfer. IT and digitalized contents can hardly communicate knowledge acquired through experience, which can be included only if prior gathered through interviews or observations.

There are other elements difficult to replicate with technological tools, such as workers’ attitude towards the job. Trainers that work paying attention to details and that have a sense of belonging towards the Company can transmit the Company culture to new employees. AR can hardly replicate these behaviours, which, nonetheless, varies greatly for person to person.

On the other hand, indeed, introducing a technological tool reduces the risk of teaching bad habits or carelessness during the work to new employees. The advantages and drawbacks should be carefully considered when deciding if to introduce such a tool,

(57)

47

being aware of the limitations that technology has and of the importance of human interactions in the learning process.

Companies should consider that competitive advantage is often gained through tacit knowledge transfer, rather than explicit knowledge, replicable by competitors.

7.1.3.3 Final users’ reactions and change management

During the validation phase, it was observed that the response to this innovative tool varied greatly from person to person and did not depend only on the age, as it might be thought. Users either accepted the tool, enjoying the way training was delivered or did not appreciate the tool, finding it uncomfortable to work. Reactions are determined both by the innate nature of a person (that can be open to innovations or not) and by the way the tool is introduced to the users.

It is therefore worth considering developing a change management plan to support users before the training starts, involving them and explaining them the benefits that can come from AR, since motivation plays a key role when training personnel. Considering this aspect might influence performances and the effectiveness of the AR-enabled training itself.

7.1.4 Limitation of the environment

The environment selected was the most indicated among the possible options. Nonetheless, the production rate and the availability of the resources can depend on the demand. If there are demand picks, the production rate becomes a constraint. It is worthy to think if this environment could be replaced by a dedicated area, independent of the customers’ demand. Training could be delivered on the same machine, which would not need to be prepared each time, and operators would not work in an environment with many distractions’ sources.

A place separated from the shop-floor would also allow using voice commands, out of scope due to the noisy environment. Another drawback of the noise is the battery life of the HoloLens: when they detect a rumour, they try to decipher if it was a voice command or not, significantly consuming battery.

(58)

7.2 Applicability in other environments

The research work can find applicability in another manufacturing environments, to test the feasibility of introducing AR tools. Prior to this, it is crucial to consider two elements: costs of the investment and Companies’ culture.

The relatively high value of Niftylift products leads to high costs related to the training process. Costs are directly proportional to the time spent on training new employees, which affects both production rates and product quality.

Investing in AR and in its customised development needs to be considered carefully through a cost-benefit analysis. Other manufacturing Companies, with lower expenses related to training, might not find it convenient.

Companies’ culture has a great influence on the success of introducing a highly innovative solution to the shop-floor. The organisation needs to be ready to introduce changes and the management attitude towards the solutions is crucial to obtain a successful introduction of technological tools.

(59)

49

8 Conclusion and future work

8.1 Conclusions

The aim of the project, to develop and validate an AR demonstrator to prove the possibility of introducing the technology within the Niftylift context, was achieved. Experiments’ results showed improvements of performances, which should be studied further through validation at the Niftylift site. Results from operators’ training would validate the solution and confirm the benefits.

Despite the difficulties related to the components, the Niftylift environment is suitable for introducing the AR application that can potentially bring benefits according to Company’s KPIs. The tool was introduced on the shop-floor during the preliminary validation and operators used it successfully to perform the process.

The application was developed only on a small part of the assembly procedure; further difficulties may arise when and if applying the technology to the whole training. The size of the machines could impede the possibility of using HoloLens for every step, due to the limited FOV and the length of the process that could create discomfort to the users. The technology could, therefore, be introduced only on certain parts of the process, such as the most complex tasks or the most difficult to learn for the operators. This solution would allow gaining benefits brought by AR and reducing the time necessary to develop the application and relative costs. The risk of discomfort for the users, due to long usage, would be reduced and the environment would not need to be entirely prepared, reducing the time dedicated to this activity.

Technical knowledge could be transmitted through AR, while the human contact with other operators during other parts of the training would allow the transfer of tacit knowledge and Company’s values.

To conclude, Augmented Reality can be a valid solution to address Niftylift training needs and can bring significant benefits to the organization.