Process Simulate DT application

on a snapshot of the wheels taken with an invisible camera in the VE, at the same position and perspective of the real one. The result of the detection will be a tensor, a container that stores data in N-dimensions. The trained YOLO system generates an output of dimension 24× 52 × 52 containing the results of the detection, given by the bounding boxes and the confidence of the detection.

For each object detected in the input image, the output tensor saves the x-axis coordinate, the y-axis coordinate, width, and height. After filtering these results by selecting the ones with higher confidence, it is possible to draw the corresponding bounding box to the snapshot used as input, using different colors to distinguish the three classes (top, back, side).

Inside the VE, a GUI panel displayed near the cobot shows at each frame both the acquired snapshot and the detection result, whereas the user can restart the detection by a button (e.g., after correcting the pose of a misplaced wheel). The ONNX module is able to run YOLO algorithm and give back the results in about 2 seconds, which is a reasonable delay for testing the behavior of the cobot inside the DT. Figure 3.21 shows an example of the detection output inside the VR application.

Figure 3.21: Screenshot of YOLO detection result inside the VR application

develop a complex DT solution for a manufacturing process due to its capabilities to analyze ergonomics for human workstation, recreate robot movements and moreover to simulate the entire process by collecting data from physical assets.

The application developed for this work does not implement this features yet, but it focuses of providing a VR experience of the production line to navigate and analyze the main assets and to test the Inverse Kinematics (IK) solver of the Racer 5 robot.

3.4.1 Design of the Virtual Environment

The virtual environment has been set up by importing the CAD files for each component of the production line; a 2D map of the real site has been used to replicate the exact position and proportion of the assets. Figure 3.22 illustrates how the Process Simulate GUI is organized: the VE comprehends the MIO automated warehouse, the Vir.GIL digital assistant and the Racer 5 robot installed above the conveyor belt; there are two virtual human models that represent the operators responsible for the interaction with the Vir.GIL and the cobot; the Object Tree panel on the left contains a hierarchy of the objects in the scene, organized in different directories; the upper part displays many panels (Robot, Process, Operation) that permits to access to specific features and properties of the software.

Figure 3.22: Screenshot of the DT model in Process Simulate

When the Racer 5 CAD model was imported, it was identified as a Robot component; the Robot panel on the top of the GUI provides all the tools for editing and validating robotic tasks in a 3D engineering environment. The first step was to define its kinematics through the Kinematics Editor panel (see figure 3.23). The Racer 5 is composed of 6 joints; by consulting the datasheet on the vendor’s website it is possible to retrieve the rotations limits for each joint and set them accordingly.

To perform the pick and place operation, the Racer 5 has a gripper mounted on the last joint; after importing the specific CAD model the Mount tool panel enables to install this tool for the robot; the gripper also has his specific kinematics to open and close itself for the pick and place operations. Once completed, the robot has been successfully set up and can be manipulated through the Robot Jog window (see figure 3.24). The All Joints area enables to adjust the values of each robot’s joint in Forward Kinematics (FK), while the Manipulation area permits to exploit the Inverse Kinematics (IK) solver. Therefore, each translation or rotation of the gripper (end effector) will adjust each joint of the Racer 5 accordingly.

Figure 3.23: View of the Process Simulate Kinematics Editor

Figure 3.24: View of the Process Simu-late Robot Jog Panel

3.4.2 Interaction Design

This stage of the DT model refers to the development and simulation of each asset’s behaviour including the human tasks and the interaction with the machines’

GUI. In particular, this section will describe how the Racer 5 movement have been recreated after setting up its kinematics.

Process Simulate provides tool to create a digital simulation of the production

line. Figure 3.25 shows the two panels involved: Operation Tree and Path Editor.

From the former panel a new Operation is defined. In this case the simulation wants to illustrate the operations conducted by the Racer 5 robot to pick and place the trucks and the board. The movements of the robot are defined from the Robot Jog panel exploiting the IK solver and being registered in the Path Editor that provides an easy way to visualize and manipulate path data by displaying detailed information about paths and locations. It is possible to specify the Motion Type for the interpolation between two position (PTP, LINEAR) and also define some macros to control the tool mounted on the robot. For example, the #Drive CLOSE macro will close the gripper when the robotic arm is approaching to the component to be picked up.

Figure 3.25: Process Simulate panels for setting up a simulation

In addition, Process Simulate offers the possibility to load the virtual environment created into a Virtual Reality world. It supports only the HTC VIVE Pro headset and the SteamVR plugin, the same used for the Unity application. In particular, after setting up the robot kinematics it is possible to interact with it and simulate the IK solver by grabbing the end-effector. Another important feature allows to generate a VR Invitation to collaborate in a virtual reality session, event if the guest machine does not run Process Simulate. The collaboration can begin after the host submits an invitation file. After the guests open the invitation file, they share the same virtual environment of the host and can interact with the same assets.