Department of Computer Science
Master’s Degree in Computer Science
Academic Year 2016/2017
Injection of an efficient scripting
language in a large C++ framework
for fully immersive Virtual
Environments
Supervisors Candidate
Prof. Franco Tecchia Gabriele Marini
Prof. Marcello Carrozzino
Reviewer
To my parents, my sister my loved one and my friends.
I would first like to thank my supervisors, Dr. Franco Tecchia and Dr. Mar-cello Carrozzino of Scuola Superiore Sant’Anna for giving me the opportunity to work in the PERCRO lab and experience some of the technologies being developed, and for the rapid responses to all of my emails.
I would like to thank my family for the continuous support, and especially for the latest months which were harder than expected. Their affection and love never left my side even during the hardest days of all these years, and I’ll be eternally grateful to them for giving me the possibility to complete my studies so far from home.
I would also like to thank Lorena, who has always believed in me and in my capabilities.
I want to express my gratitude also to all of my dear friends, especially Francesco who has been a colleague and a friend for many years.
Finally, a special thanks to the Internet for being an infinite source of infor-mation. I would also like to curse the Internet for being an infinite source of information.
This thesis covers the design and the development of the injection of a script-ing language in a large C++ framework for the development of fully immersive Virtual Environments. The aim of the project is to simplify and speed up the process of developing 3D and Virtual Reality applications by providing a new, powerful and easy-to-read scripting language. Professional and popular soft-wares and engines to develop such virtual environemnts make often extensive use of the C++ programming language and while its power and versatility is well known it may be hard to read and understand especially for newcomers and inexperienced programmers, for this reason a different, simpler approach may be useful in such environments.
Epic Games created a powerful, efficient and feature rich framework, the Un-real Engine which has become in the latest years the main choice for many big and small companies or even for single amateur programmers. The Engine provides a full set of components to control every aspect of a virtual environ-ment application: Graphics, Audio, Controls and of course application logic, are all core components of an intricate system.
The implemented language, namely S3D, is a dynamically typed, object-oriented scripting language with a rather small learning curve. It has been developed with the idea of an easy-to-read reusable code which would allow different pro-grammers to understand the mechanics of the application in a simpler way. The aforementioned engine already provides a way to define the logic com-ponent of the application, a visual scripting language called Blueprint, which may explode as the project grows in magnitude. The thesis’ approach aims instead to strictly integrate the S3D language and its environment, the XVR Virtual Machine at the lowest possible level as a new way to code the game-play elements, maintaining the capabilities already offered by the Engine while allowing previously developed XVR libraries to be implemented and used in a project.
1 Introduction 14
1.1 Virtual, Augmented and Mixed Reality . . . 15
1.2 The S3D Language and the XVR System . . . 18
1.3 Objectives Of The Thesis . . . 19
1.4 Thesis Outline . . . 24
2 Background And Related Work 26 2.1 Virtual Reality Frameworks . . . 27
2.1.1 Academic VR Frameworks . . . 27
CAVELib . . . 27
VR Juggler . . . 28
VRUI . . . 28
FreeVR . . . 28
Open Scene Graph . . . 28
SGI Performer . . . 29 CalVR . . . 29 2.1.2 Commercial VR Frameworks . . . 29 WorldViz . . . 29 EON . . . 30 COVISE . . . 30 Avocado . . . 30
OpenVR and SteamVR . . . 31
2.1.3 Commercial SDKs and Game Engines . . . 31
Google Daydream . . . 31
Unity3D Engine and VR . . . 31
Unreal Engine VR . . . 32
2.2 The XVR Framework . . . 32
2.2.1 The S3D language . . . 33 Identifiers . . . 33 Preprocessors . . . 33 Variables . . . 34 Classes . . . 35 Functions . . . 36 Data Types . . . 37 Statements . . . 38 Garbage Collector . . . 39
2.2.2 The Virtual Machine . . . 39
Registers . . . 40
2.2.3 The XVR Execution Environment . . . 40
Stack . . . 42 Heap . . . 43 Instructions Set . . . 43 Interrupts . . . 43 Exceptions . . . 44 Fetch-Decode-Execute Cycle . . . 44 2.2.4 The S3D Compiler . . . 44 Front End . . . 45 Back End . . . 47 2.3 The XVR Debugger . . . 47
2.3.1 Implementation of the XVR Debugger . . . 48
2.4 The Unreal Engine . . . 49
2.4.1 Terminology . . . 50
2.4.2 The Blueprint Visual Scripting language . . . 52
The Blueprint Compiler . . . 54
2.5 The Visual Studio Code Editor . . . 57
3 Overall System Design 62
3.1 The Project Recap . . . 63
3.2 The Unreal Engine . . . 65
Relevant Gameplay Elements . . . 66
Extending the Engine . . . 67
3.2.1 The Unreal - XVR Plugin Structure . . . 68
3.2.2 The XVRScriptEditor Module . . . 70
3.2.3 The XVRScriptEditorGUI Module . . . 72
3.2.4 The XVRScriptInit Module . . . 73
3.2.5 The XVRScriptRuntime Module . . . 73
Game Instance . . . 75
Game Elements . . . 76
The Blueprint Injection . . . 78
3.3 The XVR Framework’s Structure . . . 81
3.3.1 The S3D Compiler Design . . . 82
Symbol Table . . . 83
Parser & Lexer . . . 85
Code Optimizer . . . 85
Code Generation . . . 87
3.3.2 The XVR Virtual Machine structure . . . 88
Virtual Machine’s API . . . 91
3.4 The XVR Debugger and IDE . . . 92
3.4.1 The XVR Debugger structure . . . 94
Debug Symbol Table . . . 94
Lexical Scopes . . . 95
The Debugger communication system . . . 97
3.4.2 Overview of Visual Studio Code . . . 100
The VSCode Debug Adapter . . . 101
4 Project Implementation 108
4.1 Changes To The XVR Framework . . . 110
4.1.1 Modifications to the S3D Compiler . . . 111
Compiler Data Structures . . . 111
Parser and Lexer . . . 113
Bison and Flex . . . 116
4.1.2 The XVR Virtual Machine integration . . . 119
XVR Class Declarations . . . 122
External classes usage in S3D Scripts . . . 129
4.2 The Unreal Engine XVR Plugin implementation . . . 131
4.2.1 Overview of the Unreal Reflection System . . . 132
4.2.2 Relevant Unreal Type Classes . . . 135
4.2.3 Anatomy of the Plugin . . . 137
4.2.4 Communication with the IDE and the Unreal Engine . . 139
IDE Communication . . . 141
The update of the Editor’s Blueprint Graph . . . 143
4.2.5 The Plugin’s core implementation . . . 144
Integration with the Blueprint langugae . . . 145
Injecting XVR in the Unreal C++ Framework . . . 150
Reverse injection: Integrating Unreal in XVR . . . 155
Main XVR classes and methods . . . 157
XVR to Unreal Type conversion . . . 167
XVR Basic Types Conversion . . . 168
Collections type conversion . . . 171
Objects type conversion . . . 171
Unreal to XVR Type conversion . . . 174
Basic Unreal types conversion . . . 175
Unreal UObject to XVR conversion . . . 176
Unreal Collection to XVR conversion . . . 177
4.2.6 The Integration with Unreal Editor . . . 180
The XVRScriptEditorGUI module implementation . . . 181
The XVRScriptInit module . . . 181
4.3 The XVR Debugger and IDE integration . . . 183
4.3.1 The Debugger’s Communication Channels . . . 185
4.3.2 The Visual Studio Code extensions . . . 189
The XVR Debug protocol in VSCode . . . 193
4.4 Benchmarks and Performance results . . . 197
Benchmark setting . . . 197
Results . . . 200
Project Installation And Use Case . . . 205
5 Conclusions 208 5.1 Future improvements and projects . . . 212
Glossary 214
List of Code Snippets 220
List of Figures 224
Virtual Reality[1] (VR) along with Augmented and Mixed Reality has seen an incredible growth in the latest years, bringing a whole new way of interact-ing and experiencinteract-ing 3D worlds, or even the real world itself enhanced with virtual elements. With the advent of technologies such as Oculus Rift and HTC Vive, and a slightly different approach brought by Microsoft HoloLens and Google Daydreaming, many developers started moving towards VR and Mixed Reality development. Nevertheless, developing Virtual Environments always represented a challenge. Computer Graphics and real-time rendering have always been a big portion of the development process and final outcome, and plays a key role in the user immersion in the virtual world. Thanks to the most recent technologies, 3D development is now accessible to many, even without the need of perfectly knowing graphic libraries such as OpenGL or DirectX. To simplify and add a new layer between the low-level graphics and core implementation and the actual designing and creative process of any Vir-tual Environment, engines started to become a popular choice for creating 3D applications. While as of today is still in use to develop private, company’s engine, for many designers and developers, creating an engine from scratch to manage graphics, audio and core elements may represents an obstacle, as its creation would require the knowledge of many fields, not only programming and Computer Graphics.
In the last ten years some engines started to become more popular, stand-ing out of the crowd, and eventually becomstand-ing free and accessible to every-one. Such softwares offer a user-friendly interface, an already fully-functional graphic engine packed with the latest technologies in Computer Graphics and real-time rendering, while providing overall good performances. Hiding the implementations details and offering a user-friendly interface allowed design-ers or single developdesign-ers to create content in a simpler way, focusing on the virtual world creation and core development, without having to worry about the graphics fidelity, audio management, physics implementation and many other aspects.
Among the many engines present in the market, some of them stands out for their multitude of features, ease of use and performance for any kind of 3D ap-plication, the most popular being Unity3D, CryEngine and Unreal Engine[2]. The Unreal Engine specifically, developed by Epic Games, is one of the most popular choices, used for developing any kind of virtual environment, from pro-fessional training systems, console and mobile games, to virtual, augmented and mixed reality applications. Being free for private developers and starters and thanks to its portability, performances and extensibility made it one of the top choices for both professional and amateur development. Among the plethora of its features, Unreal offers a visual scripting language, namely Blueprint[3], to develop the core aspect of an application, as opposite to a more traditional C++. Blueprint has been designed specifically for newcom-ers and designnewcom-ers which supposedly don’t have the expertise and knowledge to smoothly use a more complex programming language as C++. While it offers a reduced but still large number of functions and types with respect to those present in their C++ language, it may eventually result in a complicated and large graph and lower performances.
1.1
Virtual, Augmented and Mixed Reality
VRMedia[?], along with the Scuola Superiore Sant’Anna, is the company
be-hind the S3D language[5] and the XVR framework[7], operating in the virtual and augmented reality sector, developing innovative hardware/software prod-ucts and solutions for Companies and Research Centers. Its prodprod-ucts are spe-cially used in the industrial sector for technical personnel training, advanced after sale services and research activities.
Virtual Reality or VR completely revolutionized the way people interact with a virtual environment. In the past, 3D world and virtual environments had to be explored by using various non-immersive devices, and while the presented world was in three dimensions, even with the most advanced technologies and graphics fidelity, it stayed inside a computer screen, being effectively, a
two-dimensional representation of a virtual world. Stereoscopic vision and 3D glasses were already a reality before the year 2000 but it is in 2010 with the first prototype of the Oculus Rift that VR started to become popular and accessible to consumers and developers.
The main obstacle to a full immersion in a virtual world was represented by the screen used to reproduce the application, it could never reproduce the depth and the ”real” feeling, no matter how photorealistic graphics could be. The VR headset aims to bring the screen closer to the eyes, and reflect the way human vision works by exploiting binocular and stereoscopic vision. By using one or two screens with high resolution and high Dots Per Inch (DPI) and showing a different point of view per eye, the headset gives the illusion of depth to the user. Furthermore, the headset’s gyroscope and accelerometer keep track of the head movement, moving and tilting the virtual camera of the world accordingly to achieve a natural realistic feeling of the head movement.
Figure 1.1: Representation of a VR headset functioning
Since the Oculus Rift presentation, many companies developed their own VR headset with more or less success, each one proposing innovative solutions to the other big issue for a full immersion, the interaction. As the user is directly transported in a virtual world, for a full immersion it should be able to interact with it as if it was the real world. While this still represents a big challenge in the field, some companies as HTC and Oculus proposed some effective solutions by tracking the user movement in a small room or delimited space. The controllers supposedly substitute the hands of the user, allowing an interaction with the virtual world, obviously this still represents a big limitation, there is still no touch feedback, and the user’s hands and
fingers movement are still a challenge to reproduce in the virtual world. Other solutions such as the one developed at the PERCRO, aim to bring a more realistinc feeling to the virtual environment using haptic devices and body tracking to bring the user’s body into the virtual world.
Figure 1.2: Example of VR controllers
While VR aims to bring the user in a virtual world, Augmented Reality or simply AR, aims to bring virtual elements directly into the real world. Both the real world and the virtual elements are still limited to a screen, be it a computer or a mobile screen. By using a camera, the idea is to ”augment” the reality by introducing virtual objects. Only lately, cameras specialized in this field started to become more accessible to consumers and phones started to offer a better tracking system with new possible applications.
Finally, the last frontier is the Mixed Reality, which as the name suggests, aims to mix virtual and augmented reality, bringing the user immersion to a new level. By using a VR headset and a high resolution 3D camera, the user is directly transported to a virtual representation of the real world augmented with virtual objects. In the most recent times, Microsoft has made some big steps in this direction by showing its HoloLens, a VR headset loaded with a powerful 3D camera which constantly scans the real world. The camera creates an internal 3D representation of the world used to provide an extremely precise tracking system to position and visualize 3D elements in it. As the user head moves, the headset moves and rotates the objects accordingly, effectively giving to the user the illusion of the presence of virtual objects in the real world. While the whole process is hidden to the final user, it results in a precise, high quality blending of Virtual and Augmented Reality.
Figure 1.3: Different kind of realities
The virtual world in which the user is transported can be either a virtual representation of the real world of even a completely new world, realized in such a way to convince the user of being part of it. As of today, there are not yet devices able to make use of all of the human five senses. In fact usually virtual environments as videogames or virtual training systems, can solely appeal to the human vision and hearing, reducing the user immersion in the virtual world. For this specific reason, the focus is usually on the vision and hearing senses, trying to make the virtual world as convincing as possible, and letting the user navigate and freely explore it. With specific devices, called
haptic interfaces it can be possible to receive tactile feedbacks in response to
actions performed in the virtual world. The PERCRO[6] laboratory of the
Scuola Superiore Sant’Anna of Pisa (where the thesis has been developed)
works in the development of innovative simulation systems, with the help of haptic interfaces and VR systems.
1.2
The S3D Language and the XVR System
XVR (eXtreme Virtual Reality)[7] is a framework for the development of VR applications, it has been developed by the Scuola Superiore Sant’Anna within the PERCRO laboratory in association with VRMedia Srl. During the years, many XVR libraries for haptic interfaces and simulation systems have been developed inside the laboratory, which has been for many years at the top for innovation and research. The development of VR application with the XVR framework, starts by writing scripts in a proprietary language, namely S3D. Thanks to the gentle learning curve of the powerful language, it has
been possible to build in a simpler way, complex interactive applications. The scripts are compiled to a bytecode which is then executed by the XVR Virtual Machine, making it portable and multi-platform. The XVR system also inte-grates a graphic engine used to represent virtual environments in which the user would be immersed and could interact with the interfaces developed by the laboratory.
The XVR framework also offered a complete IDE, with its own project management system and debugger. A debugger is an important component in the process of developing applications. Its main role is to help the programmer to find runtime errors and bugs, often helping to understand highly complex code and its functioning. The debugger is intimately dependent from the IDE it is integrated in, from the language it is designed for, the compiler and the machine in which it is executed.
Figure 1.4: Some of the systems developed at the PERCRO laboratory
1.3
Objectives Of The Thesis
With the big companies growing interest for Virtual Reality, the XVR Frame-work started to lose its appeal, and its development gradually slowed down. The S3D language is still a powerful and expressive language with many features and a short learning time which means it could possibly be mas-tered faster than other programming languages. The XVR Virtual Machine
with its portability and efficiency still represents a valid choice for program-mers, and specifically for the PERCRO laboratory, backed by a large number of libraries to support the systems developed inside the lab. Considering the amount of possibilities brought by the powerful, versatile and portable Unreal Engine, the lab choice for developing VR application often fell on it. While the Blueprint system offered by Unreal is mainly thought and developed for new programmers and designers, its readabilty may be affected as the project grows and it lacks of compatibility with other systems and languages. Thanks to the plugin system of Unreal, programmers can expand the engine functionalities and add new features to accomodate their needs. This system proved to be the perfect choice to revamp and give new life to the XVR environment.
This thesis faces the challenge of injecting the S3D language and the XVR Virtual Machine, into the already mentioned Unreal Engine, with the aim of maintaining a more traditional development workflow, without the need of managing a large C++ project and its compilation, and with many other fea-tures, among all, a Just In Time (JIT)[8] compilation for testing and fast features implementation. The project’s goal is to bring a full packaged devel-oping suite, with an IDE and a debugger complete with its interface, everything completely integrated into the Unreal development workflow.
The project represents a different approach to the one offered by Unreal, of-fering the right compromise between C++ programming and a higher level AI and core development language. To better understand and describe each of the project’s many sides, it can be further divided in three macro parts. Each one of these parts is essential and strictly connected and dependent on each other:
• The S3D Language and the XVR environment. • The Unreal Engine
• The XVR Debugger.
• The Visual Studio Code[9] editor and its debug adapter extension. The main purpose is to accelerate and simplify the development of VR and more generally 3D applications. The debugging of S3D scripts should be fast
and a tool for the user to individuate bugs and errors in the code, and to better understand code which he or she is not familiar with. Some relevant features provided by the integration are listed below:
• Fast compilation time. • JIT compilation.
• Simple and fast debugging.
• A new, lightweight and interactive IDE which integrates an adapter for the XVR debugger and a simple, clean debug UI.
• Portability and multi-platform.
• Low performance hit with respect to Blueprint.
• Code reusability: Moving single scripts and line of codes is easier than copying and replicate a Blueprint graph.
• The ability to define gameplay elements and the logical side of an appli-cation by writing S3D scripts.
• The ability to replicate a behavior among copies of the same entity by reusing the same code.
Injecting the S3D language has several implications:
• The language should not be seen as a complete substitute of either C++ or Blueprint, but more like a complement to them.
• Integrating the whole Virtual Machine and the S3D compiler directly in-side the Engine may cause performance, along with memory management issues.
• There should be a direct correspondence of what is written in S3D thor-ough scripts and what actually happens in the Engine, that is, the lan-guage should correctly communicate and control the engine.
• The language should not be more complex and should not require a higher compilation time than both Blueprint and C++, as it is supposed to be a more efficient approach from both technical and usability aspects. • The Virtual Machine and the S3D compiler need to be modified ac-cordingly, in order to maintain backward compatibility. These changes should kept as minimal as possible and should not affect performance in any way.
• Debugging should be a possibility, hence a debugger and a proper text editor or IDE should be provided with the Unreal extension.
• The debugger should be a separated process, an observer of the current execution, and should be able to communicate with the chosen IDE • The proposed scripting language should interface with both C++ and
Blueprint, allowing the user to expose functions to both parts, while also integrating the already present functionalities.
Figure 1.5: A view of the Unreal Engine editor with the XVR-integration
Most of the objectives were eventually met, except for a full integration of all the Unreal types and functions, included the ones not reachable through the
Unreal reflection system (described in the following chapter). The injection of the S3D scripting language proved to be efficient and a better option with respect to the ones offered by the Unreal Engine. The result has been achieved through the development of an Unreal plugin and a set of modules which directly create and communicate with the XVR Virtual Machine. The S3D language has been extended with some new rules to define and expose blueprint
functions, which can be directly converted in a spawnable blueprint node. The
VM is created at each game session an deleted at their end in order to keep it consistent and coherent with the Unreal philosophy seen in the Blueprint system. Finally, the XVR debugger has been extended and properly modified to support an asynchronous communication with the Virtual Machine and the chosen IDE. The IDE choice fell on the new Visual Studio Code, developed by Microsoft, due to its extension system, workspace management and debug UI. Finally, an extension has been developed to allow a simple syntax-highlighting of the S3D language and to directly communicate with the Unreal Engine, the debugger and the XVR VM, allowing the user to start, stop and control a debugging session with all the features usually provided by any IDE.
1.4
Thesis Outline
The next chapter gives an overview of some VR frameworks used for developing VR applications. Among these, the Unreal Engine. The section 2.2 generally describes the XVR framework and the S3D language, showing its grammar rules and features. At the end of the secion an overview of the XVE Execution Environment is given by describing some of its components and phases.
The chapter 2.3 present another component of the XVR environment, the debugger. The focus will shift then to an overall presentation of the main features offered by the Unreal Engine, and its Blueprint visuals scripting lan-guage. Finally in the last part of the chapter, the editor used to integrate a communication protocol with the XVR debugger is presented along with some of its features.
The chapter 3 focuses on the project structure and system design, pre-senting a high-level description of the implemented system. First, the plugin implemented in the Unreal Engine to provide an integration with the XVR framework and its structure is described, by listing and showing the connec-tions among all of its modules. The section 3.3 explains how the XVR frame-work structure has been used and in some cases modified to be injected into the Unreal Engine, while the last section 3.4 will show a general picture of the XVR Debugger’s structure and functionalities and an overview of the editor and its extensions used to communicate with it.
Chapter 4 enters to a more detailed level, describing the realization of the system by showing some of its most relevant classes and functions. Before showing the plugin implementation in the Unreal Engine, the changes applied to the XVR Virtual Machine and to the S3D compiler are listed and described. The focus eventually shifts to the Unreal Engine’s part of the project, showing how the classes and functions provided by the XVR Virtual Machine’s API along with the Unreal property system have been exploited to bring an almost complete integration of both environments. In Section 4.3 the debugger’s mod-ification details and its editor’s adapter implementation are described. Finally, some benchmark’s result shows the performance of the realized system com-pared to the other possibilities.
The Chapter 5 exposes the general conclusions of the project and some future improvements and ideas.
This chapter gives an overview and a description of the most relevant features offered by the libraries and software used for this thesis. The Unreal Engine provided a suitable but limited documentation on its core functionalities and low-level implementation details. The S3D Compiler and the XVR Virtual Machine instead are projects of the PERCRO laboratory in which the thesis has been developed. Also, the IDE has been chosen mainly for its lightweight, portability, extension feature, rich documentation and largely used framework (Node.Js and TypeScript). Therefore, all the parts hereby described are gears of the same package.
First it gives an overview of some frameworks used for the development of VR applications, among all the Unreal Engine.
In section 2.2 the XVR framework is described giving a general overview of the XVR environment and the S3D scripting language’s grammar, while in section 2.3 its debugger is presented. Section 2.4 presents the Unreal Engine and some of its features, listing the terminology used to refer to specific elements of the engine and eventually describing the Blueprint Visual Scripting language. Finally, in section 2.5, an overview of the chosen editor shows its features and how it has been implemented.
2.1
Virtual Reality Frameworks
In first place, commercial virtual reality authoring systems allow users to build custom VR applications some examples are WorldViz[11] EON Studio[12] COVISE[13] or AVANGO[14]. However, these systems are not open source and only allow the programmer to implement specific applications which often do not offer advanced support to other technologies or case scenarios.
Secondly, academic VR frameworks are designed to be rather limited in scope, which mostly abstract from PC clusters to drive a multi-screen dis-play device and support a variety of tracking systems such as Mechdyne’s CAVElib[15] and Carolina Cruz-Neira’s VR Juggler[16]. Nevertheless, they leave higher level tasks to the programmer, such as navigation or menu widgets. Some existing open source VR systems are just as extensible as CalVR and can be used for many tasks interchangeably with it. Examples are Oliver Krey-los’ VRUI[18] Bill Sherman’s FreeVR[19] or Fraunhofer IAO’s Lightning.[20] CalVR integrates OpenSceneGraph[21] (OSG) for its graphical output. Open-SceneGraph was modeled after SGI Performer[22] and is syntactically very similar, but it has evolved into its own programming library over the years. However, OpenSceneGraph lacks cluster-ability, tracking support, a way to de-scribe multi-display layouts,menu widget and interaction system. Thus, Open-SceneGraph cannot drive alone VR systems.
In the latest years, many companies developed their own VR framework. Among all the most commonly used in game engines is OpenVR[26] along its software support, SteamVR[24].
The following sections give an overview of the previously listed frameworks and their peculiarities.
2.1.1
Academic VR Frameworks
CAVELib
CAVELib provides the cornerstone for creating robust, interactive and three-dimensional environments. CAVELib simplifies programming and it is con-sidering as the most widely used Application Programmer Interface (API)
for developing immersive displays. Developers concentrate on the application while CAVELib handles all the CAVE software details, such as the operating and display systems and other programming components keeping the system platform independent.
VR Juggler
VR Juggler is a cross-platform virtual reality application development frame-work maintained by IOWA State University. Their motto ”Code once, run
everywhere” sums up their goal to simplify common tasks in VR applications.
VRUI
The Vrui VR toolkit aims to support fully scalable and portable applications that run on a range of VR environments starting from a laptop with a touch-pad, over desktop environments with special input devices such as space balls, to full-blown immersive VR environments ranging from a single-screen work-bench to a multi-screen tiled display wall or CAVE.
FreeVR
FreeVR is an open-source virtual reality interface/integration library. It has been designed to work with a wide variety of input and output hardware, with many device interfaces already implemented. One of the design goals was for FreeVR applications to be easily run in existing virtual reality facilities, as well as newly established VR systems. The other major design goal is to make it easier for VR applications to be shared among active VR research sites using different hardware from each other.
Open Scene Graph
The OpenSceneGraph is an open source high performance 3D graphics toolkit. It is used by application developers in fields of visual simulation, games, virtual reality, scientific visualization and modelling. It is written entirely in Standard C++ and OpenGL and it runs on all Windows platforms, OSX, GNU/Linux,
IRIX, Solaris, HP-Ux, AIX and FreeBSD operating systems. The OpenScene-Graph is established as the world leading scene graph technology, used widely in the vis-sim, space, scientific, oil-gas, games and virtual reality industries.
SGI Performer
OpenGL Performer is a powerful and comprehensive programming interface for developers creating real-time visual simulation and other professional performance-oriented 3D graphics application.
CalVR
CalVR combines features from multiple existing VR frameworks into an open-source system. It is a new virtual reality middleware system which we devel-oped from the ground. In addition, CalVR implements the core functional-ity of commonly used existing virtual realfunctional-ity middleware, such as CAVElib, VRUI, FreeVR, VR Juggler or COVISE. It adds to those that it supports sev-eral non-standard VR system configurations, multiple users and input devices, sound effects, and high level programming interfaces for interactive applica-tions. CalVR consists of an object-oriented class hierarchy which is written in C++.
2.1.2
Commercial VR Frameworks
WorldViz
WorldViz is a virtual reality software company that provides 3D interactive and immersive visualization and simulation solutions for universities, govern-ment institutions, and commercial organizations. WorldViz offers a full range of products and support including enterprise grade software, complete VR sys-tems, custom solution design and application development. Vizard is one of the products offered by WorldViz and it is a virtual reality software toolkit for building, rendering, and deploying 3D visualization & simulation applica-tions. It natively supports input and output devices including head-mounted displays, CAVEs, Powerwalls, 3D TVs, motion capture systems, haptic
tech-nologies and gamepads. Vizard uses Python for scripting and OpenSceneGraph for rendering.
EON
EON Reality is a Virtual and Augmented Reality based knowledge transfer for industry and education. EON offers a wide range of true cross platform solution enabling Augmented Reality, Virtual Reality, and interactive 3D ap-plications to seamlessly work with over 30 platforms. One of its goals is to bring VR and AR in the everyday life, from mobile to commercial uses, pushing the research and development on holographic solutions and immersive systems.
COVISE
COVISE stands for COllaborative VIsualization and Simulation Environment. It is an extendable distributed software environment to integrate simulations, post processing and visualization functionalities in a seamless manner. It was designed for collaborative working, letting engineers and scientists to spread on a network infrastructure. In COVISE an application is divided into several processing steps which are represented by COVISE modules. Each module is implemented as separate processes and it can be arbitrarily spread across different heterogeneous machine platforms. Major emphasis was put on the usage of high performance infrastructures such as parallel and vector computers and fast networks.
Avocado
Avocado is an object-oriented framework for the development of distributed and interactive VE applications. Data distribution is achieved by transparent replication of a shared scene graph among the participating processes of a distributed application. Avocado focuses on high-end, real-time, virtual en-vironments like CAVEs[17] and Workbenches], therefore, the development is based on SGI Performer.
OpenVR and SteamVR
OpenVR is an API and runtime that allows access to VR hardware from multi-ple vendors without requiring that applications have specific knowledge of the hardware they are targeting. This repository is an SDK that contains the API and samples. The OpenVR API provides a game with a way to interact with Virtual Reality displays without relying on a specific hardware vendor’s SDK. It can be updated independently of the game to add support for new hardware or software updates. The API is implemented as a set of C++ interface classes full of pure virtual functions.
Many engines and VR applications simplify their integration of VR by using SteamVR. It leverages OpenVR and provides the software support to offer the best possible VR experience.
2.1.3
Commercial SDKs and Game Engines
At a higher development level, engines offer a wide range of tools to improve VR applications, from graphics and animation tools to audio management and obviously gameplay and core programming for the basic elements of the application. C++ is widely used among engines, and most of them implements one or more of the previously mentioned frameworks to provide a full immersive VR and AR experience, leaving to the programmer the only task of creating the world.
Google Daydream
Google Daydream aims to bring Virtual Reality to the mobile world. Thanks to the Google VR SDKs for Android supports both Daydream and Cardboard. Google provides SDKs to develop VR applications with pure Android, Unity Engine and Unreal Engine.
Unity3D Engine and VR
Unity is a cross-platform game engine developed by Unity Technologies which is primarily used to develop video games and simulations for PC, consoles, mobile
devices and websites. The engine targets popular APIs such as Direct3D, Vulkan and OpenGL. In many aspects, it is similar to the Unreal Engine used for this thesis’ project.
Unreal Engine VR
The Unreal Engine, which will be covered in detail in the following chapters, offers a full support to the most common VR headsets ans systems such as HTC Vive, Oculus Rift and Google Caradboard and Daydream. Furthermore it offers full support to SteamVR and the OpenVR framework.
2.2
The XVR Framework
XVR is an Integrated Development Environment developed by Scuola
Superi-ore San’tAnna designed to create Virtual Reaity applications. Using a modular
architecture and a VR-oriented scripting language, namely S3D, XVR content can be embedded on a variety of container application. XVR supports a wide range of VR devices (such as trackers, 3d mice, motion capture devices, stereo projection systems and HMDs). Due to its extensive usage in the VRMedia group, a large number of libraries have been developed to support the men-tioned devices along with many more developed at the PERCRO laboratory of the Scuola Superiore Sant’Anna. XVR evolved during the years to include its own engine, and while being a powerful and extensible graphic engine, its development has gradually slowed down.
For this thesis’s purpose, the XVR environment has been revamped and inte-grated in an already existing and powerful engine as the Unreal Engine. The XVR Virtual Machine (VM) allows S3D compiled scripts to be executed on any platform. One of its relevant features is the possibility to load additional modules for advanced functionalities, like the support to VR devices. The XVR VM, like many other Virtual Machines, contains a set of bytecode in-structions, a set of registers, a stack and an area for storing methods. The XVR Scripting Language (S3D) allows to specify the behaviour of the applica-tion, providing the basic language functionalities. The script is then compiled in a bytecode which is processed and executed by the XVR VM.
2.2.1
The S3D language
The S3D script language has a syntax very similar to JavaScript and C++. It is possible to declare variables, functions and classes.
Identifiers
Identifiers are names which are used to declare variables, functions and classes. The rules to follow to build a valid identifier are the same of the C language, i.e. they can be constituted by:
• Alphanumeric characters. • Underscore character: ‘ ’ .
As it happens for other programming languages, the first character must not be a number; besides, it is not possible to use reserved names used by the compiler.
Preprocessors
Preprocessor directives, such as #define and #ifdef, are typically used to make source programs easy to change and easy to compile in different execu-tion environments. Directives in the source file tell the preprocessor to perform specific actions. For example, the preprocessor can replace tokens in the text, insert the contents of other files into the source file, or suppress compilation of part of the file by removing sections of text. Preprocessor lines are recog-nized and carried out before macro expansion. Therefore, if a macro expands into something that looks like a preprocessor command, that command is not recognized by the preprocessor.
In order to facilitate the exchange of software modules and to preserve the IP of developers, XVR was added a new pre-processor directive, #import, allowing to import a pre-compiled XVR script under the shape of a .s3d.bin file and to enable using the functions defined and implemented in the external imported module. Functions must be explicitely declared as extern in order to be called by the host XVR application. By using the pre-processor directive #include .s3d files may be included into others.
Figure 2.1: Example of importing an external library in S3D
Variables
Variables may be declared by using the var keyword.
Listing 2.1: S3D Variable declaration syntax
1 var < var1 >[= < value1 >] [ , < var2 >[ = < value2 > ] . . . ] ;
It is possible to declare more than one variable with a single instruction and to assign an initial value. Scope rules are the same of the C/C++ language. There are two special type of variable:
• static: Static variables are declared by using the static keyword. It specifies that the variable has static duration (it is allocated when the program begins and deallocated when the program ends) and initializes it to 0 unless another value is specified. The reset command restores the variable to the initialization value.
• global: A global variable is a variable declared in the main body of the source code, outside all functions. The global keyword specifies that the variable is global even if it is declared inside a block.
Variables declared inside a block are local variables and are destroyed when the block in which they have been declared is left.
Classes
A class may be declared by using the class keyword and declaring inside its body:
• (Optional) Member variables (by using the var keyword. • A constructor function.
• A destructor function.
Listing 2.2: S3D class declaration syntax
1 // d e c l a r a t i o n of c l a s s n a m e ( and b a s e c l a s s e s )
2 c l a s s < c l a s s n a m e >[: < c l a s s b a s e 1 >][ , < c l a s s b a s e 2 > ] [ , . . . ] { 3 // c l a s s d e c l a r a t i o n b o d y
4 var < var1 > [ , < var2 > . . . ] ; // d e c l a r a t i o n of m e m b e r v a r i a b l e s
5 ... 6 < m e t h o d n a m e > ( [ < var1 >] [ ,[ < var2 >] ... ] ) ; // d e c l a r a t i o n of m e t h o d s 7 ... 8 }; 9 10 f u n c t i o n < c l a s s n a m e >:: < c l a s s n a m e > ( [ < c_var1 >] [ ,[ < c_var2 >] ... ] ) { 11 [ < c o n s t r u c t o r body >] 12 } 13 14 f u n c t i o n < c l a s s n a m e >::˜ < c l a s s n a m e > ( [ < d_var1 >] [ ,[ < d_var2 >] ... ] ) { 15 [ < d e s t r u c t o r body >] 16 }
Classes objects may be instantiated by writing:
Listing 2.3: S3D object instancing syntax
1 var < obj > = < c l a s s n a m e > ( [ < c _ p a r a m 1 >] [ ,[ < c _ p a r a m 2 >] ... ] )
As seen in many Object-Oriented language, to call a method or access a vari-able of an instanced object, the ”.” may be used after the identifier of the instanced object.
Listing 2.4: S3D object method calling
1 < obj >. < m e t h o d n a m e >( [ < m _ p a r a m 1 >] [ ,[ < m _ p a r a m 2 >] ... ] )
Every class also implements some built-in to call user-defined methods by name:
• GetName(): Returns the name of the class.
• Call("name",[<m param1>] [,[<m param2>] ... ]): Calls the method ”name” with the specified list of parameters.
• Set("namevar",<value>): Sets the value of a member variable. • Get("namevar"): Returns the value of a member variable.
Functions
A function may be declared by using the function keyword and declaring its body.
Listing 2.5: S3D function definition syntax
1 f u n c t i o n < func > ( [ < var1 >] [ ,[ < var2 >] ... ] ) { 2 <F u n c t i o n Body >
3 }
Inside the function body it is possible to declare new local variables, call other functions and methods (both user defined or predefined), use language instruction etc. The return instruction allows to specify the return value of the function. If the value is not specified, or return is not used, the function returns a null value. If a function is used before its declaration, it is necessary to declare a prototype of the function before its first usage. With a prototype declaration, the compiler is instructed about the function existance and about the number of parameters. In the real declaration it is possible to specify dif-ferent names for the formal parameters, but the number has to be the same. The number (APN) of actual parameters can be different from the number (FPN) of formal parameters. If APN < FPN, the missing parameters will be passed with NULL values, otherwise if APN ¿ FPN the exceeding parameters will be ignored. Variables can be passed to the function by reference or by value. If by reference, the actual parameters undergo the possible modifica-tions operated by the function, otherwise they cannot be modified. To pass a
variable by reference, the & symbol must be put before the identifier. It should be noted that Arrays or Objects are always passed by reference. Some basic functions are embedded in the core of XVR, therefore their im-plementation is independent from the XVR virtual machine. These functions allow to perform basic data manipulation:
• Math functions: sin, cos, tan, asin, acos, atan, log, ln, sqrt, int, abs.
• Vector functions: Vector, modulus. • Array functions: aadd, adel, asize.
• String functions: upper, lower, left, right, space. • Other functions: valtype, str, char, asc, val, trace.
Data Types
Constants and variables assume values which may own to predefined data types. It is not necessary to specify the type at declaration time, as it is au-tomatically assumed at the first instancing. Therefore, although the following data types exist, they are not specified with the corresponding keyword, except from vector and array (for instance, int is not a data type but a predefined function). The basic types are:
• Bool. • Float. • Int.
• Object: In addition to user-defined classes, the S3D language offers lots of predefined classes. It is possible to declare objects of these classes (except from classes which allow just one object automatically created) and to access methods and members with the operator ‘ . ’
• String: Strings are sequences of character with no limits on length. As for the arrays, it is possible to access an element (a character) using an index and the operator [ ].
• Vector: Vectors are array of floats. They can have any dimension. In the XVR reference, when not differently specified, vectors are assumed to be three-component.
• Matrices: Although not directly supported, often XVR functions re-turn, or accept as input, 16-dim vectors representing matrices
• Array: An Array is a set of elements which can be accessed through an index. The array size is not fixed, but it may be changed with particular functions. The S3D array is much more flexible than in other languages, as its elements may be of differeny type. To specify the index in order to access the desired element, the operator [ ] is used. The first element has index = 0.
The type system is composed by:
• A mechanism to define types and associate them with language con-structs.
• A set of rules to manage type equivalence and their compatibility. The language is strongly typed and does not allow operations on types which are not compatibles. It is also dynamically typed, in fact most of the checks are done at runtime.
Statements
A statement is a piece of code that performs an action. Some statements de-clare user-defined elements, such as variables, functions, classes, and enumer-ations, while other statements control program flow. The possible statements in the S3D language are:
• if..else: Conditionally executes a group of statements, depending on the value of an expression.
• while: Executes a statement until a specified condition is false. • do..while: Repeats a statement until a specified condition is false.
• for: Executes a block of statements for as long as a specified condition is true.
• foreach: Executes a block of statements iterating through a collection • continue: Stops the current iteration of a loop, and starts a new
itera-tion.
• break: Terminates the current loop, causing the program to exit from the current for, while or switch block.
• switch..case: Enables the execution of one or more statements when a specified expression’s value matches a constant.
• goto: Transfers control to a label. The given label must reside in the same function and can appear before only one statement in the same function.
• set: It is used to set the value of some system constants before the program actually starts.
Garbage Collector
All of the instantiated objects, vectors and arrays are dynamically allocated. The programmer cannot control their allocation. S3D implements a garbage collection mechanism based on reference counting[51] which automatically cleans the virtual memory.
2.2.2
The Virtual Machine
All the S3D scripts are compiled to an intermediate format called bytecode. Theoretically speaking, the bytecode should be executable on any combination of hardware and Operative System for which there exists a VM implementa-tion, similarly to Java and its Virtual Machine. The VM’s interpreter trans-lates at runtime the bytecode to code that can be executed by the machine on which the VM is being executed. This guarantees the code portability while causing a light performance drop. The VM is composed by:
• A set of registers. • An execution environment. • An instructions set. • An exceptions set. • An interrupts set.
Registers
The registers are used to rapidly retrieve the state of a process in the VM. The machine state is held by the following registers:
• Code Segment (CS): Contains the index to the first instruction of the executable.
• Stack Segment (SS): The base stack pointer.
• Data Segment (SA): Address of the array containing statically allo-cated data such as global and static variables.
• Instruction Pointer (IP): Contains the address to the next instruction to be executed.
• Current Instruction (CP): It contains the instruction which is cur-rently being executed.
• Stack Pointer (SP): The top stack pointer.
• Base Pointer (BP): Contains the address to the base of the activation records stack of the current function.
2.2.3
The XVR Execution Environment
An XVR process is executed in its own execution environment. This environ-ment deals with the allocation and the objects layout. Furthermore it imple-ments the mechanism of function calling, method calling, parameter passing and garbage collection. Follows a list of the various tasks of the environment with a brief descripition:
• Allocation and layout of a process’ objects: An XVR executable is executed in its own indexing space, a subset of the virtual memory used by the VM. The indexing space of an executable is made by:
– A set of instructions. – Statically allocated data. – Stack.
– Heap.
It should be noted that objects, arrays and vectors are accessed by refer-ence while variables of primitive types are accessed by value. Referrefer-ences and values are stored in the CVmSymbol class (more about it in Chapter 4).
• Function calling and parameter passing: Whenever a function or a method is invoked, a new activation record is pushed into the stack. The activation record acts as container for:
– Actual parameters. – Local variables. – Temporary variables.
The record and all of its content will be removed at the end of the function call, for this reason it keeps tracking of the IP and the BP at calling time for them to be restored afterwards.
• Garbage Collection: The programmer cannot directly manage the memory, dynamic allocation is not permitted. Nevertheless, a subspace of the indexing space is solely dedicated to this purpose and it is executed in background in a transparent way. Whenever a new object is created, it is allocated on the heap and a reference is returned. A reference counting mechanism takes care of deallocating the object when it is not referenced anymore.
• Dynamic member dispatching: Since the S3D language is dynami-cally typed, an object’s type and possible members may be only known
at runtime. To each of the class’ members identifiers another, numeric identifier, is associated. Each class has its own dictionary which asso-ciates to each numeric identifier a pointer to the class member.
The virtual memory of an XVR process, showed in Figure 2.2, it can be divided in an area of statically allocated data, one of the code instructions and an expandable area for the stack and the heap.
Figure 2.2: Virtual memory of an XVR process
Stack
The Stack is used by the Virtual Machine to pass actual parameters to methods or functions or to pass their return value to the caller. Each function/method has an associated activation record which contains:
• Actual parameters • Local variables
• Return instruction address. The address of the instruction which imme-diately follows the function/method call
• Address to the base of the activation record of the caller, which will be used to restore the BP register at the call end.
Heap
The heap is the area of the virtual memory of a process where the memory is dynamically allocated. As previously stated, the programmer cannot directly allocated memory, it is instead taken care automatically and in a transparent way whenever an object, array or vector is created. The runtime frees the memory when possible thanks to the reference counting mechanism, which keeps track of how many entities holds a reference to the object and whenever the counter reach zero the object is destroyed and its memory deallocated. This mechanism does not ensure a complete cleaning of the memory.
Instructions Set
The instructions set is made up by:
• Operators to add and remove elements from the stack (push/pop). • Arithmetic/logic operators.
• Operators to invoke functions or methods which allocate a new activation record on the stack whenever a new call is requested.
• Jump and branching operators to modify the IP register. • Operators to access and modify instance variables of objects. Each operand can be:
• A value contained in the instructions itself. • A constant.
• A register.
• A memory location.
Interrupts
In real machines, interrupts are used to notify to the processor an I/O device’s request. In the XVR Virtual Machine, interrupts can be used by the debugger
to notify whenever a breakpoint has been hit. The presence of interrupts is checked at each machine iteration and is asynchronous with respect to the instructions execution and to a single interrupts one or more handlers can be associated. The handlers will take care of the interrupt and eventually restart the process execution.
Exceptions
The machine can raise exceptions to terminate the execution whenever an operation is executed at runtime which would affect the further execution of the process. For example, whenever a method which does not belong to a particular object is called, an exception is raised.
Fetch-Decode-Execute Cycle
The VM’s main loop can be expressed by the following operations:
1. Fetch a new instruction to execute by accessing CS[IP] and assign it to CP
2. Decode the instruction contained in CP 3. Execute the instruction and update tht IP 4. Handle interrupts
2.2.4
The S3D Compiler
In order for a program to be executed, it needs to be translated in a format that the Virtual Machine can understand. This is the task of the compiler. The compiler takes as an input a series of sources and returns as a binary XVR the bytecode as output. At the time of writing, there is not a linker unit that merges more compiler units. The compiler can be conceptually divided in three different phases:
1. Front End: In this phases the sources are parsed, that is, the code is transformed in an intermediate form to be elaborated and structured.
2. Optimizer: The optimizer executes some optimization in order to im-prove code performance.
3. Back End: In this phases the bytecode is generated and is thus ready to be execute by the Virtual Machine.
The second and the third phases take as an input an Intermediate Represen-tation (IR) of the program, perform a series of operations on it and forward the new format to the next phase.
Figure 2.3: An example of a Blueprint EventGraph
The three phases are not executed sequentially in the compiler implementation but separating them conceptually make it easier to describe them and their subphases.
Front End
The Front End of the compiler handles the input sources. Its main task is to structure the information of the sources in an IR that can be used in the next phases. The IR in this case is an Abstract Syntax Tree (AST), which is a tree representation of the relations between the grammar rules. Usually not all of the production are kept, in this case the tree is called Concrete Syntax Tree. The front end may be divided in three subphases:
• Scanning: The scanner, or lexer or lexical analyzer, takes as an input a stream of characters and produces as an output a stream of strings associated to their syntactical category. Its main task is to aggregate characters and determine if such aggregation is a valid language token and define its syntactical category. This can be reduced to a pattern matching problem, solved by using regular expressions which could be represented as a finite state automata. The XVR scanner has been im-plemented by making use of Flex[40], a popular open source tool which generates a scanner staring by a set of regular expressions
• Parsing: The parser’s task is to recognize the syntax to assure that the written code belongs to the specified grammar. Its input is the IR given by the scanning phase, if such input is not a valid program generated by the grammar, the compilation terminates with an error. The S3D syntax is expressed by a context-free grammar. The parser in generated by
Bison[39], a popular open-source parser generator, Yacc[41] compatible.
Its output is an Abstract Syntax Tree[31] (AST).
• Contextual Analysis: Differently from the previous phases, this one’s task is to verify the correctness of the program that cannot be by means of a CF grammar. An example could be to verify the presence of the declaration of an used variable. In order to execute this checks, the compiler needs and uses a Symbol Table. This table is automatically build during the parsing phase, in the code associated to each grammar rule (syntax-directed translation). In fact this phase is executed at the same time as the parsing phase.
Figure 2.4: Scanning of an instruction line
At the end of this phase, an IR of the Three Address Code (3AC) type is generated. Each instruction is represented as a quadruple (Quad) <Operator A, Operator B, Result>.
Figure 2.5: Example of AST The optimizer will be better described in Chapter 4.
Back End
This phase focuses on the bytecode generation. Its task is to convert the IR received as input, in a bytecode which can be execute by the virtual machine.
2.3
The XVR Debugger
Even though the XVR debugger is directly integrated inside the XVR Virtual Machine it can be seen as a separated entity and has been in fact added at a later time. The role of a debugger is to help a programmer to find ”bugs”. A bug is a defect in the software. Usually the debugger is internally connected to the machine architecture on which the application is executed, to the language compiler and in the case of graphical debuggers, to the IDE. The debugger should always have some base principles:
• It should always provide true and valid information.
• It should always show to the user relevant information about the pro-gram such as source code information, activation record, thread states, variables values.
• Heisenberg principle: The debugger should always be a mere passive observer of the program and should not modify it in any way.
A debugger may make use of a symbol table in which case it is called a symbolic debugger. The symbols are provided by the compiler through a proper table. A symbol table is a common data structure used by the compiler to associate identifiers to variables, classes, methods and functions, and to each one its scope. It should me noted that the symbol table used by the compiler may not be the same used by the debugger.
A debugger may implement a Graphical User Interface (GUI) or merely provide a set of commands to communicate with it (textual debugger).
The most recent and modern debuggers implements a feature called ”reversible debugging” which lets the user move backwards in the execution of a program.
2.3.1
Implementation of the XVR Debugger
The XVR debugger is a textual debugger which is completely immersed into the XVR VM and the S3D compiler. It uses its own symbol table, different from the one used by the S3D compiler, and implements the basic debugging command such as:
• Step over and step in. • Start and Stop.
• Adding and removing breakpoint. • Activation records stack.
• State interrogation.
The debugger symbol table is defined as a JSON[44] object and allows to: • Given a bytecode instructions, retrieve the source file name, and the line
number which generated it.
• Starting from a code line number and a file name, retrieve a set of in-structions associated to it
• Retrieve the scope of an instruction of the bytecode. • Retrieve the global scope of the compilation unit.
• Given a class name, retrieve its scope.
• Serialize/Deserialize the whole symbol table.
Differently from a debugger, the compiler does not need to know and store the scope associated to a specific symbol after the parsing phase. The debugger may need this information to correctly retrieve the visible variables at a given time and their memory addresses. More details about the scopes will be given in Chapter 3.
The communication with the debugger may be achieved through command console or by socket communication. For this thesis a new communication system has been developed which allows the debugger to asynchronously com-municate with the IDE.
2.4
The Unreal Engine
The Unreal Engine, as the name suggests, is a graphic engine developed by the Epic Games team, and mainly used to develop games and 3D applications. Unreal offer a complete suite of development tools for both professional and amateur developers to deliver high-quality application accross PC, consoles, mobile, VR and AR. The engine has a strong and large C++ framework which can be used to develop any kind of Virtual Environment. It offers a large set of tools such as:
• A user-friendly, customizable and extensible editor with a built-in real-time preview of the application.
• Robust online multiplayer framework. • Visual effects and particle system. • Material Editor.
• Animation toolset. • Full editor in VR.
• A large number of free assets. • Audio engine.
• A Marketplace where assets and plugins can be shared and downloaded The possibility that the engine offers allow the developer to not worry about performance or portability as every project can be easily ported to almost any platform with few or no effort. The Editor’s UI offers a lot of features and different section such as: the Animation flowgraphs and the Blueprint event graph. The UI is meant for both programmers and designers.
2.4.1
Terminology
Figure 2.6: Every object in the scene is an actor
This subsection explains some of the the terms used later when describing the Unreal plugin implementation and structure[28]:
• Project: A Project is a unity that holds all the content and code that make up an individual game. item Objects: The base building blocks in the Unreal Engine are called Objects. Almost everything in the Unreal Engine inherits (or gets some functionality) from an Object. In C++, UObject is the base class of all objects; it implements features such as garbage collections, metadata (UProperty) support for exposing vari-ables to the Unreal Editor, and serialization for loading and saving. • Classes: A Class defines the behaviors and properties of a particular
entity or Object used in the creation of an Unreal Engine game. Classes are hierarchical as seen in many programming languages and can be created in C++ code or in Blueprints.
• Actors: An Actor is any object that can be placed into a level. Actors are a generic Class that support 3D transformations such as translation, rotation, and scale. Actors can be created (spawned) and destroyed through gameplay code (C++ or Blueprints). In C++, AActor is the base class of all Actors.
• Components: A Component is a piece of functionality that can be added to an Actor. Components cannot exist by themselves, however they can be added to an Actor, giving it access to the Component func-tionalities. They could be seen as modules which any entity may imple-ment.
• Pawns: A subclass of Actor and serve as an in-game avatar or persona, for example the characters in a game. Pawns can be controlled by a player or by the game’s AI, in the form of non-player characters (NPCs). • Character: A Character is a subclass of a Pawn Actor that is intended to be used as a player character. The Character subclass includes a collision setup, input bindings for bipedal movement, and additional code for movement controlled by the player.
• Level: A Level is a user defined area of gameplay. Levels are created, viewed, and modified mainly by placing, transforming, and editing the properties of the Actors it contains.
• World: A World contains a list of Levels that are loaded. It handles the list of Levels and the spawning/creation of dynamic Actors (created at runtime).
• GameModes: The GameMode Class is responsible for setting the rules of the game that is being played.
• GameInstance: The GameInstance class contains information shared through all levels within a world. It’s the first entity created when a game session starts and the last one to be destroyed when the game ends.
2.4.2
The Blueprint Visual Scripting language
As already mentioned earlier, the Unreal Engine offers a new, different way of defining the game core logic and its entities’ behavior. The Blueprints Visual Scripting system is a complete gameplay scripting system based on the concept of using a node-based interface to create gameplay elements from within Unreal Editor. It may be used to define object-oriented (OO) classes or objects in the engine. Objects defined by using the Blueprint scripting language are often referred as ’Blueprints’. The system itself is designed to be powerful and flexible, and is mainly aimed to designer and graphics with close to no C++ programming background (for this reason the Epic Team is pushing toward its development). It especially gives the possibility to define functions and objects which can be directly spawned as nodes in a Blueprint graph.
Figure 2.7: An example of a Blueprint EventGraph The purpose of Blueprint is generally:
• Extending classes
• Storing and modifying default properties • Managing subobject instancing classes
