Carmelo Ardito
1, Paolo Buono
1, Maria Costabile
1, Danilo Caivano
1,
Giuseppe Desolda
1, Rosa Lanzilotti
1, Maristella Matera
2, Antonio Piccinno
11 Università degli Studi di Bari Aldo Moro, Bari, Italy - {name.surname@uniba.it} 2
Politecnico di Milano, Milano, Italy - maristella.matera@polimi.it
ABSTRACT
Information and communication technologies have a great potential to increase awareness and appreciation of cultural heritage. In the last decade, at the Computer Science Department of the University of Bari (Italy), we have been working to the design, development and evaluation of software systems supporting people during their visits to historical sites and museums, with the goal of improving the overall user experience. We focused on pervasive games to offer an engaging way to experience the cultural heritage site. This article briefly describes some games we developed and presents some of our new recent approaches. Specifically, we are investigating possible use of the Internet of Things (IoT) technology, as well as Mixed Reality, in order to ensure a more engaging visit experience and to increase the appropriation of CH contents by visitors.
KEYWORDS
Internet of Things, Serious games, Mashup
1. INTRODUCTION
Cultural heritage assets keep alive the history of a territory and of its inhabitants. Italy has one of the richest patrimony of historical sites and Cultural Heritage (CH) in the world. Information and Communication Technologies (ICT) can effectively provide new ways to support people’s understanding of CH and make visits to CH sites more engaging.
In this article we briefly illustrate the work we have performed and the systems that we have developed in the CH application domain together with some ongoing research. Specifically, in Section 2 we report about serious games mainly addressing pupils of elementary and middle school, while in Section 3 and Section 4 we describe novel approaches for improving the experience of people visiting CH sites, which exploit novel ICTs such as Internet of Things and Mixed Reality, respectively. Section 5 concludes the article.
2. SERIOUS GAMES FOR VISITING CH SITES AND LEARNING MORE ABOUT HISTORY
It is well known that pupils make up a large proportion of visitors to archaeological sites. However, traditional visits generate little interest in young visitors. New approaches must be proposed in order to capture student’s attention and to engage them. Gameplay showed to be successful. Pervasive games break away from the usual static paradigm of room play and go towards a more dynamic and social experience, by combining game and physical reality.Figure 1. Two different phases of Explore!, a pervasive serious game for visiting archaeological parks: play and debriefing.
In the last ten years, we have created serious games on different devices, not only to engage pupils in a more enjoyable visit to CH sites, but also to support them to reflect and deepen knowledge acquired through their active involvement during visits to these sites and followup activities at school. These games have been developed by setting up multidisciplinary teams that included experts in the CH domain, e.g., archaeologists, directors and employees of archaeological parks, software developers and HCI experts, schoolteachers, as well as end users, i.e., children and other types of visitors. We developed Explore!, a mobile learning framework incorporating a pedagogical technique in the form of the excursion-game, and a reusable technological architecture, which can be adapted to different historical sites by inserting appropriate content [1, 2]. We experimented it with several classes of middle schools and fifth grade of elementary schools visiting the archaeological park of Egnazia in the Apulia region. Special attention has been devoted to the design of a soundscape, which improves players’ navigation in degraded physical environments and enrich their overall experience [3]. The left image of Figure 1 shows pupils deciding the game strategy; the backpack brought by a team member contains two speakers that reproduce ancient sounds while moving in the site. The right image of Figure 1 shows the debriefing phase after the visit, during which students consolidate the acquired knowledge by discussing together on the paths they followed in the archaeological park and the items they found.
Other serious games aiming at fostering visitors’ learning of history have been proposed on large interactive displays. For example, History-Puzzle requires that pupils to complete puzzles of historical monuments/objects they saw during the visit to a cultural heritage site [22]. Time-Voyager is another serious game proposed on large interactive displays that invites pupils to organize photos of ancient artefacts according to a chronological order corresponding to the different historical eras [4, 23] (see Figure 2).
Figure 2. Pupils playing Time voyager, a serious game on large interactive displays.
3. MASHUP AND IOT TECHNOLOGIES TO ENHANCE THE VISIT EXPERIENCE
The discussions with the stakeholders involved in the multidisciplinary teams and various field studies provided inspiration about new ways of using current technology to support the work of the professional guides in organizing and conducting the visits, with the double goal of enhancing the visitors’ experience on the site and extending the overall experience beyond the specific visit. In the last years, also in collaboration with researchers of the “Politecnico di Milano”, we have been developing EFESTO, a platform that, according to End-User Development (EUD) [5-7] and participatory design approaches [8], supports end users, not technology skilled, to create personalized visits to CH sites [9]. End users become both information consumers and producers, and they are actively involved in the management of CH information. The platform, based on mashup technology [10], implements a new composition paradigm to allow end users to extract contents from heterogeneous (personal or third-party) sources, and compose Interactive Workspaces that satisfy their situational information needs and can be ubiquitously executed on different devices. The platform is general and is flexible enough to be adopted in different contexts of use [11]. In the CH domain, we have used to support the work of professional guides when accompanying visitors to archaeological parks, also allowing guides and visitors to collaborate among them to create new visit experiences [12].
(a) (b)
Figure 3. a) the guide introduces a visit to the archaeological park of Egnazia by using the EFESTO platform on a multitouch screen. b) the guide and visitors sharing information through the EFESTO platform deployed on the guide’s tablet and visitors’ smartphones.
The Internet of Things (IoT) has recently emerged as a technology able to enhance the access to CH collections. It promotes the use of smart objects, i.e., physical devices connected to the Internet, equipped with sensors and/or actuators and embedded software. Visitors of CH sites can touch, manipulate and interact with smart objects, for example to receive personalized information during the visit. Such interaction fosters the appropriation of CH content [13]: as recognized by some recent works [14]. Indeed, physical manipulation is an effective supplementary channel for the visitor to gather and store information, since they are enabled to activate real-world knowledge. In the CH domain, research has especially focused on technical aspects related to the development of sensor and actuator infrastructures [13]. A few works have studied how non-technical users can configure smart objects to define smart visit experiences. To fill this gap, we implemented our End-User Development approach within the EFESTO platform [15], in order to support users without skills in computer programming to configure the behavior of smart objects by creating Event-Condition-Action (ECA) rules [15, 24]. The platform empowers professional guides to enhance visit experiences by managing multiple synchronized smart objects. A peculiarity of this new version of EFESTO is the adoption of a model, called 5W, that introduces some specification constructs (Which, What, When, Where, Why) to build ECA rules coupling multiple events and conditions exposed by smart objects, and to define temporal and spatial constraints on rule activation and actions execution. Once a professional guide has defined the behavior of certain smart objects placed in the CH site, visitors are enabled to acquire more tangible CH content by interacting with the surrounding environment and the smart objects included in it.
4. EXPLOITING MIXED REALITY FOR CH SITES
Technological advances are fostering the proliferation of low-cost devices, including head-mounted displays, that make more real and natural the human-computer interaction experiences, as for example Virtual Reality (VR), Substitutional Reality (SR), Augmented Reality (AR), and Mixed Reality (MR). VR consists in an immersive multimedia reality that reproduces an environment simulating a physical presence in real or imaginary interactive worlds [16]. SR is a class of VR where every physical object surrounding a user is paired, with some degree of discrepancy, to a virtual counterpart [17]. AR is a live, direct or indirect view of a physical, real-world environment whose elements are augmented by virtual contents, which are not anchored to the real worlds and are not able to respond to each other [18]. Mixed Reality merges real and virtual worlds to yield new hybrid environments where physical objects and interactive digital artifacts are blended [19-20]. In other words, in VR digital environments shut out the real world, in AR digital content is on top of the real world, in MR digital content interacts with the real world. While MR is widely considered as one of the most promising technology, its potential is still unexploited. In the last few years, we have been working on the design, development and evaluation of MR applications based on Microsoft HoloLens1. HoloLens is a holographic computer built into a headset that lets users see, hear, and interact with holograms within a real environment. The real world is used as a 3D canvas in which to position and show virtual objects. Mid-air hand gestures is the most used technique to interact with holograms. MR devices are very appealing. However, we observed several issues related to interaction. In particular, as happened in the past with other novel technologies, companies and researchers spent most of their effort on improving MR hardware, giving much less attention to the design of heuristics, frameworks, guidelines and models driving the creation of MR worlds and the interaction with them. MR devices require specifically designed interaction techniques, but designers improperly adapt those of other traditional devices. For example, when the multi-touch displays appeared [21], users were asked to use the fingers as a mouse pointer, losing the potentialities offered by thesimultaneous use of multiple fingers or multiple hands. During the years, it became evident the need of designing ad-hoc interaction mechanisms. A similar story is happening MR-devices, which are often implemented by converting
traditional mouse interaction in mid-air gestures. In Cultural Heritage, MR could reveal a fruitful approach in a situation like the following.
The stored content can be exploited to rebuild location using MR and HoloLens. The tourists, through such technologies, could interact with the artifacts retrieved in situ or in other places, but relevant for the location, in order to study them in their original or expected context. Examples are lost items or reproduction of items exhibited in other places. This is a research that we aim to investigate in the near future.
5. CONCLUSION
In this paper we have described some research we have performed in the last decade. We have also presented new approaches that may have great potential to enhance the experience of visitors of CH sites by allowing them either to manipulate smart objects available at the site, exploiting IoT technology, or by permitting the interaction with virtual objects through Mixed Reality. Our ongoing and future research aims to improve the support CH sites experts in designing smart visit experiences. To this aim, we are currently investigating novel natural interaction paradigms to enrich smart objects with custom attributes, which can be used to create ECA rules in an even simpler and more powerful way.
6. REFERENCES
[1] Costabile M. F., De Angeli A., Lanzilotti R., Ardito C., Buono P., and T. Pederson. 2008. Explore! possibilities and challenges of mobile learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). ACM, New York, NY, USA, 145-154. DOI: https://doi.org/10.1145/1357054.1357080
[2] Ardito C., Buono P., Costabile M. F., Lanzilotti R., Pederson T., Piccinno A. (2008). Experiencing the Past through the Senses: An M-Learning Game at Archaeological Parks. IEEE Multimedia. vol. October-December 2008, pp. 76-81. ISSN: 1070-986X
[3] Ardito C., Costabile M. F., De Angeli A., and Lanzilotti R. (2012). "Enriching exploration of archaeological parks with mobile technology," ACM Trans. Comput.-Hum. Interact., vol. 19, pp. 1-30, 2012.
[4] Ardito C., Costabile M. F., and Lanzilotti R. (2010). "Gameplay on a Multitouch Screen to Foster Learning about Historical Sites". International Congerence on Advanced Visual Interfaces, Rome, Italy, 2010.
[5] Ardito C., Buono P., Costabile M. F., Lanzilotti R., and Piccinno A. (2012). "End users as co-designers of their own tools and products," Journal of Visual Languages & Computing, vol. 23, pp. 78-90, 2012.
[6] Costabile M. F., Fogli D., Mussio P., and Piccinno A. (2007). "Visual Interactive Systems for End-User Development: A Model- Based Design Methodology," IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, vol. 37, pp. 1029- 1046, 2007.
[7] Lieberman H., Paternò F., and Wulf V., Eds., End User Development. Springer, 2006.
[8] Schuler D., Participatory design: principles and practices. Hillsdale N.J.: L. Erlbaum Associates, 1993.
[9] Ardito C., Costabile M. F., Desolda G., Lanzilotti R., Matera M., Piccinno A., and Picozzi M. (2014). "User-driven visual composition of service-based interactive spaces," Journal of Visual Languages & Computing, vol. 25, pp. 278-296, 2014.
[10] Daniel F. and Matera M., Mashups - Concepts, Models and Architectures. Springer Series: Data-Centric Systems and Applications, XIX, 319. 2014.
[11] Ardito C., Bottoni P., Costabile M. F., Desolda G., Matera M., and Picozzi M. (2014). "Creation and Use of Service-based Distributed Interactive Workspaces," Journal of Visual Languages & Computing, vol. 25, pp. 717-726, 2014.
[12] Ardito C., Bottoni P., Costabile M. F., Desolda G., Matera M., Piccinno A., and Picozzi M. (2013)."Enabling End Users to Create, Annotate and Share Personal Information Spaces," in End-User Development - IS-EUD 2013. vol. LNCS 7897, Dittrich Y., Burnett M., Mørch A., and Redmiles D., Eds., ed Berlin Heidelberg: Springer, 2013, pp. 40-55.
[13] Mighali V., Fiore G. D., Patrono L., Mainetti L., Alletto S., Serra G., and Cucchiara R. (2015). "Innovative IoT-aware Services for a Smart Museum," presented at the International Conference on World Wide Web, Florence, Italy, 2015.
[14] Uckelmann D., Harrison M., and Michahelles F. (2011)."An Architectural Approach Towards the Future Internet of Things," in Architecting the Internet of Things, Uckelmann D., Harrison M., and Michahelles F., Eds., ed Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 1-24.
[15] Desolda G., Ardito C., and Matera M. (2017). "Empowering End Users to Customize their Smart Environments: Model, Composition Paradigms, and Domain-Specific Tools," ACM Trans. Comput.-Hum. Interact., vol. 24, pp. 1-52, 2017.
[16] Burdea G. C. and Coiffet P. Virtual reality technology. London: Wiley-Interscience, 1994.
[17] Simeone A. L. and Velloso E. (2015). Substitutional reality: bringing virtual reality home. XRDS 22, 1 (November 2015), 24-29. DOI: https://doi.org/10.1145/2810044
[18] Azuma R. T. (1997). A survey of augmented reality. Presence: Teleoperators and virtual environments 6(4), pp. 355-85.
[19] Milgram P. and Kishino F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems 77(12), 1321-9.
[20] Ohta Y. and Tamura H. 2014. Mixed Reality: Merging Real and Virtual Worlds. Springer Publishing Company, Incorporated. [21] Ardito C., Buono P., Costabile M. F., and Desolda G. (2015). Interaction with large displays: a survey. ACM Computing Survey
47(3), pp. 1-38.
[22] Ardito C., Lanzilotti R. (2011). An EUD Approach to the Design of Educational Games. International Journal of Distance Education Technologies, vol. 9(4); pp. 25-40, ISSN: 1539-3100, doi: 10.4018/IJDET.2011100103.
[23] Ardito C., Costabile M. F., Lanzilotti R., Desolda G. (2013). Integrating traditional learning and games on large displays: an experimental study. Educational Technology & Society, vol. 16(1); pp. 44-56.
[24] Ardito C., Buono P., Desolda G., Matera M., From smart objects to smart experiences: An end-user development approach, International Journal of Human-Computer Studies, ISSN 1071-5819, https://doi.org/10.1016/j.ijhcs.2017.12.002.