Developing technological solutions to assist children with ASD with application to real-life and diagnostic scenarios

(1)

P

H

D T

HESIS

Developing technological solutions to

assist children with ASD with application

to real-life and diagnostic scenarios

Author:

Mariasole BONDIOLI

Supervisor: Prof.ssa Susanna PELAGATTI

Prof. Stefano CHESSA

A thesis submitted in fulfillment of the requirements for the degree of Doctor of Computer Science (XXXII Cycle)

in the

Computer Science Department

(2)

1 Abstract

The thesis aims at presenting the development of technological solutions aimed at assisting in two different aspects (one diagnostic-oriented and one oriented to real-life problems) in the life of children with autistic spectrum disorders. The development is framed into a process of “adaptive proto-typing”, which is here refined to meet the specific needs of software devel-opment arising in the considered application domain. The first application domain is motivated by the fact that children with autism perceive sensory experiences differently and have problems accepting unknown social con-texts. A notable case is that of dental care setting, where there are many strong sound-visual stimuli, which are usually not found in known environ-ments such as home, school or therapy session rooms. This usually upsets a child with autism, often forcing dentists to administer an anesthetic even to perform normal dental hygiene. In recent years, technology-enhanced sys-tems and apps have been developed to help people with autism adapt to new contexts and cope with distressing social situations. In the thesis we discuss the development of a web application (MyDentist) which provides dentists, parents and children with a set of resources to teach children with autism the proper oral procedures at home and correct behavior during dental visits. MyDentist includes multimodal games and activities to lighten children’s tension and to make them familiar with dental procedures. Games and ac-tivities can be personalized according to child’s needs and preferences. A unique software environment allows professionals to take and collect photos and videos during dental visits and make them accessible to parents and care-givers in the private profile of each child. In the second applicative domain, we considered the use of Internet of Things devices to monitor the behav-ior of children with autism in a niche application, namely the observation of the children during their games, which is a common test performed by specialists to diagnose autistic spectrum disorders in an early stage. Specif-ically, the thesis describes the development of an IoT system that employs miniaturized sensors and data fusion algorithms based on machine learning to identify automatically the movements applied to the toys by the children. Starting from this two cases we propose a methodological solution to face many of the typical challenges that several e-health scenarios involving ASD needs could pose to the software designer.

(3)

(4)

Acknowledgements

Grazie alla professoressa Susanna Pelagatti e al professor Stefano Chessa, per avermi accompagnata e sostenuta di fronte a tutte le sfide, piccole e grandi, che questi tre anni ci hanno riservato. Per avermi spinta sempre un po’ più in là, oltre ciò che pensavo potessero essere i miei limiti, dotandomi di strumenti fondamen-tali per riuscire ad arrivare ad oggi con la certezza di avere fatto qualcosa davvero. Grazie a tutte le professioniste e i professionisti con cui abbiamo collaborato in questi anni. Tutte e tutti coloro che, con i loro saperi e le loro esperienze, tutti di-versi ma in egual modo importanti, sono riusciti a rendere delle idee, delle ipotesi, qualcosa di concreto e reale, indirizzandomi con scambi, contributi e confronti importantissimi non solo per il lavoro che è stato fatto, ma anche per tutto ciò che verrà. Quindi grazie a Claudia e Marina Buzzi, a Caterina Senette, a Fabio Uscidda. Grazie alla professoressa Maria Rita Giuca. Grazie al dottor Antonio Narzisi e alla dottoressa Martina Pinzino. Alle logopediste Valentina Semucci e Benedetta Vagelli.

Grazie a (la dottoressa) Francesca. Compagna di avventure, di sperimentazioni, di soddisfazioni. Di lacrime e risate. Grazie all’ambulatorio Autismo, a chi si `e aggiunto e aggiunta negli anni, a chi ha imparato a conoscerlo e ad appassionarcisi insieme a noi, ai nostri costumi, alle nostre follie.

Grazie a Marco, Margherita, Dario, Yvonne, Camilla che hanno incrociato pezzi della loro strada universitaria con questa avventura. Imbattutisi nel loro per-corso di studi in questo strano mondo delle tecnologie assistive per l’autismo, cos`ı poco convenzionalmente accademico, ma cos`ı avvincente.

Grazie all’associazione Autismo Pisa e Valdera, alle famiglie e, soprattutto, ai ragazzi e le ragazze, i bimbi e le bimbe con autismo che hanno frequentato le nostre sperimentazioni e che tutt’ora continuiamo a incontrare nei nostri percorsi attivi. Sono troppe da scrivere le cose che ho capito in questi anni grazie a voi. Di me e di ci`o che `e intorno a me.

Grazie a chi e a cosa ogni giorno in tutti questi anni mi ha aiutato e mi aiuta a ricordare cosa `e importante e cosa no. Che da soli possiamo fare qualcosa, ma insieme possiamo fare molto di pi`u. Che per cambiare qualcosa prima di tutto

(5)

dob-biamo essere pronte e pronti a mettere in discussione noi stessi.

Grazie a chi c’è, vicino e lontano, famiglie nel senso più o meno tradizionale del termine, amici e amiche pronte/i a festeggiare ad ogni correzione, ad ogni con-segna, ad ogni nuova versione di questo lavoro, ad ogni nuovo progetto. A chi si è ascoltato tutte le avventure di questi anni, al bar o in lunghe telefonate. A chi, tra una corsa in bici e un’altra, un capitolo ed un altro, senza accorgermene, mi ha fatto sentire sempre più a casa. Anche qui. Anche quando meno me l’aspettavo.

(6)

5 Different, not less

(7)

(8)

List of Figures

3.1 Our classification matrix . . . 35

3.2 ICT sytems in the analyzed literature . . . 36

4.1 MyDentist software design methodology . . . 61

4.2 Multimedia activities proposed during the pilot study . . . 68

4.3 Example of exploited Cognitive Learning Game. . . 69

4.4 A digital social story . . . 69

4.5 A videomodelling-based video about teeth brush . . . 70

4.6 Pilot test application guidelines . . . 72

4.7 Dentist UI . . . 77

4.8 Patient UI . . . 78

4.9 Selfie Mode tablet during the visits . . . 79

5.1 Our MoVEAS design process . . . 83

5.2 Detail of sensor installation in the truck . . . 87

5.3 The particle photon with the accelerometer sensor. . . 87

5.4 The circuit for battery charging and the on/off switch. . . 88

5.5 Orientation of three axis accelerometer . . . 88

5.6 The data fusion algorithm . . . 89

5.7 Motion Capture prototype UI . . . 90

5.8 Child1 and Child2 play session graphs . . . 92

5.9 Sequential Keras model structure . . . 94

5.10 MoVEAS Interface . . . 96

5.11 Recording session page . . . 96

5.12 Video and data synchronization . . . 97

5.13 Neural network training MoVEAS page . . . 98

5.14 MoVEAS play session . . . 99

6.1 Our refinement of the adaptive prototyping model. . . 102

6.2 Using MyDentist after the visit . . . 112

6.3 Graphs T0-T1 answers question 16 . . . 114

6.4 Modification in the ECG indices. . . 115

6.5 Modification in SCR indices . . . 116 11

(13)

7.1 Hevner three cycle model . . . 122

B.1 SUS graph 1 . . . 131

B.2 SUS graph 2 . . . 131

C.1 Parents collaborating during MyDentist visits . . . 133

C.2 Questionnaire retrospective form . . . 134

C.3 Questionnaire Multimodal Form . . . 135

C.4 Questionnaire Multimodal Form pt2 . . . 136

C.5 Questionnaire present form . . . 137

C.6 Graphs T0-T1 answers question 3 . . . 138

D.1 The IFC-CNR ECG chest belt . . . 141

(14)

List of Tables

4.1 UI Guidelines for Users with ASD . . . 63

4.2 Participants in the initial expert interviews . . . 64

4.3 MyDentist usability . . . 75

5.1 Participants in the initial expert interviews MoVEAS . . . 85

6.1 UI guidelines . . . 107

6.2 Retrospective and present questionnair results . . . 113

6.3 Confusion matrices of MoVEAS Neural Network first test . . . . 117

6.4 Accuracy, Precision, Recall and F-Measure values . . . 117

A.1 Overview of the state of the art . . . 127

B.1 Users Dentists SUS results. . . 130

B.2 Users Psycologists SUS results . . . 130

(15)

(16)

Chapter 1

Introduction

Autism is a spectrum of neuro-developmental conditions that seriously compro-mises the way in which people communicate with and relate to other people and, in general, their ability to make sense of the world around them. In recent years, increasing attention has been focused on the use of ICT to help autistic people in several contexts (school, home, therapy) [104].

This interest is motivated by the rapidly growing prevalence of autism and by the positive effects of the interaction of autistic people with computers and predictable environments [135, 181].

In 2010, the overall prevalence of ASD measured by the network of Centers for Disease Control an Prevention in the US was 14.7 per 1000 (one in 68) chil-dren aged 8 years [69]. A more recent study [93] estimated the prevalence of ASD in 1 in 87 in school-aged children in the province of Pisa (Italy). As a matter of fact, prevalence has increased by more than an order of magnitude in the past 15 years. Genetic research indicates a very complex scenario, with possibly hundreds of genes implicated, and a genetic diagnosis is currently impossible except for a few specific genetic configurations [30]. Consequently, at present, the only possi-ble intervention is based on early diagnosis based on clinical observation (usually, in Italy at around age 1.5/2 years), followed by intensive treatment. On the basis of group-level data, research suggests that behavioral programs implemented as early as possible and in an intensive manner can be effective in improving cogni-tive, adapcogni-tive, and social–communicative outcomes in young children with ASD [224, 227].

On the other hand, no matter how early treatment is started, none of the present techniques can resolve autism completely. Autistic people can acquire many abil-ities, become more autonomous, work and even live alone. However, many autis-tic traits remain and all the environments surrounding an autisautis-tic person (family, school, hospital, co-workers etc.) must learn how to deal with them. Moreover, autism is characterized by a remarkable heterogeneity at the behavioral level, with substantial individual differences [257]. Consequently, in the near future, a very large number of people without specific training on autism will need to cope with

(17)

many different kinds of autism in normal everyday activities.

This may become a serious social problem in the forthcoming years, requiring great effort in various fields to address it correctly and effectively (not surpris-ingly the Oscar-winning documentary titled Life, Animated – director Roger Ross Williams – celebrates the problems encountered by an adult ASD person in his everyday life).

1.1 Motivations

This thesis explores the potential of a generalization of an ICT development pro-cess centered on the design and testing of systems aiming to improve the effective-ness of the current strategies adopted in ASD every day challenges.

In particular, the model that we exploit in these three years ad been refined and verified with its application to two important problems concerning the improve-ment of the quality of life of an autistic person and of its family:

the ability to identify the ASD in an very early stage (early diagnosis) and the ability to manage the sense of anxiety felt by ASD people when they are exposed to a unknown or distressing environments (reducing anxiety).

These two cases study have been identified with ASD specialists of the hospital of the IRCCS Stella Maris in Calambrone (Italy), in over three years of coopera-tion. Note that the hospital is a main center in Tuscany for ASD in children, hence our work have addressed primarily children, although some of its aspects can be extended to other age groups. Note also that, although very important, the early diagnosis and the reducing anxiety problems cover only two aspects of interven-tion, leaving other aspects apart (for example treatment). This because these two problems allow a direct introduction of ICT technologies, due the way in which they are managed by the specialists.

Among the two, the first one identified with the specialists was the reducing anxiety problem, because it currently involves all children under treatment, no matter of the gravity of the syndrome. This is a problem in a very general sense, because the range situations and environments to which ASD children can be ex-posed is rather wide. For this reason, as a specific case of study, we focused on a scenario of dental care. This scenario is particularly challenging and relevant for autistic people and for their families because, usually, dentists are not trained for autism. Nevertheless, the dental health of people with autism presents many challenges, since they usually perceive sensory experiences differently and have problems accepting unknown social contexts. In a dental care setting, there are many strong sound-visual stimuli, which are different from these encountered in any other setting. This usually upsets an autistic patient, often forcing dentists to administer chemical sedation in order to be able to complete dental work.

We addressed this problem exploiting the positive attitude of autistic people towards technology which is recorded in the literature and has been used to simplify some aspects of oral care with positive results [127, 23]. Thus, in our requirement

(18)

1.1. MOTIVATIONS 17 analysis phase, we started a cooperation with dental clinic (Clinica Odontoiatrica) of the hospital ”Santa Chiara” in Pisa and analyzed the problem in a pilot study including dentists and clinicians from Stella Maris. This study opened up very interesting research directions.

To start, the potential of using ICT to reduce anxiety using photos, videos, games and other digital material was completely evident from the study [37]. The study also underlined the difficulty of the dentists of managing large amounts of multimodal material which needs to be personalized for each patient without an ICT support.

This lead us to the development of an application able to organize all the per-sonalized material which was then tested and validated in a study involving 120 children in the Santa Chiara Hospital. During this study, the clinicians of IRCCS Stella Maris measured the improvements in dental care attitude, quality of life and the impact of using ICT versus non digital approaches.

Analyzing the experiment with the clinicians it become apparent that other devices could be used to have quantitative measures of the anxiety, for instance using ECG or heartbeats. Thus, we settled up a new experiment using wearable sensor devices to record measurement during each visit which is still going on with the new patients of the clinic to record precisely the behavior of the stress curve during the dental treatment, from the first approach to the sealing of molars. The approach followed to accustom ASD patients to the characteristics of den-tal surgery and treatment could be actually applied to other different distressing environments. Thus, with the help of clinicians, we defined a general methodology exploiting ICT to reduce anxiety in different environments. The methodology will be used in the Botanic Garden (Orto Botanico) of the University of Pisa in a project startin at January 2020, to prepare ADS persons to the visit of the Garden and to reduce their stress while visiting.

Concerning instead the other case study (early diagnosis), following the sug-gestion of the clinicians, we explored the use of IoT and wearable devices to obtain a better understanding of the movements of autistic children both to improve the early diagnosis and to understand their behavior in common everyday contexts. In particular, we analyzed the problem of obtaining relevant information observing the young children while they play. This observation is a test often adopted by specialists to identify in an early stage potential autistic spectrum disorders (ASD) so to trigger specific diagnostic tests [21, 193].

The observation of the play is motivated by the fact that ASD children exhibit a behavior in the games usually different from non ASD children. For example, they often carry toys around without playing, and/or they insist in games with very rigid rules, and they often lack social play and imagination in fiction games. At the state of the art, the observation of the play of children requires specialist knowledge and it is conducted in clinics. This, however, is not the ideal setting for the observation of an ASD children since the toys are not theirs and the environment is unusual. Several recent studies in internet of things (IoT) concerning the recognition of hu-man activities by means of wearable devices [106] suggest that “intelligent”

(19)

de-vices embedded into toys and connected to the internet may be trained to remotely provide information about the play of the children in their own environments (at home or at school). Such use of IoT technologies would enable to new protocols in the early identification of red flags concerning ASD, since they may be used without the presence of a specialist (who would have access to this information from the clinic), and they would allow to cover a wider range of ages and a larger number of children.

Thus, we designed an IoT system specific for the detection of movements of toys for children using a miniaturized sensor with accelerometer, gyroscope and magnetometer to be embedded into toys for young children, and on a basic data fusion algorithm aimed at producing information about yaw, roll and pitch of the toy [152]. A preliminary test carried on on two children at IRCCS Stella Maris showed the potential of the movement capture system and lead us to improve the first system using an improved data fusion algorithm based on machine learning that operates over the data produced by the previous data fusion algorithm to clas-sify some typical movements of the toy during a play (like move forward, move backward, etc.) [39]. Tests on the improved system were performed on a set of 10 ASD children aged 8-10 and on a group of non-ASD children on a primary school in Pisa. Our two cases study could appear completely devoid of connection between them, but in reality they share a very important starting point: they are addressed to interact with a very special special world, the world of Autism. In this sense the process that drove the development of the two different software should be fitted to answer to very specific needs.

The needs that emerged from the two different research contexts were essen-tially the same: lack of clear starting requirements, several different professional figures that need to follow and contribute to the system design process, in some cases difficulties to interact with the end users.

For these reasons, the software development process followed in this thesis, applied in the same way to both our problems, can be considered an application of adaptive prototyping [129] for both problems considered, with some specific adjustments to meet some specific development aspects arising typical of e-health applications. In fact, these applications address a wide range of stakeholders (rang-ing from medical specialists to insurance, to public health systems to patients or disabled etc.), may drastically change the established medical practices (and thus require a validation of the new practices and approval from ethical committees), may make use of cyber-physical devices (and thus include aspects of hardware de-sign) and, on the software side, keep all the complexity of any other (complex) software system that thus require the application of the best SE methodologies.

From the point of view of the software designer, the e-health scenario (and thus also the problems addressed in this thesis) pose several challenges. The col-lection of requirements, for example, addresses a variety of stakeholders, and in some cases the interaction with some of them must be mediated. For example, when addressing end users with a specific disability, the interaction between sys-tem developers and end-users needs to be mediated by domain experts (medical

(20)

1.2. STRUCTURE OF THE THESIS 19 specialists) both for the collection of requirements and for the validation and eval-uation of the system.

Furthermore, the stages of development should also address the validation of the medical practice in the use of the developed technology and not just the vali-dation of a software product, which make the design of the software development process blurred with the process of developing the medical practice itself (and pos-sibly also of the hardware, if the application makes use of cyber-physical devices). To this purpose the system development passes through pilot studies whose results may however be affected by several human factors [194].

In the specific case of this thesis, the design of both experimental application (MoVEAS and MyDentist) led us through a software development process exposed to many of the above mentioned challenges, including the interaction with ASD children and the other stakeholders for the collection of requirements, and the de-sign of a preliminary pilot study aimed at testing the developed applications in a controlled but meaningful, real condition to pave the way for a subsequent large scale experimental study for the general validation of the methodology.

Specifically, the preliminary stage of requirements collection was conducted by analyzing the targeted literature and collecting experts’ interviews. This prelimi-nary stage led us to the definition of a first set of guidelines for the development of the applications, which, however, needed to be validated with users in realis-tic conditions. The first prototype of both applications was designed according to these guidelines and tested in a preliminary, small scale pilot study, aimed at testing the reliability and usability software and at validating the guidelines. The results of this first experimental study provided us enough information to proceed with a revision of the guidelines, that in the revised form are one of the main results of this work.

At the current stage of development, the application of MyDentist is ready for a large scale experimental study, while the MoVEAS application is in a earlier stage of development and it is ready to undergo a controlled pilot study aimed at making it more reliable and at testing the medical practice associated to it.

1.2 Structure of the thesis

As emerges from the previous section, the main contribution of the thesis it is an innovative effort to generalize and systematize a development process of ICT systems for special end users, such as users with ASD.

The validation and the further refinement of our model moves then along two application axes, namely exploiting ICT to help ASD people in reducing anxi-ety when exposed to unknown or distressing environments and to improve autism knowledge and diagnosis.

Along these two axes we adopted the pragmatic approach of identifying and developing the ICT solutions most suitable to solve the specific problems encoun-tered. For this reason, the thesis is actually structured in two almost independent

(21)

parts, which have a common root in the motivations and in the general methodol-ogy, but that then develop by following two independent lines of research.

1.2.1 Two case study, a general framework: our development model

From the beginning of these three years of research, taking the first steps in our first case study (that led to the development of MyDentist see??), it appeared clear that the standard models of software development could not entirely satisfy the needs implied by the complex research context that we were facing.

On the other hand, we were aware that MyDentist was just one of many con-text in which the effectiveness of the innovative approach we were studying could be verified. Therefore, having a solid model of development, reproducible, cus-tomizable and testable, has immediately resulted as an essential prerequirement to face every current and following application challenge in our research field, and, looking at the existing literature, we could not find anything that could completely satisfy our needs.

In this perspective the refinement of prototyping model, described as our first contribution in chapter 6), could be considered the result of contamination between an existing model (the prototyping model) and the considerations arose from each phase of our application studies.

Defined as an attractive idea for complicated and large systems for which there is no manual process or existing system to help determine the requirement, the prototyping model had been considered a fitting starting point to our model defi-nition. Moreover, this model is described by Jalote [129] as an effective method of demonstrating the feasibility of certain approaches, based on an high level of interaction with the users for each step of the developing process, all features that we had considered very relevant to our research purposes.

This appeared even clearer when we started the collaboration with the neu-ropsychiatric clinic Stella Maris on our second case study: the development of an IoT system to monitoring the children play for ASD diagnosis purposes.

As in the case of MyDentist, since the first phases of this second development process, the lack of specific precedents in literature or guidelines suitable for our case, the complexity of a requirement analysis phase that involved different stake-holders with different needs (some even with special needs), the implication of a multidisciplinary testing of the developed support (and its usage protocol), are all factors highlighted by the first phases of the case study and reinforced the idea that a model like the one with the features of the prototyping model could have been the basis to systematize the unexpected that we would have encountered on our road.

It is by placing them in this frame result that we can therefore read the two axes of MyDentist and MoVEAS not only as two independent case studies, but also as the first two on field applications and validations of an innovative way to conceive the software development process (and its users needs) and, at the same time, the first concrete results at the basis of the refinement of the model itself.

(22)

1.2. STRUCTURE OF THE THESIS 21

1.2.2 Reducing anxiety in everyday activities

In the first case study, we will describe a methodology using ICT to bridge the gap between unknown, new or distressing environments which can be encountered in normal everyday activities and autistic persons. For instance, we would like to ad-dress anxiety when visiting a new museum, playing a new sport, flying in holiday or going to the dentist. Usually, normal everyday activities do not require particu-lar preparation for non-autistic people and are usually run by people/professionals without a specific training in autism. We will refer to this group as non-trained professionals (NTP) in the rest of this thesis. NTP usually need a clear protocol and support to manage interacting with autistic people and reducing their anxiety and problematic behaviors during an activity.

Our goal is to outline a clear, replicable methodology to use ICT to reduce stress in different everyday contexts and provide some evidences in real case stud-ies.

In the literature, several studies underline the positive effect of daily use of ICT applications on people with ASD, not only in learning and autonomy contexts but also in social situations [55, 176, 238, 199, 10]. The effectiveness of ICT in teach-ing different skills has been studied by many authors with very interestteach-ing results. Several studies have proved the advantages of using video modeling techniques [55, 256], augmented reality [50], software to facilitate communication (based on Augmented Alternative Communication – AAC) [158] and applications for the de-velopment of social skills [14]. All these studies report that people with ASD like the innovative educational approach introduced by ICT. Using technology, they can avoid typical issues involved in human interaction, such as impatience, un-predictability of people’s behavior, misunderstanding emotions, irony, figurative language, and feelings of inadequacy or bad error management. Learning new skills is actually facilitated by machine-automation characteristics such as the lack of emotional tone, the exact repetition of the activities, the constancy of auditory and visual stimuli and the use of reinforcements. For this reason, in recent years, many technology-enhanced systems and applications have been proposed to help people with autism adapt to new contexts and to cope with distressing social sit-uations, for example to face social interaction with the schoolmates [122], find a solution in problematic social conflicts [27] or deal with medical intervention [80]. However, the current state of the art in using ICT applications for easing normal everyday activity for ASD people presents some issues. First, using an ICT appli-cation correctly to help a specific autistic person in a context may be a very difficult task. This is due to the extreme heterogeneity of the autistic syndrome: each per-son with autism is different and has individual special needs. Skills and abilities may vary from low-functioning adults able only to interact using concrete objects to high-functioning people with very high IQ values. This requires the adoption of a personalized holistic intervention taking into account behavior, skills and social abilities. The use of visual stimuli (familiar images and objects) as well as the massive use of reinforcements are motivating elements. This may require a high

(23)

level of customization of the ICT application taking into account the characteristics of each person (e.g. using personalized videos, avoiding certain colors or images, carefully tuning automatic rewards. . . ). Unfortunately, not many applications pro-posed in the literature can be easily personalized. For instance, [80] proposes an interesting mobile application to teach the basic first aid skills using the same ac-tivities and pictures regardless of the age and level of social and communicative impairment of the ASD users.

Second, in general, flexible applications with many degrees of freedom may rapidly become too cumbersome for a NTP if they expose too many variables and actions in order to be tuned to obtain a personalized version.

Finally, IoT devices can be wear or placed in the environment to have useful information that can help to measure anxiety and to personalize an ICT application. Unfortunately, NTP usually are not trained to analyze low level sensor output and to relate it with the parts of the activity that need to be reorganized to reduce the level of anxiety. Thus, sensors should be integrated transparently in an ICT application to provide NTP with useful information related to the activity.

In the second case study, we will outline a general strategy to address these problems, present a prototype application and discuss some preliminary results obtained in two real case studies.

1.2.3 Using IoT devices for capturing information and early diagnosis

In the second part of this thesis, we use IoT devices to design a system specific for the detection of the movements of toys during children play. As stated before, ASD children live in a different way the first learning process of his/her childhood. This is the reason why the observation of the play in children suspected of ASD is widely used in diagnosis. However, such an observation must be conducted in clinics and requires an observer with a deep knowledge of ASD. Consequently, it suffers from some limitations: it is conducted in an unusual environment for the child and uses toys which are brand new for him/her. This may lead to sensible differences with the common behaviour at home with well known toys, and this can hide important information and clues for therapists and caregivers.

The connection between the movements of children interacting with the daily life objects and the signals of a potential early diagnosis of autism has been inves-tigated and confirmed by previous research projects. In recent years, the medical interest in the study of motor impairments in the ASD people has been renewed. At the same time, the growing interest in ICT for easing several aspects of diagnosis and treatment of ASD people has led to a rapid increase of the experimental work based on innovative supports to clinical and educational path totally fitted on the autistic condition.

On the clinical side, recent research, conducted also with the help of family video recording, highlight how the compromises on the motor system in autism are frequently observable very early in children who will subsequently receive a diagnosis of ASD [213, 217]

(24)

1.2. STRUCTURE OF THE THESIS 23 In this field, we can even find some studies supporting a clinical approach focused on the direct observation and evaluation of the children’s movement. For instance, [45] proves how in young ASD children (mean age 33 months) the lack of social engagement and attention could be traced to their tendency to the prevalent restricted object use measured during the experimentation. In another work, with the observation and interpretation of atypical visual exploratory behaviors towards inanimate objects (AVEBIOs) Mottron et al highlight the predisposition to look to the objects with lateral glances, as a specificy more common in children with ASD than in children with typical development [178]

In the multidisciplinary perspective of our work, from the technological point of view we can identify a limited number of previous research exploiting the IoT (Internet of Things) to monitor and collect information about the ASD children ac-tivities, no ones specifically focused on early diagnosis purposes. In [109], Good-win et al. use a sensor system to investigate the automatic detection of stereotypical children movements. Their classifier algorithm trained to recognized repetitive be-haviors obtains good results in their experimental session with six children wearing three wireless accelerometer at school and during therapy.

Plotz et al. combine a device with a 3D accelerometer and a microcontroller with machine learning techniques, to demonstrate the accuracy of the machine learning approach to the detection and classification of children anomalous be-havior [210].

In other studies, the researchers investigate the recognition of stereotypical movements by the use of a Kinect [134, 108]. In particular, Kang et al use a ma-chine learning approach to recognize the movements filmed by the Kinect camera and test its performance on 12 actors performances of three different stereotypical movements [134].

Differently from this work, our goal is to build an IoT system which is non-invasive, easy to use for non-experts and able to work in real life contexts. Research on activity recognition with IoT devices is already advanced and many commercial products are widely available even if most of the existent work is focused on appli-cation fields concerning fitness or ambient assisted living in general [8]. We used similar technology to design MoVEAS, a system for remote non-intrusive control of children movements. From a technological point of view, the adoption of IoT devices for this specific application poses significant challenges in the selection of a suitable set of sensors, in the algorithms for the analysis of sensor data (the data fusion process), in the protocols and platforms form the management of the data [94] and on the tuning of the neural networks for action recognition. In a part of the thesis we describe MoVEAS and present the results of a pilot studies with ASD and non-ASD children, carried out in cooperation with the hospital of the IRCCS Stella Maris in Calambrone (Italy).

(25)

1.3 Outline

The thesis is organized as follows. In Chapter 2, we present some preliminary concepts on autism and ICT.

Chapter 3 describes the literature related to ICT and autism proposing a new taxonomy to analyze the wide and extremely fragmented landscape that character-izes the state of the art in the field. The unprecedented taxonomic system presented in this chapter is not only a useful tool for describing the state of the art of this the-sis. It has been exploited to return results and specific gaps of our research field starting from a broad and well-structured overview of the texts analyzed, but, more generally, it has been outlined as an innovative analytical taxonomy based on a three-dimensional classification. In this perspective, this taxonomic work helped us to approach our two cases study, facilitating the initial analysis of the current lacks and challenges of our intervention fields. Exploiting it to classify the state of the art, it allows the user to observe what has been produced up to his work on the basis of a multidisciplinary combination of elements that we could not find in any previous analysis. For this reasons we hope to publish it in the future.

Next, Chapter 4 describes the process to develop and test our first software to support the anxiety reduction in real life environments, with the presentation of our first case study: a dental clinic. As described in our works [37, 38] to reduce anxiety in children with ASD in stress contests as the dental clinic is a real challenge and only a few studies have attempted to explore the full potential of ICT to teach useful skills for improving ASD people’s dental care. With our web application MyDentist and the results of our on field experimentation with more then hundred children we propose an effective reproducible developing process that can be exploited to approach several stress contexts of the children with ASD lives.

Chapter 5 describes the development of an IoT system that employs minia-turized sensors and data fusion algorithms based on machine learning to identify automatically the movements applied to the toys by the children. The work, as described in the papers [152, 40] highlights the steps of developing process based on a strength collaboration with all the stakeholders of the project, involved in each phases of the development process, in order to satisfy the clinical requirements on the basis of the monitoring function of the system.

In the Chapter 6 we present the results obtained in reducing anxiety in stress contexts for ASD children and recognizing movements for the support of an ASD early diagnosis in real usage settings. Moreover, we describe the most important result of this three years of research: the refinement of a general development process for systems ICT designed to approach ASD needs. This risult, obtained on the bases of our two case studies and the model of prototyping based development process based on adaptive prototyping, we believe can be used as a reference for the development of other technological solutions supporting different aspects in the life of ASD people.

(26)

devel-1.3. OUTLINE 25 opments.

(27)

(28)

Chapter 2

Background

In this chapter, we introduce the main features of the autistic syndrome (Sec-tion 2.1) and we present an overview of the rela(Sec-tionship between ASD and ICT.(Sec(Sec-tion 2.2

2.1 Autism Spectrum Disorder

Directly derived from the ancient greek word “autus”, meaning oneself, the origin of the term autism gives the best idea about the most popular aspect of this syn-drome. Private and public interests are increasingly addressing the characteristic isolation produced autism, although not in specific medical contexts [153].

Nowadays, the autism universe comes up, in the most unexpected situations, as a matter of debate or media mainstreams discussion objects, often catching un-prepared those who are unfamiliar with the real conditions implied by the spec-trum. Indeed, if on one hand researchers have made great strides in defining and investigating the autism’s etiology [248], the efficacy of the different therapy ap-proaches [267] and diagnostic criteria [29], on the other hand the public opinion is often based on neither scientific nor objective knowledge.

First of all, to approach the ASD we have to start from the concept of autism as a cluster of symptoms that characterize a degenerative disorder of the bio-psycho-social resources [81]; we cannot consider and define ASD as a disease but as a spectrum of permanent and profound disorders. Being a spectrum, the manifesta-tions of the syndrome differ for each person and the exact degree to which such difficulties affect different individuals can vary significantly. Studies have bun-dled three main areas of difficulties proper to the autistic people, which are used to describe the spectrum and to diagnose it. As defined by the Diagnostic and Statistical Manual of Mental Disorders, the three areas, known as the “triad of impairment” [12], include:

1. Communication. Problems with both verbal and non-verbal language (e.g. literal understanding of language, problems with turn-taking in conversa-tions, difficulty in interpreting facial expressions as well as gestures, and echolalia).

(29)

2. Social interaction. Problems with recognizing and understanding other peo-ple’s emotions as well as expressing their own emotions (e.g., difficulty in understanding and following unwritten social rules, a tendency to seem in-sensitive, preference to spend time alone, difficulty in seeking comfort from other people).

3. Patterns of restricted or repetitive behaviors: problems adapting to novel environments (e.g., presence of unusually strong narrow interests, difficulty in coping with unexpected changes, restricted social imagination, difficulty in engaging in imaginative play and activities).

Beyond the triad of impairments, a series of studies investigated by O’Neil [190] offers proof of the presence of sensory disturbances in the most of the people with ASD, suggesting the importance of the role of the external stimuli facing their daily challenge toward some kind of autonomy [192, 258].

Understanding and describing the autistic perception of reality had been con-sidered very difficult for decades, nowadays, thanks to targeted researches [73, 72] and direct testimonials by people with ASD [110], we can easily grasp an idea about the sensitive issues of these people and consequently improve the projects concerning them:

1. Hypersensitivity and low sensitivity to the environmental stimuli, even with strong fluctuations between the two opposite ends. The subject could have an exaggerated sensory perception of smells, noises or physical contacts that could lead him/her to reject even apparently calm situations, or, on the con-trary, he/she could feel them too faintly, with a risk of intensifying auto-simulation (sometimes arriving at episodes of self-harm, seeking a stronger sensation).

2. Pursuit of specific sensory auto-stimulation. ASD People look for specific sensory stimuli in specific situations, recognized by themselves as benefi-cial for certain moments. For example, for autistic people, human contact could lead to increased anxiety but at the same time, the sensation of being mechanically held by a machine could help them to face anxiety [110]. 3. Perceptive distorsions. In certain cases, ASD people could have a sense

perception not corresponding to reality. In particular, for visual stimuli, it can happen that people with ASD wrongly perceive depth or perceive static objects as moving elements.

4. Perceptual overload. Noisy contexts, with unusual sounds or bright lights could cause aversion towards the world around them and also towards them-selves. The exposure to inappropriate sensory stimuli (e.g. loud sounds) could cause unexpected reactions, dominated by aggression and anger.

(30)

2.1. AUTISM SPECTRUM DISORDER 29 5. Multichannel perception. Autistic subjects may often associate a certain ty-pology of sensory stimulus with a personal sensation concerning another sensitive domain.

6. Stimuli hyperselectivity. Autistic subjects often focus their attention on de-tails or irrelevant aspects of the stimuli offered to them, with the high ten-dency to overlook the general information given by the context.

7. High developed visual-spatial discriminatory skills. The tendency to hyper-select the stimuli around them could permit autistic subjects to develop spe-cific perceptive abilities in terms of space-based conception. For instance, Grandin [110] developed important innovative projects in factory farms by applying these skills to concrete problems.

Assuming the irreversibility of the autism diagnosis, the current strategy to improve ASD people’s quality of life are based on early diagnosis and intensive treatment. Several studies have shown that early and intensive intervention, begin-ning in preschool with a load of at least thirty hours per week, produces significant improvement in cognitive development and social/communicative skills. In some cases, it is possible to achieve cognitive development corresponding to the normo-type [255, 167, 195, 42].

The possible interventions are based on different therapeutical approaches. Among the most widely used is Applied Behavioral Analysis (ABA), TEACCH, Denver model (DIR) [257]. In the following we provide more details about the ABA approach which is characterized by very clear rules and extensive data col-lection which drives the personalisation of the treatment. For this reason it is used by many application as a reference approach (see Section 3).

The main features of this approach could be resumed as follows:

• Discrete Trial Training. A one-to-one instructional approach (sequence of elementary trials) used to teach skills in a planned, controlled, and systematic manner in order to allow the child to operate independently.

• Rewards. Rewards are a central point in ABA methodology and are chosen among the items or activities preferred by the child (e.g tickling, a favorite puppet). A reward is used to gratify correct responses of the children during the trial, to motivate them to enforce wanted behaviors.

• Avoiding errors. Therapists avoid errors whenever possible during therapy stopping the child before making them. The goal of this is preventing un-wanted behaviors from settings during therapy sessions.

• Coping with problem behaviors. A large amount of data are recorded ac-cording to ABC (Antecedent Behavior Consequences) model. These data are analyzed to search quantitatively for possible unwanted reinforcements of a problem behavior.

(31)

2.2 Autism and Computer-Based Intervention

The positive effects of computer intervention have long been recognized and inves-tigated [58, 177, 181, 130]. In 1997, Murray [181] presented a preliminary study investigating the tendency of autistic people to fix their attention on small portions of their surrounding area. According to this study, the potential learning power of computers had to be attributed to the prevalence of visual stimulation of the screen, able to break into the autistic world, breaking the individual’s attention tunnel. The positive effects of the increased attention on the learning and therapeutic activity can be favorable for the of self-esteem and self-awareness of the autistic subject involved.

Starting from these considerations, some preliminary experiments on the use of computers with students with autism revealed the following positive effects of the interaction [130]: (a) increase in focused attention; (b) increase in overall at-tention span; (c) increase in in-seat behavior; (d) increase in fine motor skills; (e) increase in generalization skills (from computer to related non-computer activities) (f) decrease in agitation; (g) decrease in self-stimulation.

Other researchers investigated the issue of the concrete contribution of the com-puter based interventions to traditional approaches to the autistic children [58, 177]. Here, the critical question is whether computer-based instruction is more benefi-cial than its low-tech counterpart. Chen et al [58] compare live personal instruction to computer-assisted instruction in a study involving four participants.For 3 out of 4, better motivation and fewer behavior problems were recorded with computer-assisted instruction. However, they found no significant difference in the partici-pants’ learning rates.

Moore et al. [177] compared computerized instruction with a lower-tech be-havioral program for vocabulary instruction for children with autism. In this study children with autism were more attentive and more motivated when presented with computer-based instructions. Additionally, they found that their participants learned more vocabulary on the computer than on the lower-tech behavioral pro-gram.

Based on this evidence, in recent years many researchers, in different fields, have focused their attention on the relationship between autism and computers. The focus of our thesis is on a methodology for the development of applications and digital games, so the literature regarding this specific area will be analyzed in a specific section (Section 3). Areas which have been influenced by the use of technology with ASD people are exemplified in the rest of this section.

In the field of psychology and pedagogy, the contributions in this area are so large as to justify the publication of completely new journals (e.g., the Journal of Special Education Technology, the Journal of Educational Multimedia and Hyper-media, the Journal of Computer Assisted Learning, etc.) remarking an innovative multidisciplinary direction of teaching intervention. In addition, clinical psychol-ogy and medical journals are recognizing the importance of technolpsychol-ogy use for the study of autistic anxiety states [184].

(32)

2.2. AUTISM AND COMPUTER-BASED INTERVENTION 31 Looking at a broader area of innovative software interventions for ASD peo-ple, virtual reality [133] appears to be a promising approach since it allows the construction of environments which can be progressively filled in with details. This allows ASD people to learn step-by-step how to interact with an unknown context, coping with distracting stimuli which can be inserted progressive in the environment [137]. Another notable advantage of virtual reality is the possibility of offering a highly realistic but safe environment in which to teach skills that are associated with some level of danger (e.g., pedestrian safety, stranger safety, etc.) when taught in the natural environment [104].

(33)

(34)

Chapter 3

State of the art

This chapter provides an original taxonomy of the technological solutions adopted to deal with ASD. Differently than other surveys on this topic, we consider a three-dimensional classifications, that crosses the axes of the impairments with the axes of the research purposes and of system activity, returning a mapping of the state of the art even useful to collocate the contributions of our work in this wide research scenario.

3.1 Introduction

In most of the existent literature reviews about autism and ICT, the authors focused their analysis to a very specific domain of intervention, but a general point of view on the discussion is still missing. For example, some reviews focus their mapping only to a specific ICT system domain (as the robotic field [83, 203] or the Virtual Reality field [26, 196] or even more specific field as the multi touch application field [59]).

In other, more frequent, cases, the state of the art is selected and described in a goal-centered perspective, usually with direct on indirect reference to the defini-tion of the three clinical categories of ASD impairments sectors [12] to which the described ICT systems use to be referred to. This could be applies in a special way for the ICT intervention on the communication [260, 218] and the social interaction fields [219, 122].

In our state of the art analysis, we aim to return a view on the complex and fragmented scenario that is ICT applied to ASD, by showing the multidisciplinary complexity of the studies involved in this research field, meant in its broadest def-inition, to force an explicit contamination between the most “clinical” and a most “technical” analysis starting point, involving all the different elements investigated in the lasts twenty years.

Note that our aim is not to provide a comprehensive collection of all the ex-istent knowledge about ICT and autism, but, rather, to give a wide and organized overview of the most illustrative systems and ICT approaches developed in this

(35)

field, and to highlight the most promising and the less investigated intervention ar-eas, by selecting in our presentation papers that are particularly representative of each of the different experiences considered.

Choosing the criteria to identify the gaps and the highlights of the current state of the art, it meant for us to define a more specific combination of aspects (ther-apy or diagnosis, specific system features, impairment area addressed) that clearly brought out the inequality of the researcher attention and interests to one of the combination obtained rather then another, even aware that we where not treating a systematic review selection.

This led us to the definition of a new taxonomy generated from the combination of all the clinical aspects of the ASD condition extrapolated from the state of the art, with an innovative perspective on technology focused on the conjugation of the specific design and validation studies of the systems and the contribution to the needs of the whole life of ASD people (from the early diagnosis to the adult treatment).

We believe that this work could become a useful compass to orient the re-searchers in the exploration of the multidisciplinary and multifaceted world of ICT applied to ASD in the future and, maybe, to be exploited to address new research interests.

Specifically, we classified the literature by crossing the classical axes of the impairments (the triade defined in the DSM-V [12]) with the axes of the research purposes and of system activity.

Then, in the body of the chapter, we articulate the description of selected and significant works for each of the combination of the individuated lines and then proceed with the analysis of the properties of the main ICT systems in use to deal with the clinical purposes.

We will discuss then the different aspects of application testing, accomplished using different techniques, with a specific focus on the tests effectiveness, that could be defined as how well the described application teaches the abilities it is supposed to.

This is a significant open problem which again requires a participatory process to be addressed. In clinical settings, many tests and scale are used to assess the abil-ities of an autistic person, we considered fundamental to highlight the opportunabil-ities and the critical issues of the different approach in use. Still in this perspective, we will discuss the efficacy and the results of various ICT system interventions in their experimental contexts described in the selected papers.

3.2 A new taxonomy fo ICT in ASD

As anticipated in the introduction, the classical taxonomies used to classify works in ASD maintain the largely exploited purpose-centered perspective of the catego-rization based on the three impairments related to communication, social interac-tion and repetitive behaviours.

(36)

3.2. A NEW TAXONOMY FO ICT IN ASD 35 In our taxonomy we would to make a further step, by crossing the axis of im-pairment with the two axes of research purposes (that concern the interdisciplinary double perspective of the diagnostic or therapeutic intentions characterizing every research) and of system activity (monitoring /intervention). The reason for consid-ering the research purposes and system activity is that they both have an impact on the purpose of the ICT systems involved and thus on their design. In particular, treatment often involves interfaces towards the end users and/or some form of ac-tuators, while monitoring involves sensors and may even not have any interface for the user or actuation involved.

This three-dimensional classification is represented as a matrix in fig. 3.1, where each entry corresponds to one of the three above mentioned axes. The matrix represents graphically the intersectional nature of our research domain, and satis-fies the starting purpose of spotting gap area in the wide world of ICT for autism, highlighting with their empty or full cells the weakness and the strengths of the domain.

In a second phase, this survey reconsiders some of the works presented under the perspective of the ICT systems developed for the analyzed purposes.

The result (graphically shown in Fig. 3.2) is a classification of the analyzed researches in two main categories, based on the domain of interaction of their tech-nological support. The first category (cyber-physic) concerns all the robotic and augmented reality-based systems. The second category (only-cyber) includes all the tools based on the virtual reality or the digital reality approach. Note that, with the term ”digital reality”, we mean the category of computer applications (includ-ing mobile app, web app, etc.) that do not interact directly with the physical world (and are thus not cyber-physical) and that do not aim at simulating the reality (as in virtual reality).

(37)

Figure 3.2: ICT sytems in the analyzed literature

The multi-axis perspective and the multidisciplinary approach applied in a system-centered perspective led us to a classification that faces at the same time the clinical and the technological innovation needs, which we believe is one of the most innovative part of our work. In fact, even if in the literature we can find some previous interesting studies about the state of the art of ICT and ASD, even with comprehensive ambitions (see for example [105]), the resulting overviews tends to be partial, especially from the technological perspective.

In the following sections we will present the results obtained from the mapping of the state of the art regarding ICT and autism (presented in its extended form in the table of Appendix A) by exploiting the axes of our taxonomy for a description that aim to decompose and recompose the body of the analyzed texts, trying to highlight through it the interconnections between the aspects of the research we have just identified as salient.

3.3 A purpose-centered overview

This section aims to apply our classification, starting from three main aspects ana-lyzed for each work:

1. if the objective of the study goes in the ASD diagnosis or treatment direction; 2. if the addressed ICT intervention is to be attributed to the communication, to

the social interaction or to the behavioral disorder domain;

3. if the addressed ICT system is implemented to execute monitoring or re-sponse activities.

(38)

3.3. A PURPOSE-CENTERED OVERVIEW 37 As highlighted by the table A.1, the current state of the art, fragmented and uneven, could result more accessible and stimulating if investigated under the lens of a purpose-centered analysis.

3.3.1 Diagnosis or Therapy Perspective?

Autism spectrum disorders (ASDs) are manifested behaviorally. For now, there is no blood test, imaging method, or genetic screening for diagnosis; instead, trained clinicians evaluate individuals by observing their development history and social skills. Because ASD diagnoses are subjective, disagreement can occur among mul-tiple clinicians evaluating an individual or even during a single clinician/patient pairing over time. There is a need for quantitative, objective measurements of so-cial functioning for diagnosis, for evaluating intervention methods, and for tracking the progress of individuals. The difficulty in making reliable judgements about the presence or absence of symptoms and their severity is underscored by the substan-tial amount of training required to become proficient in diagnosing ASD [87]

In this perspective, as theorized in a very recent research by Dawson [79], the digital behavioral measurement may offer better precision and resolution and computers may provide quantitative measurements of behavior, allowing clinicians to describe behavior as a dynamic process that varies on a scale of milliseconds.

Suggestions in this direction come even from the robotic world: in another theoretical study, Scassellati et al. suggest that social robots can provide consistent, reliable actions, and clinicians can ensure that identical stimuli are presented at each diagnostic session [145]. In this sense, for Scassellati et al., social robots may aid in diagnosis by providing consistent behavioral evaluations and standardized stimuli in diagnostic settings.

At the same time, in the analyzed literature we can still count very limited pioneering experimental ICT-driven researches addressed to ASD diagnosis, even in those fields in which, as we will see, the digital monitoring activities are very commonly exploited when addressed to already diagnosed subjects with ASD, as in the case of the robotic field.

The studies to date have mainly focused on the ability to elicit, measure, and/or possibly classify behaviors for future diagnostic ambitions or monitoring, as in the case of Stribling [249] who used interactions between a robot and a child with ASD to elicit and analyze perseverative speech in one individual with ASD who was labeled as “high-functioning.”

In this diagnosis perspective, Orlandi et al. [191] developed an automatic sys-tem to record newborn cry and movements, during the first six months of life, with a specific and innovative and non-invasive recording protocol for the ASD early diagnosis. In their work they present the data analysis mechanism and the first re-sults of acoustic cry analysis in newborns that are classified as high-risk subjects (who are siblings of children already diagnosed as autistics).

Wider and less strictly bounded, the class of ”ASD treatment” includes all the research works aiming at improving the growth, the therapy and the support of

(39)

fundamental autonomies of the ASD children lives, after the diagnosis.

3.3.2 A triad of impairments

An individual with autism may show only some of the typical symptoms, and with different severity levels. However, there are some characteristics and observed indications officially defined by the DSM V[12] that let us to be able to better recognized and categorize the different traits by which the ASD manifests itself.

People with ASD have difficulties with interpreting both verbal and non-verbal languages like gestures or the tone of the voice. Many have a very literal under-standing of language, and think people always mean exactly what they say.

Some may not speak, or have fairly limited speech. They will often understand more than what others tell them rather than what they can express, yet may strug-gle with vagueness or abstract concepts, and they often have difficulty ’reading’ other people - recognizing or understanding others’ feelings and intentions - and expressing their own emotions.

They may also find it hard to form friendships. Some may want to interact with other people and make friends, but may be unsure how to go about it. Fi-nally, people with ASD may not be comfortable with the idea of change, may also experience over-or under-sensitivity to sounds, touch, tastes, smells, light, colors, temperatures or pain.

Communication issues, social interaction impairments and restricted or repeti-tive behaviors (included restricted attention and routine need) are the three clinical recognized areas to which to refer ASD symptoms as those above-mentioned. Communication issues

In the last decade, experiments involving ICT systems and communication learning paths have been numerous and intended for a wide range of severity levels of ASD communication issues. The issue is faced even by not recent studies, with particular interests in non verbal subjects with ASD [66, 65, 128, 115]

The innovations focused on the improvement of ASD communication abilities follow almost three research threads:

1. Augmentative Alternative Communication (AAC) digital support; 2. linguistic exercises (vocabulary, grammar..) softwares;

3. complex systems to monitor and improve specific linguistic abilities, for ex-ample social robots or virtual avatars for spoken interaction.

The ICT tools in this threads range from daily used tools supporting the tradi-tional AAC (whose positive effects are already thoroughly investigated [150]) to the most prototypical virtual tutor’s vocabulary lessons.

(40)

3.3. A PURPOSE-CENTERED OVERVIEW 39 There is a general agreement in the state of the art that supporting language improvement in children with ASD requires concrete tools of immediate use to facilitate the expressions of their needs.

Several studies test and promote the introduction of digital platforms to make easier, more immediate and portable traditional Alternative Augmentative Com-munication tools. These studies, even collected in very recent and specific review (see [100] ), approach the problem with ad hoc user centered design studies [164] and the development and testing of prototypical images-based mobile applications, exploiting the PECS approach [231, 60, 159]. We can also identify the emergence of a (still fragmented) discussion (more on the clinical than on the ICT side) about the speech generating, hand-held devices (or even directly mobile app for tablet or smartphone as those developed by [271, 132]) that play pre-recorded words or phrases when the user flips a switch or presses buttons or keys.

Not only to convey a non-verbal image-based communication, but also to sup-port a more long-term learning process, some researches introduce more complex AAC-based ICT systems that exploit the digitalization of the AAC intervention for a clinical approach to the therapies for the language improvement. This is the case of works as [236] describing a clinical approach that makes significant use of AAC by the Visual Immersion Program (VIP), a software application designed for the improvement of the communication skills by means of portable devices, with the main goal of a transition from scene cues to the elements of language starting from the momentous achievement of Communicating via dynamic and static scene cues. Directly focused on the vocabulary [177, 42], orthography [117] and sentences construction [236], improvement and analysis, over the years several researches have developed prototypal software programs proposing language learning paths based on diversified digital exercises.

Regarding the use of more complex ICT systems on this field, we highlight the exploitation of virtual agent interaction in experimentation aiming to increase the ASD users vocabulary acquisition. An example is Baldi, a computer-animated tutor running on a standard PC created to helps autistic children learn English grammar and vocabulary (Bosseler et al. [42]). This software is based on a set of language lessons based on the association of images and spoken words, starting from the identification of a picture to the production of words and coached by a computer-animated tutor.

At the same time, as we will see for the other impairment areas, in the last years, some researches have gone beyond an exclusively software-based treatment approach, adopting more innovative technologies as the recent Virtual and the Aug-mented reality-based researches of [112, 186] and the Humanoid robots for devel-oping language skills [234], or solutions based on innovative sensors, as in the case of [33] in which the an environmental sensors system collects data of children’s ac-tivities to feed a software that, using Natural Language Generation, helping ASD children in creating narratives based on these data.

Finally, it is worth mentioning studies focused on the narrative and compre-hension abilities of children with ASD. In particular studies as that of Davis et