Cubotto: smart cubes for children's tangible play

(1)

"Cubotto": Smart cubes for children’s

tangible play

Edgar Manuel Amezquita Orozco

Student Id: 873129

Supervisor: Prof. Franca Garzotto

Advisor: Fabiano Riccardi

Dipartimento di Elettronica, Informazione e Bioingegneria

Politecnico di Milano

This dissertation is submitted for the degree of

Master of Science in Computer Science and Engineering

(2)

(3)

(4)

(5)

Acknowledgements

I would like to thank,

My dear family that has been supporting me throughout all my education and life, to my friends that have been there cheering me and sharing some awesome moments, especially my cousin Nicolas has been a great support during these two years.

Fabiano Riccardi for the continuous support, helpful pieces of advice and his knowledge about electronics, that he taught me to finish this work.

Professor Franca Garzotto for giving me this opportunity to work on something that can help people and improve their lives.

(6)

(7)

Abstract

The neurodevelopmental disorder(NDD) is a term used to describe a group of disorders related to a faulty development of the central nervous system, that can manifest in several forms, as attention deficit hyperactivity disorder(ADHD), impaired motor function, learning, language or non-verbal communication. Treating these disorders includes therapy sessions and therapist need tools that facilitate the work.

This work consists of the improvement of a smart toy, called Cubotto, which addresses mainly people with NDD. It is an interactive puzzle consisting of nine cubes which are needed to be placed in a platform accordingly to the game activity configured via a web UI used by the therapist. The subject needs to manipulate the objects using their hands, thus developing the motor skills. Moreover, the flexibility and variety of configurations the activities help to address a wide range of impairments and types of NDD and to enhance the learning through games.

The codesing have been done interviewing therapist in order to understand better the needs of the target users, giving as result a new version with more activities that work in different skills of the person and provides new configurations.

(8)

(9)

Abstract

Il disordine del neurosviluppo (NDD) è un termine usato per descrivere un gruppo di disturbi legati ad un particolare sviluppo del sistema nervoso centrale, che può manifestarsi in diverse forme, come disturbo da deficit di attenzione e iperattività (ADHD), alterazioni nelle funzioni motorie, apprendimento, linguaggio o comunicazione non verbale. Il trattamento di questi disturbi comprende sessioni di terapia in cui lo psicologo può beneficiare di vari strumenti sia di natura digitale che tradizionali..

Questa tesi consiste nel miglioramento di uno smart toy, chiamato Cubotto, che si rivolge principalmente alle persone con NDD. Si tratta di un puzzle interattivo composto da nove cubi che devono essere collocati in una piattaforma di conseguenza all’attività di gioco configurata tramite un’interfaccia utente Web utilizzata dal terapeuta. Il soggetto ha bisogno di manipolare gli oggetti usando le loro mani, sviluppando così le capacità motorie. Inoltre, la flessibilità e la varietà delle configurazioni consentono alle attività di affrontare un’ampia gamma di menomazioni e tipi di NDD e di migliorare l’apprendimento attraverso i giochi.

I miglioramenti sono stati fatti seguendo i suggerimenti del terapeuta per capire meglio le esigenze degli utenti target, dando come risultato una nuova versione con più attività che agiscono sulle diverse abilità della persona.

(10)

(11)

List of Figures

2.1 Montessori Toys: 1-rainbow tower,9-balancing tower . . . 9

2.2 Montessori Wooden puzzle . . . 9

2.3 Skoolzy Learning Colors . . . 9

2.4 Froebel Gifts: Gift 1 . . . 10

2.5 Cube puzzle . . . 11

2.6 Solar blocks . . . 11

2.7 Dolphin Sam: Smart pet for children with ID . . . 13

2.8 Teo: a mobile robot for children with NDD . . . 13

2.9 Puffy: mobile inflatable interactive companion for children with NDD . . . 13

2.10 Code a Pillar . . . 14

3.1 Components of Cubotto . . . 16

3.2 Configuration of a Color game . . . 18

3.3 Pattern shown on the screen . . . 18

3.4 Pattern shown on the platform . . . 19

3.5 Countdown to show the pattern again . . . 19

3.6 Pattern of the letter V . . . 20

3.7 Pattern of the number four . . . 20

3.8 Configuration of a Sound game . . . 21

3.9 Configuration of a Treasure hunt game . . . 22

3.10 Fraternità e Amicizia . . . 24

3.11 I3Lab . . . 24

4.1 Starting and ending game messages . . . 25

4.2 Landing page . . . 26

4.3 Activity flow of the web page . . . 28

4.4 Veroboard version . . . 30

(14)

xiv List of Figures 4.6 PCB Connected . . . 31 4.7 File structure for Color mode . . . 35

(15)

List of Tables

(16)

Nomenclature

Roman Symbols

ADHD Attention deficit hyperactivity disorder DCD Developmental coordination disorder DM Digital Manipulatives

I3Lab Innovative Interactive Interfaces ID Intellectual Disability

IoT Internet of Things

(17)

Chapter 1 Introduction

In our daily life, we have to interact with different people, make decisions, learn and the difficulty is taken for granted, given the fact that the person has all the capabilities. Although, there’s a population that has difficulties with this daily task because they have a condition called Neurodevelopmental disorder (NDD).

The neurodevelopmental disorder is a term used to describe a group of disorders related to a faulty development of the central nervous system, that can manifest in several forms, as attention deficit hyperactivity disorder(ADHD), impaired motor function, learning, language or non-verbal communication. The treating of these disorders includes therapy sessions, pharmaceuticals, external programs when the person is not in therapy.

This project shows the improvements done to a smart toy called Cubotto, which the main aim is to help people with these disorders to improve memory, attention, motor skills and so-cialization. Moreover, it can be used to enhance the learning process by playing the different game’s modes and configurations, given it the possibility to a bigger audience including all kids. It is composed by a platform, nine cubes, and a web UI to configure the games. The first steps were reviewing what has been done and setting the goals for the devel-opment of the improvements through several meetings, during this process we realized that there’s room for a lot of improvements in different aspects, from these were chosen the most relevant ones to developed and the rest will be left for the future work as described in chapter 5.

(18)

2 Introduction The structure is as follows:

• Chapter 2 State of Art,is split into related sections, that shows the concepts Cubotto is based on and related works that have been done in the area.

• Chapter 3 Cubotto, describes the project, the components, activities, and stakeholder for the project.

• Chapter 4 Development, from a technical point of view the improvements are de-scribed..

• Chapter 5 Conclusion and future work, summarizes the improvements done and describes future work.

(19)

Chapter 2 State of the art

2.1 Neurodevelopmental Disorders

2.1.1 Definition

According to the European definition [1], Neurodevelopmental disorders are disabilities that include limitations of the brain’s functioning that affect a person’s behavior, memory or ability to learn e.g. intellectual disability, dyslexia, ADHD, learning deficits and autism. In North America, the definition of neurodevelopmental behavioral intellectual disorder has traditionally included physical as well as functional abnormalities. These disorders engage during the developmental period and may manifest in many degrees of limitations in mental, emotional, physical, social and cognitive capabilities. Although some cases are linked to identified exposures, e.g. fetal alcohol, tobacco smoke, low birth weight and obstetric complications, in most cases specific etiology is unknown. Genetics can play an important role in many neurodevelopmental disorders, and some cases of certain conditions such as intellectual disability are associated with specific genes. However, most neurodevelopmental disorders have complex and multiple contributors rather than any one clear cause. [6] Some symptoms may progressively vanish, while some others remain the same.

The treatment can be difficult and it can involve a combination of professional therapies, pharmaceuticals, and home- and school-based programs. Lately, the use of interactive technology has proven its benefits: it can help alleviate the symptoms and also assist the therapist during the therapy sessions.

(20)

4 State of the art

2.1.2 Different forms of NDD

Examples of neurodevelopmental disorders in children include attention-deficit/hyperactivity disorder (ADHD), autism, learning disabilities, intellectual disability (also known as men-tal retardation), conduct disorders, cerebral palsy, and impairments in vision and hearing. Children with neurodevelopmental disorders can experience difficulties with language and speech, motor skills, behavior, memory, learning, or other neurological functions. [6]

In our case, we have been focusing more in 4 types of disorders: • Autism

Autism is characterized by troubles in social interaction, communication (verbal,non-verbal), repetitive behavior, maintaining attention, sharing a focus in different interests. People with autism have difficulty in understanding other’s intentions and emotions. They may not realize if another person’s behaviors are intentional or unintentional. Their answers to somebody’s questions tend to be very straightforward and sometimes inappropriate since they usually misinterpret the current situation. It is difficult for them to understand metaphors, facial expressions or other non-literal interpretations of communication. As we already know, children have fantasy, imagination which comes handy in role-playing games. That is not the case with children with autism, they are incapable of engaging in these sort of activities due to their lack of imagination.These symptoms apply to both adults and children. According to Autism Speaks, [17],adults with autism may develop tremendous interest in numbers, symbols, dates or science topics.

• Attention-deficit/hyperactivity disorder (ADHD)

ADHD affects children and teens and can continue into adulthood. Children with ADHD may be hyperactive and unable control their impulses. Or they may have trouble paying attention. Adults with ADHD may have trouble managing time, being organized, setting goals, and holding down a job. They may also have problems with relationships, self-esteem, and addiction. [22]

• Intellectual Disability

Intellectual disability (ID), once called mental retardation, is characterized by below-average intelligence or mental ability and a lack of skills necessary for day-to-day living. People with ID have limitations in adaptive behaviors such as skills necessary for the everyday life and in learning processes. Since they have an IQ below 70, their learning process is much slower.

(21)

2.2 Smart Objects and Internet of Things 5 • Developmental coordination disorder (DCD)

DCD also known as developmental dyspraxia, is a chronic neurological disorder beginning in childhood. Developmental coordination disorder is classified (by doctors) in the fifth revision of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) as a motor disorder, in the category of neurodevelopmental disorders. [2]

2.2 Smart Objects and Internet of Things

A smart object is a physical object that interacts with people based on voice, body movements (e.g touching,manipulation). They can also interact with other smart objects, thus creating a network of communication between devices. The data is exchanged through the combination of the Internet and emerging technologies such as near-field communications, real-time localization and embedded sensors. The term Internet of Things has recently become popular to emphasize the vision of a global infrastructure of networked physical objects and is partly inspired by the success of RFID technology, which is now widely used for tracking objects, people, and animals. [13] Given the fact that commercial products are appearing on the market, the technology side is ready and is gaining more success.

2.2.1 Smart object typology

There are several questions to answer before building a smart object such as : What is the level of interaction between the smart object and its sorroundings? How to model and represent the smart objects’ intelligence? What is the role and how can people make use of the gathered chunks of data? What programming models can be used? How can smart objects cooperate to form the "Internet of smart objects"?

Researches on smart objects have been going on for over one decade following many definitions and categorizations.

According to research [13], there are 3 main smart-object types: activity-aware objects,policy-aware objectsand process-aware objects which are characterized by 3 dimensions:

• Awareness: Ability to sense and interpret events happening in the physical world. • Representation: Programming abstractions.

• Interaction: Object’s ability to interact with the user in terms of output, feedback. These types relate to the above-mentioned dimensions:

(22)

6 State of the art • An activity-aware object can analyze the stream of data from the physical world via sensors. Then it uses recognition algorithms to detect specific events and aggregation functions.

• A policy-aware object is an activity-aware object with an embedded policy model. Besides collecting and analyzing data, it also gives timely information to the users. • A process-aware object can "understand" how and when it should be used and the

work activities to be performed next.

This categorization is defined by the requirements of industrial application scenarios such as the petrochemical and road construction industries, where the resulting smart objects can interpret sensor data, make decisions and cooperate with each other.

Another category from the research of Bruce Sterling, [19],is space-time objects that are aware of the sorrounding and memorize real world events. Another research, [21], augmented physical objects with passive RFID so that they could represent their unique information to their users. More recent work has been focusing on the technological aspects. Now smart objects are used in many areas and application scenarios such as: health-care, industrial workplace support, entertainment, etc. Smart Object and Human interaction aspect is still evolving and receiving more attention.

2.2.2 Smart objects and their usage for people with NDD

Advances in IoT, have offered many possibilities in developing smart objects with affordable cost in terms of physical devices and software development. They have the potential to enhance the learning process in the therapy session and save research data for later analysis. Smart objects have the potential of helping persons with NDD to explore and understand cause-effect relationship, to interpret visual contents at different levels of complexity, to learn elementary abstract concepts, to exercise agency, sense of control, and capability to make choices. [9]

There are several design guidelines to take into consideration when it comes to developing a smart object for people with NDD. Each functionality has an impact on the child/adult’s behavior. Table 2.1 concludes the cause-effect relationship of the design and functionality choices, basically how the person with NDD is affected accordingly to the interaction with the smart object. It includes some characteristics retrieved from papers [11],[5] and personal learning while working with Cubotto. Although the research main focus is the design of a robot companion, some role and features may apply also for smart objects.

(23)

2.2 Smart Objects and Internet of Things 7 Detail Effect

Roles

Feedback Reacts to actions per-formed, acts as a rewarding agent

Promote cause-effect rela-tioship and self-esteem Facilitator Suggests the task sequence Understand task execution

sequence Prompt Occupies the person NDD

to take action

Enhance attention and en-gagement

Restrictor Limits spatial& action pos-sibilities

Promote body and space awareness

Emotional agent Unlock emotions Stimulates the capability to manifest the emotions and interepreting other’s emo-tions

Features

Appearance Visual appearance should be familiar, with colors pleasant to the eye.

Multi-modality Manipulation, tactile ex-periences and functional touches help in the develop-ment of sensory-motor ca-pabilities.

Mobility and spatial inter-action trigger engagement and spatial awareness Multi-sensoriality The object should contain

different sensors.

Light,visual digital con-tents (images, videos, ani-mations, etc), sound attract the person with NDD and can be used as instructions , feedbacks, rewards, and to convey emotions.

Configurability Every person with NDD is unique, and this requires the smart object to be con-figurable.

A high degree of configura-bility can address a variety of needs to each individ-ual with specifics levels of NDD.

Autonomy The smart object once con-figured by the caregiver, should perform indepen-dently.

This gives a "magical" feel-ing to the person with NDD.

(24)

8 State of the art

2.3 Related work

In the traditional therapy for children with NDD, there are many methodologies used that involve physical object manipulation. Two people stand out as the great thinkers and designers of the field: Friedrich Froebel, who designed the “Froebel Gifts” for his Kindergarten, and Maria Montessori who created the “Montessori Materials” for her Casa de Bambini. Until now, the physical objects designed by Froebel, Montessori, and others have been collectively called “Manipulatives”. [23] Even though, the toys are not specifically designed for children and adults with NDD, they can be applied in the therapy sessions.

2.3.1 Montessori Method

Montessori is a child-centered educational approach which is applied for over 100 years. It consists in the creation of an environment where the children could learn by exploring and playing independently with toys which stimulate social, physical,emotional and cognitive skills. Montessori toys are made of natural materials and their simple design enables the game levels to be easily managed by the therapist. Montessori-inspired manipulatives are a set of building blocks focused on modeling conceptual, abstract structures. [23]

Montessori Toys

A Montessori toy is one that stimulates learning by encouraging kids to experiment. It is a toy that they can hold and touch, as learning to manipulate objects is key in helping children develop their fine motor skills. The toy should also give them the opportunity to work independently at their own pace and use their creativity. [20]

Some famous examples of Montessori toys are: legos,abacus, rainbow tower, balancing tower, wooden puzzles, Skoolzy Learning Colors, etc

Skoolzy Learning Colors and Shapes Wooden Sorting Box, fig.2.3, keeps the children engaged as they improve their fine motor skills, sequencing and also learn basic color and shape recognition.

(25)

2.3 Related work 9

Figure 2.1 Montessori Toys: 1-rainbow tower,9-balancing tower

(26)

10 State of the art

2.3.2 Fröbel Theory

Friedrich Fröbel, the German educationalist, is best known as the originator of the ’kinder-garten system’. His method inspired and informed the work of Maria Montessori, Rudolf Steiner and others, who adopted his ideas and adapted his materials according to their own work.

Froebel Gifts

Froebel Gifts are considered as the first educational toys. They were so named because they were both given to the child (to be properly respected as gifts) and also function as tools for adults to observe the innate human "gifts" each child posseses from birth.

The set consists of 10 "gifts", where each has a specific goal ad function. It is important to mention Gift 1, which consists of several soft yarn equally-sized balls which differentiate from one another depending on their color. There are 3 primary colors (red/yellow/blue) and 3 secondary colors (purple, orange, green). Froebel taught that red was often the color that most often attracted infants. The game starts with only one ball and then continue to introduce the others with songs.

Later, two balls may be used together. Froebel suggested that balls of complimentary color be used (red/green or orange/purple). If using three balls, start with the three primary colors (red/yellow/blue) and interchange them with the secondary colors. In this way, the child begins to associate the three colors as a group. [8].

Gift 1, fig.2.4, helps a child begin to distinguish form, color and movement.

(27)

2.3 Related work 11 Relatable toys

Some relatable toys to this thesis are the solar blocks and cube puzzle.

Figure 2.5 Cube puzzle _{Figure 2.6 Solar blocks} Although, they are toys used to teach children in kindergarten and pre-school, their concept applied in the creation of non/-smart toys for children and adults with NDD.

2.3.3 Examples of Smart Toys

Digital Manipulatives definition

As mentioned above, physical toys help develop motor skills, autonomy, creativity and when combined with other elements such as colors,lights and sound there are plenty of possibilities in which a game could be played in order to achieve good results.

Digital Manipulatives (DM) are computationally enhanced versions of physical objects, created in an effort to expand the range of concepts that children can explore through direct manipulation. [14].

DM based on the Montessori method are computationally enhanced building blocks systems that support learning of abstract concepts while the DM based on Froebel theory enable children to design real world things, objects, and physical structures.

Examples of smart objects for people with NDD

Nowadays, many smart toys are based on the theories and toys of Montessori and Froebel. Adding digital elements help achieve not only autonomy and creativity, but also an increase in level of configurability.

Some examples of smart objects for people with NDD developed in I3Lab (Innovative Interactive Interfaces) are Dolphin Sam [4], Teo [3], Puffy [10].

(28)

12 State of the art • Dolphin Sam is an interactive stuffed dolphin which aims to help children with ID. It has many "play tasks’ and its behavior changes by emitting light, sound and vibrations according to the child’s manipulations. It is also integrated with lights and multimedia animations or videos in the ambient customized by the therapists. Sam’s goal is to increase cognitive, emotional and social skills by using the different features for various gaming activities such as : touching, caressing , feeding the dolphin while the ambient in the physical environment changes by emitting more light or showing multimedia animations to offer feedbacks or rewards. This helps the children to relax, sustain attention, understand the cause-effect relationship, elementary abstract concepts, make choices and build affective bonds.

• Teo is a mobile huggable companion which reacts accordingly to different types of touch (punch, hugs, or slaps), can move freely on the floor and manifest different emotions through light, sound and movement effects. It can move in all directions and with the same speed as humans, thus giving the ability to NDD subjects to explore the spatial interactions with the robot. The set of stimuli (light, sound, body movements), create fun , build an emotional bond and develop the awareness of cause-effect relationship.

• Puffy is a mobile inflatable interactive companion for children with NDD inspired by the character Baymax in the movie Big Hero 6. It can interpret children’s gestures and movements, facial expressions and emotions,communicated via voice, music, move-ments in space, lights and projections embedded in its body. It has speech recognition features and exploits IBM Watson cognitive computing technology. [10] Puffy engages users in free and structured play and it promotes imagination,communication,memory, space and body awareness. Through physical interaction children with NDD can interpret the various forms of human interaction. Since it provides not only positive feedbacks but also negative in cases where the children acts with inappropriate tactile interaction (e.g punching), which helps them reflect on their actions and build a sense of stability and confidence. Puffy can also express emotions through sound, lights, movements, voice tones. This helps children to interpret these emotions. Also Puffy can move freely, this helps children to learn about spatial awareness.

(29)

2.3 Related work 13

Figure 2.7 Dolphin Sam: Smart pet for children with ID

Figure 2.8 Teo: a mobile robot for children with NDD

(30)

14 State of the art Another example of a smart toy is CodeaPillar, see fig. 2.10 which is a caterpillar that encourages children to experiment and play while developing coding, sequencing and critical thinking skills. By connecting the pieces together in numerous combinations, the caterpillar moves left,right, forward, stops moving for a couple of seconds, all accordingly to the sequence chosen by the child. This encourages children to experiment and play while developing coding, sequencing and critical thinking skills.

(31)

2.3 Related work 15 Multi-sensory enviroment

Another important mention is the Magic Room, a multi-sensory environment which plays an important role in the future development of Cubotto (further details in chapter 5 ).

Multi-sensory environment, is a dedicated space or room where sensory stimulation can be controlled (intensified or reduced), presented in isolation or combination, packaged for active or passive interaction, and matched to fit the perceived motivation, interests, leisure, relaxation, therapeutic and/or educational needs of the user.[7]. These rooms include: lighting effects, sensory activities, tactile experiences, sound effects, activities with physical smart toys, etc.

Magic Room, which is a part of P3S project,is a multi-sensory environment created by I3Lab . The goal of this room is to combine the virtual world presented in a large projected screen, mobile robots and smart objects depending on the activity chosen, light intensity, color, music and animation change.

(32)

Chapter 3 Cubotto

Cubotto is an interactive game for children composed by three components: the platform, the cubes, and the web interface, that aims to help people with Neurodevelopmental disor-ders(NDD) with different activities that works with a system of rewards, in the way that when the user does an action correctly it makes a sound to indicate that it was a good move or another sound to indicate that it was bad move, and after fully completing the activity a song will be played and the platform will light up as the final reward.

Figure 3.1 Components of Cubotto

The arrows show the communication between the components, the black arrow is the main communication that sends configurations and instructions, while the gray one is a feedback communication to perform another action in the receiver further explanation and technical details can be found in chapter 4.

(33)

3.1 Activities 17 The web interface is used to configure the games and send the information to the platform. The platform can light up completely or by parts to be used in the activities or indicate that the set-up is done, it can also play songs or sounds like the reward at the end of the game or to indicate that a cube has been found or a correct/incorrect action. The cubes can also light up in different colors to match the lit parts of the platform or follow a pattern shown on the web interface as in the activity described in subsection 3.1.1, and play a beep or a tone to make sequence game like in subsection 3.1.2 or a hint like in subsection 3.1.3.

3.1 Activities

There are 3 activities/games implemented and totally defined and one that’s still on brain-storming the picture mode, the first one is the color mode that allows you to select an already created color pattern or create one to play, the second one is sound mode that allows you to select sequence on how the cubes will sound and play, third one is treasure hunt that allows you to select the time after the cubes should start making sound to indicate the position and the last one is the picture mode that can be divided in several sub-modes depending on the desired topic.

3.1.1 Color mode

The Color Game consists in reproducing a pattern displayed on the screen or on the platform. Play with visual stimulus and in particular with colors is one of most basic and flexible activities allowed by Cubotto.

The process of configuration for the game can be manual by selecting the color for each cube taking into account that if the selected color is black the cube will not participate in the game, the music that will play when the kid finishes the game, and two timers, were the first one indicates after how long the pattern will show again, the second one indicates for how long is the pattern shown. Once the new configuration is played, this can be saved by the platform for future reuse. For each case, the supervisor can select where is the pattern going to show, by default is on the screen, see fig 3.2.

(34)

18 Cubotto

Figure 3.2 Configuration of a Color game

When the game starts the pattern will show, after that the kid will start putting the cubes on the platform, each time a cube is placed the platform makes a sound indicating whether it is in a correct or an incorrect position, to give feedback to the kid for the next action. In the case, the first timer runs out the pattern will show again for the chosen duration. After all the cubes are placed in the correct positions the game ends, and the platform plays the previously selected song and is ready for the next game.

(35)

3.1 Activities 19

Figure 3.4 Pattern shown on the platform

Figure 3.3 and figure 3.4, illustrates how is the pattern presented in the different forms and also that in the web interface there’s always the remaining time for the showing pattern as in fig. and when it reaches zero, it changes to the countdown that indicates when the pattern is going to show again as in fig. 3.5.

(36)

20 Cubotto This game works on the visual memory and attention as the kid has to observe the pattern and remember where to put the cubes when is gone, also the motor skills as he has to place the cubes in the platform.

The flexibility of the game allows it to teach the colors o even some letters or numbers according to the configuration given, in a way that by selecting the correct colors and po-sitions it’s possible to represent the mixture of colors or some numbers and letters too. An example is that green and blue makes cyan, or organize them in a column to represent the number 1 or the letter I, this depends on the creativity of the supervisor and the capacity of abstraction, as you can make a V by using only three cubes as shown in fig. 3.6 or make the number four as in fig. 3.7.

(37)

3.1 Activities 21

3.1.2 Sound mode

The sound mode is a game were the cubes will sound by groups in the order selected by the supervisor in the web interface, these have to be position in the platform after they make a sound. This groups can be of different size in a way that the sum of the cubes is at most the number of cubes in the platform in this case 9.

The process of configuration for the game is always manual as it is a simple one, the supervisor has to choose the sound/music the cubes will play, the music that will play when the kid finishes the game, a timer that indicates how long is the interval to reproduce again a group if there was no interaction with the platform and between the prearranged options for the number of cubes per group, see fig. 3.8.

Figure 3.8 Configuration of a Sound game

When the game starts the first group of cubes will sound one by one, the kid has to ar-range the cubes that played on the platform, if he does not place a cube for the chosen duration the group will start again, when the kid arranges all the cubes for that group, the next group will sound. As in the color mode, each time the kid places a cube in the platform makes a sound indicating if it’s correct or incorrect, for this case is determined by whether the cube belongs to the group that’s currently playing. After all the cubes are placed the game ends, and the platform plays the previously selected song and is ready for the next game. This game works the sequential memory and attention as the kid has to pay attention which are the cubes that sound for each group so he can place them in the platform and also the motor skills as he has to move the cubes and arrange them on the platform.

(38)

22 Cubotto

3.1.3 Treasure hunt

The treasure hunt is a game were the cubes will be hidden by the supervisor, and the kids have to look for them around the place to put it back on the platform, after a previously set timer the cubes will start making a sound, so it will be easier to find.

The process of configuration for the game is always manual as it is a simple one, after hiding the cubes the supervisor has to choose how many cubes did he hide, the music that will play when the kid puts all the hidden cubes in the platform, the sound/music the cubes will play, a timer that indicates after how long the cubes are going to sound and the number of cubes that are going to produce a sound, see fig. 3.9.

Figure 3.9 Configuration of a Treasure hunt game

When the game starts the kid will begin searching for the hidden cubes, after some time if no cube has been placed on the platform a random cube will start playing, for this game there’s no wrong position because the idea is to retrieve the cubes, so every time a cube is placed on the platform it will make a sound indicating is right, as in the other two when the game finishes the platform plays the song and is ready for the next game.

This game works the spatial coordination and motor skills as the kid has to walk searching for the cubes and returning them to the platform and also the attention as he has to be alert when the cubes are making a sound, so he gets a hint where they are.

(39)

3.2 Stakeholders 23

3.1.4 Picture mode

The main idea behind the picture mode is to make use of the faces of the cube, so by putting images on the sides, it’s possible to produce different games as follows:

• Puzzle: Take one empty face for each cube to put a piece of an image, to make an image puzzle, and the kids have to arrange it on the platform, to provide a variety of puzzles we can use magnetic tape so it’s easy to change between the images.

• Maths: Take another empty face for each cube and put the numbers on it, with the help of the web interface is possible to show a number on the screen and the kid has to put the necessary cubes to form that number.

• Money: Take another empty face for each cube and put the denominations from 5 cents to 20 euros, this to make a more interactive game where the kid has to buy a product and arrange the necessary cubes on the platform to pay that amount.

• Planets: As the solar system is composed of eight planets plus the sun, we can take another empty face of the two that remain empty, to put them and inform about them and how big they’re organized. To do this we can play a short video of the planet that the kid places on the platform, and for the size, we can put an option to order it ascendant or descendant and so the kids will learn by playing.

3.2 Stakeholders

The stakeholders involved in this project are "Fraternità e Amicizia", I3lab and the future end-users of the product, with this in mind the project has been developed to fulfill their needs and requirements.

"Fraternità e Amicizia" is a non-profit social cooperative that works for the creation of services corresponding to the needs of people with disabilities, and as one of the goals is to help people with NDD we have been in contact with one of their therapists Eleonora Beccaluva to understand the requirements from an interdisciplinary and more experienced point of view.

I3Lab is a research group, led by Prof. Franca Garzotto that focuses on innovative interactive technologies and applications in various domains including learning, health, tourism, retail, and fashion. Here was developed the project under the guidance of Fabiano Riccardi as a

(40)

24 Cubotto

Figure 3.10 Fraternità e Amicizia

master thesis. The Lab has several projects that also target children with special needs that are mentioned in chapter 2.

Figure 3.11 I3Lab

The end-users are Fraternita e amicizia as we collaborate with them to understand need, goals and experimentation, people with NDD along with the therapist that will use in sessions, also kids along their supervisor to have fun and learn.

(41)

Chapter 4 Development

Cubotto is a project that started with advanced user interfaces(AUI) course and continued as Tesina project. In this Chapter, I will explain the improvements obtained during these months. This chapter is divided into 2 main sections in order to explain from a technical point of view, the changes made to the components of Cubotto: Web UI, Platform, and Cubes.

The Web UI was rebuilt on Angular to provide a better experience to the user, to be flexible and extendable for future versions and modular to be able to change a page with another one without problems.

The platform now can communicate with the web interface to get feedback via WebSocket, the web server changed from just accepting request at the beginning to work asynchronously to improve the interaction between the platform and the interface.

(42)

26 Development Color modegot a new feature, choose where to show the pattern in the web or the platform with the integration of more LED’s, the connection between cubes and platform is still HTTP, and it communicates to the interface when the game ends via WebSocket.

Sound mode was redefined and is fully functional, in the game there’s communication from the cubes to the platform that was planned to use WebSocket but for technical reasons it end up using HTTP.

Treasure Huntis fully functional, uses the basis of the sound mode, to create a new form of playing with the cubes and a different interaction with the cubes and other kids.

Fast demo mode is the mode that the platforms starts, to make a quick demo of how the platform works and how the cubes has to be put in the platform to be recognized.

4.1 Web Interface

The web application was made using Angular and material, and build from scratch to make a better experience to the user and the code is modular to make it easier to maintain and growth.

(43)

4.1 Web Interface 27 The modular part comes from the design of the application, for each option except the number of cubes, there’s a constant with the values of the options, so you can add more options if necessary without modifying the code, also the each page is different component and two classes, one to use the platform web API and the other one to manage the WebSocket connection.

For each option that’s store as a constant, exists a TypeScript with the structure of the data and the hardcoded values for that option, this gives the flexibility to change it and retrieve from a database or from the platform these values or to add more values without the need to check other files.

Each page is a component, and each component is a folder that contains two Typescript files wherein one is specified all the actions of the page, and the other one is for unit tests, an HTML file with all the content of the page, a CSS file for personalized styles.

For the communication, the components shouldn’t be managing data as it is the view as in an MVC model, so a class is created to manage the data, one for the HTTP connections and another one for the WebSocket connection, this allows us to be more flexible and have high cohesion that means each class is focused on what it should be doing.

(44)

28 Development The following Figure 4.3 represents briefly the action course of the web UI to play a game.

(45)

4.2 Platform and Cubes 29

4.2 Platform and Cubes

As mentioned in chapter 3, in this section the communication will be explained and also the technical details for the games. The principal communications are handled over HTTP and are mostly POST request as the idea is to send configuration parameters over to realize an action on the other device, the feedback communication from the platform to the web interface is via WebSockets as it cannot function as a server to accept HTTP request and the feedback communication from the cubes to the cubes is also HTTP, as we were having problems with the connection via WebSockets.

In order to receive feedback from the cubes, the platform now works as an asynchronous server all the time and not just during the setup, this also allows to reset the game if something happens or ask the platform to turn on/off a group of LEDs or in the future more complex data to make a report on how the kid played with the platform.

Thinking that the project can grow and evolve, the code was refactored, therefore it al-lows a better understandability and facilitates the use of different platform sizes by keeping the code changes to a minimum, also most of the functionalities depends on the defined constants and the parameters received, which gives it flexibility.

To make it more robust a PCB was sent to print (see Figure 4.5) to replace the initial version (see Figure 4.4) that was done from scratch with a Veroboard and was weaker as it had more connections that can suffer from the movement than in the PCB version that is already integrated.

(46)

30 Development

Figure 4.4 Veroboard version

Figure 4.5 PCB version

4.2.1 Platform and Cube API

For the cubes and the platform is established a REST API to handle the request to perform actions, that are built on top of the HTTP protocol, no authentication needed. The returned HTTP code is always 200, you shouldn’t see any other code than 200 unless Wifi connectivity problems.

(47)

Figure 4.6 PCB Connected Platform

Get Previous Configuration

Retrieve from the platform the previous configuration of the different game modes GET /getPreviousPattern?mode={{mode}}

Parameters:

• mode : mode of the game Color|Sound|Treasure Play

Sends to the platform the game mode and data. The platform prepares for the game and set the data.

POST /play

Parameters(A JSON with):

• sentMode: A string among the following: – Color : For a stored Game

– Sound : For a sound game

– Treasure : For a Treasure hunt game

• data: An Object containing the necesary information for the game mode selected – Color: index on the list of patterns {index: number}

(48)

32 Development – Sound:

∗ groups: An object with the groups of the cubes i.e. {g1:[1,2],g2:[3,4]}} ∗ cubeMusic: The music that the cube will reproduce i.e. "dragonball" ∗ music: The music the platform will reproduce when the user wins i.e.

"/DragonBall.mp3"

∗ time: Suggestion time to make the group play again – Treasure:

∗ repeatCubes: A number indicating the number of cubes for the hint ∗ cubeMusic: The music that the cube will reproduce i.e. "dragonball" ∗ music: The music the platform will reproduce when the user wins i.e.

"/DragonBall.mp3"

∗ time: Suggestion time to make the group play again Save Color Configuration

Send the data to the platform to store it, for future games POST /saveConfig

• timer1 : timer in seconds that indicates how much the platform should wait for the cube to be placed

• timer2 : timer in second that determines for how long will the pattern be shown(Screen or Platform)

• cubes : Array containing the color of the cubes

• melody : path to the song in the platform i.e. "/frozen.mp3" Delete Color Configuration

Delete a previous stored game in the platform POST /deleteConfig

Parameters(A JSON with): • id : Id of the game to delete Insta Color Game

Used for instant color play. Send to the platform the data, which then writes it in the tempColorGame file

POST /tempColorGame Parameters(A JSON with):

(49)

4.2 Platform and Cubes 33 • timer1 : timer in seconds that indicates how much the platform should wait for the

cube to be placed

• timer2 : timer in second that determines for how long will the pattern be shown(Screen or Platform)

• cubes : Array containing the color of the cubes

• melody : path to the song in the platform i.e. "/frozen.mp3" Turn on LED group

Sends the request to the platform to turn on a group of LEDs. POST /turnOnGroups

Parameters(An array of JSON with):

• id : id of the group of LEDs from 0 to the number of cubes, with 0 being the side LEDs of the platform and 1 to n the respective to each cube

• color : color in hexadecimal for the LEDs to turn on Turn off LED group

Sends the request to the platform to turn off a group of LEDs. POST /turnOffGroups

Parameters(An array of JSON with):

• id : id of the group of LEDs from 0 to the number of cubes, with 0 beign the side LEDs of the platform and 1 to n the respective to each cube

Cubes

LED Request

Send the request to the cubes to turn on the LED in the color indicated in JSON. POST /LED

• color: a string that indicates the color in natural language (i.e. "green") ID request

Sends the request to the cube to retrieve the ID POST /ID

TAG request

Sends the request to the cubes to retrieve the tag POST /TAG

(50)

34 Development Play Melody

Send the command to play the specified melody POST /MELODY

• melodyID: a string that indicates the melody to play

• tempo: an integer that indicates the single note duration, in ms, if not supplied will be used the one in the file

Gameover

Sends to the cubes the request turn off the light POST /GAMEOVER

4.2.2 Color Mode

The platform receives a JSON via POST request from the interface when the play button is clicked, the data contained in the JSON depends on the configuration of the game if it’s a stored one or a manual one, in the first case it just holds the position in the array, in the second it has the colors of the cubes, the song that is going to play when the game finishes and the timers.

When the JSON arrives at the platform, is parsed to know the contents and made the correspondent action, that is either read the configuration from the file in memory or config-ure the game with the data just received.

For the pattern to be shown on the platform, we added four LEDs around every RFID reader and two new endpoints were created to manage them from the interface so the plat-form is not conscious of the timers.

The file with the previous configurations for the color mode is a simple text archive with the structure of a JSON array to ease the reading and writing without the necessity of a more complex structure and to keep it uniform as all the configurations are sent in JSON, an example of the file is Figure 4.7.

As the file has the structure of a JSON array when the interface asks for the configurations, the response is just reading the file and interpreting it as a JSON to send it back.

For the storage of a new pattern, the platform receives a JSON with the same informa-tion that is sent for a new game, with this informainforma-tion and a new id that comes from the id of

(51)

Figure 4.7 File structure for Color mode

the last stored configuration incremented by one, we make a JSON and store it at the end of the file.

For the delete of a pattern, the platform receives a JSON with the id of the configura-tion the user wants to remove, next it reads the file in memory search for the configuraconfigura-tion to delete, removes it and writes it back to the file.

4.2.3 Sound Mode

As in the Color mode, the platform receives a JSON via POST request from the interface when the play button is clicked, the data contained in the JSON holds the sound/song that the cubes are going to play, the song that is going to play when the game finishes, the timer and the groups of cubes.

When the JSON arrives at the platform, is parsed to know the contents and configure the platform to play, create the groups of cubes, store the songs, configure the tags that are valid for each reader, set the game variable and play the first group.

(52)

36 Development The creation of the cube groups is completely generic as it goes through the object and for every element it extracts the cubes that are inside and stored them in a matrix where the position [i,j] is the j-th element is a cube that belongs to the i-th group.

Different from the color mode, the platform is now in charge of the timer, as is neces-sary to know when was the last time a cube was read by an RFID reader to start counting and make the group play again.

4.2.4 Treasure hunt

As in the previous modes, the platform receives a JSON via POST request from the interface when the play button is clicked, the data contained in the JSON holds the sound/song that the cubes are going to play, the song that is going to play when the game finishes, the timer and how many cubes are going to sound.

When the JSON arrives at the platform, is parsed to know the contents and configure the platform to play, store the songs, configure the tags that are valid for each reader, set the number of cubes that are going to sound and start the timer.

When the timer finishes, the platform will make n random cubes sound one after the other excluding the cubes that are already in the platform, and will not repeat any cube, this function is based on the randomHat from Arduino.

Similar to the sound mode, the platform is now in charge of the timer, as is necessary to know when was the last time a cube was read by an RFID reader to reset the timer.

(53)

Chapter 5 Conclusion and Future Work

5.1 Conclusion

These months improving Cubotto has led to the second version of it, that has more activities and options. During the process meetings with the advisor and the therapist were conducted, in order to understand the needs, what can be done to give a better experience to the user and settling details for the new activities developed.

Moreover, the project has evolved to facilitate the addition of new activities by making it modular and extensible for future development, also the hardware was modified to give it more robustness and complement the color mode to show the pattern in the platform. This thesis has been an experience of learning and discovering how to improve a toy to help people with neurodevelopmental disorders, from the point of the development was a very interesting work as I had to learn new topics and take into account considerations as the resources were limited.

5.2 Future work

As stated before, cubotto has many potentials for future improvements from doing more activities or offer analytics and data collection from the platform to give more details of the interaction to the therapist, for a better analysis of the overtime development for the kid. One activity can be the picture mode that’s described in chapter 3 or one of the variations ex-plained or more creative activities taking advantage of the components or modification to the

(54)

38 Conclusion and Future Work hardware to add more sensors to capture specific data to produce more complex activities and combinations. Define levels for the previously defined games, so it will be according to the degree of the impairment of the person, so the parents can use it at home without any problem. With the help of the WebSocket connection, data can be sent in real time to be stored in a database or in a file, for a posterior analysis for each kid, so the therapist can have a better understanding of the progress of the kid over time, and make charts to represent the data.

(55)

Bibliography

[1] (2003). Draft baseline report on neurodevelopmental disorders in the framework of the european environment and health strategy.

[2] Arlington (2003). Diagnostic and Statistical Manual of Mental Disorders, volume 5. American Psychiatric Publishing.

[3] Bonarini, A., Clasadonte, F., Garzotto, F., Gelsomini, M., and Romero, M. (2016). Playful interaction with teo, a mobile robot for children with neurodevelopmental disor-ders. In Proceedings of the 7th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion, pages 223–231. ACM.

[4] Colombo, S., Garzotto, F., Gelsomini, M., Melli, M., and Clasadonte, F. (2016). Dol-phin sam: a smart pet for children with intellectual disability. In Proceedings of the International Working Conference on Advanced Visual Interfaces, pages 352–353. ACM. [5] Escobedo, L., Ibarra, C., Hernandez, J., Alvelais, M., and Tentori, M. (2014). Smart

objects to support the discrimination training of children with autism. Personal and ubiquitous computing, 18(6):1485–1497.

[6] Flamez, B., King, J. H., and Francis, J. (2015). Conceptualizing dsm-5 disorders in children and adolescents. Diagnosing and Treating Children and Adolescents: A Guide for Mental Health Professionals, 1.

[7] Foundation, H. A. (2006). What are multi-sensory environments. http://www.cdhaf.org/ what-are-multi-sensory-environments/.

[8] froebelgifts (2003). http://www.froebelgifts.com/gifts.htm. Accessed on 13-08-2018. [9] Garzotto, F. (2017). Smart objects for people with cognitive impairment.

[10] Garzotto, F., Gelsomini, M., and Kinoe, Y. (2017). Puffy: A mobile inflatable interactive companion for children with neurodevelopmental disorder. In IFIP Conference on Human-Computer Interaction, pages 467–492. Springer.

[11] Gelsomini, M., Degiorgi, M., Garzotto, F., Leonardi, G., Penati, S., Ramuzat, N., Silvestri, J., and Clasadonte, F. (2017). Designing a robot companion for children with neuro-developmental disorders. In Proceedings of the 2017 Conference on Interaction Design and Children, pages 451–457. ACM.

(56)

40 Bibliography [12] Howlin, P. (2000). Outcome in adult life for more able individuals with autism or

asperger syndrome. Autism, 4(1):63–83.

[13] Kortuem, G., Kawsar, F., Sundramoorthy, V., and Fitton, D. (2010). Smart objects as building blocks for the internet of things. IEEE Internet Computing, 14(1):44–51. [14] Patten, J., Ishii, H., Hines, J., and Pangaro, G. (2001). Sensetable: a wireless object

tracking platform for tangible user interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 253–260. ACM.

[15] Plowman, L. and Luckin, R. (2004). Interactivity, interfaces, and smart toys. Computer, 37(2):98–100.

[16] Sonuga-Barke, E. J. (2003). The dual pathway model of ad/hd: an elaboration of neuro-developmental characteristics. Neuroscience & Biobehavioral Reviews, 27(7):593–604. [17] speaks, A. (2005). https://www.autismspeaks.org. Accessed on 13-08-2018.

[18] Srivastava, M., Muntz, R., and Potkonjak, M. (2001). Smart kindergarten: sensor-based wireless networks for smart developmental problem-solving environments. In Proceedings of the 7th annual international conference on Mobile computing and networking, pages 132–138. ACM.

[19] Sterling, B. (2005). Shaping things (mediaworks pamphlets).

[20] Toys, M. (2018). https://www.whitbyschool.org/passionforlearning/ 12-montessori-toys-to-help-your-child-learn-through-play. Accessed on 15-08-2018. [21] Want, R., Fishkin, K. P., Gujar, A., and Harrison, B. L. (1999). Bridging physical and

virtual worlds with electronic tags. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pages 370–377. ACM.

[22] webMD (2005). https://www.webmd.com/add-adhd/guide/ attention-deficit-hyperactivity-disorder-adhd#1. Accessed on 13-08-2018.

[23] Zuckerman, O., Arida, S., and Resnick, M. (2005). Extending tangible interfaces for education: Digital montessori-inspired manipulatives. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’05, pages 859–868, New York, NY, USA. ACM.

Cubotto: smart cubes for children's tangible play