A cognitive architecture based on an amygdala thalamo cortical model for developing new goals and behaviors : application in humanoid robotics

(1)

POLITECNICO DI MILANO

Corso di Laurea Specialistica in Ingegneria Informatica Dipartimento di Elettronica e Informazione

A COGNITIVE ARCHITECTURE BASED ON

AN AMYGDALA-THALAMO-CORTICAL MODEL

FOR DEVELOPING NEW GOALS AND BEHAVIORS:

APPLICATION IN HUMANOID ROBOTICS

AI & R Lab

Laboratorio di Intelligenza Artificiale e Robotica del Politecnico di Milano

Relatore: Prof.ssa Giuseppina Gini Correlatore: Prof. Riccardo Manzotti Correlatore: Ing. Flavio Mutti

Tesi di Laurea di: Marco Burrafato, matricola 739401 Luca Florio, matricola 750029

(2)

(3)

(4)

(5)

List of Figures

2.1 The brain areas of interest. . . 8

2.2 Motor primitives combination . . . 10

3.1 From the brain to our architecture . . . 14

3.2 Sensory integration in our architecture . . . 15

3.3 Intentional Module . . . 18

3.4 Categorization Module . . . 20

3.5 Ontogenetic Module . . . 21

3.6 The Somatosensory Cortex and the Motor Cortex . . . 24

3.7 An example of clustering . . . 25

3.8 The State-Action table . . . 26

4.1 UML diagram of the architecture . . . 30

4.1 The IDRA window . . . 37

4.2 The RobotEditor window . . . 38

4.3 The BehaviorsEditor window . . . 38

4.4 The EditSensors window . . . 39

4.5 The EditActuators window . . . 39

4.6 The NetEditor window . . . 40

4.7 Intentional Modules in the NetEditor window . . . 40

4.8 The trainingSetup window . . . 41

4.9 The TrainingViewer window . . . 42

4.10 The TrainingFocus window in training mode. . . 43

4.11 The Viewer window . . . 43

4.12 The formFocus window. . . 44

4.13 Activity Diagram 1 . . . 45

4.14 Activity Diagram 2 . . . 46

5.1 The Aldebaran Robotics NAO robot. . . 49

5.2 The two boards used in test one. . . 49

5.3 The architecture used in test one. . . 50

(8)

5.4 From Gaussian values to head rotation. . . 51

5.5 The first test, part one. . . 53

5.6 The first test, part two. . . 53

5.7 The first test, part three. . . 54

5.8 The NAO robot and its beloved heart. . . 56

5.9 The architecture used in test two. . . 58

5.10 The hand positions in test two. . . 60

5.11 The most relevant hand positions in test two. . . 60

6.1 An ideal complex network of intentional modules . . . 66

B.1 ICA signals: (a) original signals; (b) mixed signals; (c) inde-pendent components; (d) reconstructed signals . . . 90

B.2 An example of Gaussian distribution . . . 93

C.1 KmeansLib simulator . . . 101

C.2 K-Means clustering . . . 102

D.1 Graphical interface . . . 104

(9)

Abstract

Cognitive development concerns the evolution of human mental capabilities, through experience earned during life; however, the way human beings de-velop new goals and behaviors during their lifetime is not completely under-stood. Many researches have been done in order to realize agents that could develop autonomously through experience, interacting with the environment and adapting to it. Different approaches in robotics aim to realize this kind of agents, from behavior-based robotics to developmental robotics. An im-portant feature needed to accomplish this objective is the self-generation of motivations and goals, as well as the development of complex behaviors consistent with them.

The objective of this thesis is to realize a bio-inspired cognitive architecture, based on an amygdala-thalamo-cortical model, capable of autonomously de-velop new goals and behaviors. This cognitive architecture has been im-plemented and tested using a humanoid robot. Experimental results show the main features of the architecture: the development of new goals starting from hard-coded ones, and the generation of complex movements consistent with those goals. Despite the architecture is still under development, good experimental results demonstrate that it is a good starting point in order to develop a more complex and effective system.

(10)

(11)

Estratto in lingua italiana

Nel corso della sua vita, dall’infanzia fino all’età adulta, l’essere umano sviluppa le proprie capacità mentali attraverso un processo detto sviluppo cognitivo, che riguarda il modo in cui una persona percepisce il mondo at-torno a se, come pensa, e come comprende il mondo attraverso l’interazione di fattori genetici ed acquisiti. Un aspetto fondamentale nello sviluppo co-gnitivo di una persona è la generazione autonoma di nuovi comportamenti ed obiettivi, il che permette all’individuo di adattarsi alle diverse situazioni che deve affrontare ogni giorno. Come gli esseri umani siano in grado di svi-luppare autonomamente questi nuovi obiettivi nel corso della loro esistenza non è completamente noto. La robotica, per realizzare agenti in grado di interagire in modo efficace con gli esseri umani e integrarli nella loro vita, si deve interessare dei processi interni al cervello umano che permettono lo sviluppo cognitivo dell’individuo, così come delle modalità alla base della generazione di nuovi obiettivi e comportamenti.

Scopo della nostra tesi è creare un modello robotico bio-ispirato basato sui processi interni al cervello umano, che permetta all’agente di sviluppare au-tonomamente nuovi obiettivi, nonché nuovi comportamenti che siano coe-renti con questi obiettivi.

Esistono già diversi approcci nel campo della robotica per risolvere il problema dell’adattabilità e sviluppo autonomo delle motivazioni in un ro-bot. La robotica “behavior-based” permette all’agente di adattare il proprio comportamento ai cambiamenti nell’ambiente, al fine di raggiungere i propri scopi. In questo approccio, gli obiettivi sono preimpostati nel robot, e non è possibile svilupparne di nuovi. La robotica epigenetica studia lo sviluppo cognitivo nei sistemi naturali e artificiali allo scopo di adattarsi all’ambiente e sviluppare autonomamente nuove motivazioni ed obiettivi che non erano previsti in fase di progettazione. A differenza di questi approcci, il sistema qui presentato si propone di investigare quel livello cognitivo intermedio che permette agli esseri umani di essere consapevoli dell’ambiente circostante, e

(12)

quindi di interagire con esso per la maggior parte delle operazioni di base dell’organismo.

Questa capacità è una condizione essenziale per consentire al robot di adat-tarsi nella vita quotidiana dell’essere umano: un robot deve essere in grado non solo di agire in modo coerente ai cambiamenti nell’ambiente circostante, ma anche di sviluppare obiettivi che possono emergere dalla situazione in cui si trova, in modo da interagire efficacemente con le persone con cui essa verrebbe in contatto. Tale robot arriverà anche a sviluppare una personali-tà unica, a seconda delle esperienze che hanno contribuito alla creazione dei suoi nuovi obiettivi e comportamenti. Queste caratteristiche renderebbo il robot ideale per applicazioni complesse come quelle dell’intrattenimento, sia nel privato che nel pubblico, permettendo alla robotica di entrare in campi di applicazione di attività comuni.

Il sistema che presentiamo consente ad un agente di sviluppare nuovi obiettivi, oltre a quelli già presenti, e di adeguare il proprio comportamen-to a questi obiettivi. Quescomportamen-to sistema si ispira alla meccanica del cervello umano, in particolare alla comunicazione di tre diverse aree del cervello: la corteccia cerebrale, il talamo e l’amigdala. L’interazione di queste tre aree è un elemento chiave per lo sviluppo cognitivo umano (ovvero lo sviluppo delle capacità intellettuali), ed è quindi considerato come un buon punto di riferimento per l’attuazione del nostro sistema. Su questa base è stata creata l’Architettura Intenzionale: si tratta di una rete di unità elementari, chiamate Moduli Intenzionali, che consente lo sviluppo di nuovi obiettivi. In aggiunta ai Moduli Intenzionali nella rete è presente anche un Modulo Filogenetico, il quale contiene gli obiettivi preimpostati, cioè gli “istinti in-nati” dell’agente. Questo modulo esegue le stesse funzioni dell’amigdala nei sistemi biologici. Attraverso l’azione del Modulo Filogenetico, più lo stato corrente del robot incontra gli obiettivi già fissati, maggiore è l’intensità del segnale in uscita dal modulo.

Ogni Modulo Intenzionale è composto da due moduli: il Modulo di Ca-tegorizzazione e il Modulo Ontogenetico. Il Modulo di Categorizzazione rappresenta la corteccia cerebrale del nostro sistema, e il suo ruolo è quello di restituire un vettore che rappresenti l’attivazione neurale della corteccia in risposta allo stato d’input.

Il Modulo Ontogenetico è la base dello sviluppo di nuovi obiettivi; riceve il vettore di attivazioni neurali dal Modulo di Categorizzazione, e impiega una funzione di apprendimento Hebbiano per sviluppare nuovi obiettivi. Una volta che la sua elaborazione è conclusa, restituisce un segnale che indica se lo stato corrente è conforme ai nuovi obiettivi.

(13)

Il Modulo Intenzionale analizza i segnali provenienti dai moduli Filogenetico e Ontogenetico, restituendo il più rilevante dei due, e restituendo anche il vettore di attivazione neurale calcolato dal Modulo di Categorizzazione. Pertanto, il flusso di esecuzione dell’Architettura Intenzionale inizia quando l’input sensoriale viene filtrato e quindi inviato alla rete di Moduli Inten-zionali, la quale elabora le informazioni ricevute; ciascun modulo restituisce un vettore contenente informazioni sullo stato dell’ambiente, e un segnale che indica quanto lo stato effettivo soddisfi gli obiettivi attuali. La rete può essere composta da più strati, ognuno contenente diversi Moduli Intenzio-nali, che possono essere collegati tra loro in modo diretto o retroazionato. L’uso di diversi livelli di Moduli Intenzionali permette all’utente di astrarre lo stato del sistema dall’ingresso sensoriale originale.

Il vettore di attivazioni neurali e il segnale calcolato dall’Architectura Inten-zionale vengono poi utilizzati dal Sistema Motorio per generare movimenti che possano essere coerenti con gli obiettivi dell’agente. Ogni movimento è formato da una serie di componenti elementari, chiamati primitive motorie, che sono alla base del movimento sia umano sia animale: esse rappresentano le attivazioni muscolari nel tempo, e la loro composizione (cioè la sinergia muscolare) porta alla realizzazione di movimenti complessi. Per ottenere questo risultato anche nel sistema qui proposto, le informazioni provenienti dall’Architectura Intenzionale sono utilizzate per adattare il comportamento dell’agente ai nuovi obiettivi e alle nuove situazioni: ogni Comportamento Dinamico analizza i dati di ingresso per suggerire all’agente il miglior movi-mento da applicare a quella specifica situazione, ovvero, quale azione porterà alla realizzazione dei suoi obiettivi, siano essi innati o acquisiti. Il Compor-tamento Dinamico restituisce un insieme di attivazioni muscolari (cioè una composizione di primitive motorie), ognuna riferita a ciascun giunto dell’a-gente collegato al comportamento specifico. Queste attivazioni muscolari vengono poi utilizzate dall’agente per eseguire il movimento corretto.

L’Architettura Intenzionale si occupa quindi dello sviluppo cognitivo del robot, con l’analisi dell’input e la generazione di nuovi obiettivi, mentre il Sistema Motorio consente al robot di muoversi e intraprendere azioni al fine di raggiungere tali obiettivi.

Il sistema presentato è stato testato con due esperimenti, che ci per-mettono di verificare l’abilità dell’architettura di sviluppare nuovi goal così come la capacità di adattare il proprio comportamento ai suoi obiettivi. In un primo esperimento l’Architettura Intenzionale è composta da una sem-plice rete con un singolo Modulo Intenzionale; l’agente (NAO, un robot

(14)

umanoide prodotto dalla Aldebaran Robotics) impara un nuovo obiettivo a partire da uno preimpostato: a partire da un istinto innato relativo a figure con colori saturi, l’agente è in grado di sviluppare autonomamente un inte-resse per la forma delle figure.

In un secondo esperimento, invece, vengono testate le capacità motorie. Que-sta volta la rete ha due livelli e tre Moduli Intenzionali, e il robot NAO si muove per massimizzare il valore del segnale rilevante proveniente dall’Archi-tettura Intenzionale. Inoltre i movimenti sono generati dalla combinazione lineare delle primitive motorie.

Questi risultati sperimentali dimostrano l’efficacia del sistema, e hanno for-nito alcuni suggerimenti su possibili miglioramenti e sviluppi futuri del pro-getto. L’architettura ha dimostrato di imparare nuovi obiettivi, nonché di agire in modo intelligente per soddisfare questi obiettivi. Tuttavia, il sistema è ancora in fase di sviluppo e molti miglioramenti potrebbero essere appor-tati, dalla capacità di sviluppare obiettivi e comportamenti più complessi, al miglioramento dell’implementazione per ottenere prestazioni più eleva-te. Oltre ai miglioramenti del sistema, sarà interessante l’implementazione dell’architettura in diversi agenti, per studiare l’interazione e il possibile svi-luppo di obiettivi di gruppo.

I contributi principali di questo lavoro sono:

• la progettazione e realizzazione di un’architettura cognitiva basata su un modello amigdala-talamo-corticale;

• il test dell’architettura stessa, sperimentando la funzionalità di genera-zione di obiettivi dell’architettura replicando un esperimento eseguito con una precedente implementazione di un singolo Modulo Intenzio-nale;

• l’estensione dell’architettura cognitiva con l’implementazione del Si-stema Motorio, che consente la creazione di movimenti tramite la composizione lineare delle primitive motorie.

Questa tesi è strutturata come segue.

Nel capitolo 2 presentiamo lo stato dell’arte, con vari riferimenti riguardanti l’aspetto biologico relativo al nostro sistema, e quindi diverse applicazioni tecniche in cui questi concetti sono stati utilizzati.

Nel capitolo 3 si discute l’Architettura Cognitiva, composta dall’Architettu-ra Intenzionale e dal Sistema Motorio. In primo luogo l’Architettudall’Architettu-ra Inten-zionale è descritta con tutte le sue componenti: il Modulo IntenInten-zionale e il Modulo Filogenetico. Per ognuna di queste parti forniamo una descrizione

(15)

dettagliata delle sue funzionalità. Dopodichè mostriamo il funzionamen-to del Sistema Mofunzionamen-torio con i Comportamenti Dinamici, e mostriamo come l’output dell’Architettura Intenzionale sia utilizzato per la generazione di un movimento che possa essere coerente con gli obiettivi dell’agente, mediante composizione di primitive motorie.

Nel capitolo 4 viene descritto il progetto IDRA - Intentional Distributed Robotic Architecture, ovvero l’implementazione dell’architettura cognitiva descritta nel capitolo 3. Viene fornita una descrizione tecnica del program-ma, così come le linee guida per l’utilizzo del software.

Nel capitolo 5 sono descritti i test sperimentali e i risultati ottenuti dal si-stema presentato. Vengono presentati due esperimenti, ciascuno a prova delle caratteristiche principali dell’architettura: sviluppo di nuovi obiettivi e generazione di movimenti.

Il capitolo 6 contiene le conclusioni e gli sviluppi futuri: riassumiamo i no-stri obiettivi, le nostre valutazioni dei risultati ottenuti e i possibili sviluppi futuri del progetto.

In appendice A vi presentiamo alcuni estratti del codice di IDRA, con una breve descrizione.

In appendice B mostriamo la libreria che abbiamo implementato per l’ese-cuzione dell’algoritmo Independent Component Analysis (ICA), nonché una breve descrizione teorica dell’algoritmo.

In appendice C mostriamo la libreria che abbiamo implementato per l’ese-cuzione dell’algoritmo K-Means per la clusterizzazione, nonché una breve descrizione teorica dell’algoritmo.

In appendice D presentiamo brevemente il progetto Behavior Simulation, un software che abbiamo sviluppato per testare le funzionalità dei comporta-menti dinamici prima della loro implementazione nel Sistema Motorio.

(16)

(17)

Acknowledgements

Luca

Il ringraziamento più grande va sicuramente ai miei familiari, che mi hanno sostenuto (e mantenuto) in questi duri anni di studio. Mi spiace di essere stato un po’ intrattabile in alcune occasioni, ma si sa che solo con le persone alle quali vogliamo più bene riusciamo ad essere completamente noi stessi, nel bene e nel male. Grazie mamma e papà, per avermi sempre lasciato libero di seguire la mia strada ed avermi sempre sostenuto nelle mie scelte. Grazie con tutto il cuore.

Grazie a Giorgia, la mia splendida ragazza, che da poco più di un anno a questa parte è entrata nella mia vita, standomi vicino e riuscendo nella non facile impresa di farmi sentire amato e capito ogni giorno che abbiamo pas-sato insieme. Grazie, per avermi fatto capire la differenza tra essere sereno ed essere felice.

Grazie agli amici della mia compagnia, che non sto a nominare tutti perché (per mia fortuna) sono veramente tanti: mi avete regalato quei momenti di allegria e spensieratezza senza i quali sarebbe stata sicuramente più dura raggiungere questo obiettivo. Grazie anche agli amici che ho conosciuto al Politecnico e che hanno condiviso con me gioie e dolori di questa vita univer-sitaria. In particolare grazie al Coach e Teo, due meravigliose persone che mi hanno accompagnato in questa avventura ogni giorno dal primo all’ultimo anno, e a Frank, senza il quale questa tesi avrebbe richiesto sicuramente più tempo per essere realizzata. Grazie a tutti voi amici miei, per me siete più importanti di quello che pensate.

Grazie alla professoressa Gini per averci fatto da relatrice. Grazie a Flavio e Riccardo, che non solo ci hanno aiutato e supervisionato durante la rea-lizzazione di questo progetto, ma ci hanno anche insegnato tanto.

Grazie a Marco, perché è stato un ottimo compagno di tesi (che grazie a lui ha delle immagini veramente belle!)

Grazie al Politecnico, perché dopo tutto mi ha reso una persona migliore.

(18)

Marco

Alla nostra relatrice, la prof.ssa Giuseppina Gini, senza la quale non po-tremmo scrivere questa tesi. A Flavio, che ci ha seguiti lungo tutta la preparazione della tesi. A Riccardo, che ci ha fornito le basi concettuali, e soprattutto i mezzi tecnici, sui quali lavorare. Ai miei compagni di tesi, Lu-ca e Alessio, senza il vostro aiuto non avrei mai potuto finire questo lavoro. A Ilhwan, che ci ha portato il Nao da un altro continente.

A mia madre e a mio padre, che mi hanno supportato, e sopportato, in que-sti (venti)sei anni. A mia sorella Valeria, che si litiga sempre ma alla fine ci si vuole un gran bene. Ai miei zii, cugini, nonni, parenti tutti, che mi sono stati vicini in questi anni.

A tutti i miei colleghi ed amici del Politecnico che mi hanno accompagnato in questi anni. Al Matteo, “IlCoach”, il mio clone. A Roberta, la mia BFF. A Christian, e alla nostra amicizia ventennale. A Davide, “dadokkio”, grazie al quale ho una tavoletta grafica, e grazie alla quale ho realizzato tutte le immagini di questa tesi. A Silvia, che ci si aiuta nei momenti difficili. A Smaranda, perchè è un’architetta ma è nerd lo stesso. A Fabio, e a tutti i caffè insieme. Al Dolph, e a tutti i suoi soprannomi. A Francesco, che mi ha accompagnato fino a Parigi. Al mio videgame developing team: Riccardo, Matteo, Christian. A Matteo, Erica, Gillo, Elisabetta, Giovanni, Ylenia, Francesco, Daniele, e tutti gli amici che mi hanno tenuto compagnia in se-greteria e in Airlab.

A tutta la mia compagnia di Melegnano: Andrea, Elena, Simo, Pietro, Cri-stina, Fo, Michela, Vera, Daniele, Anna, Teo, David, Valentina... che nono-stante tutto mi invitano ancora a uscire con loro.

A tutti i miei amici e compagni del karate: Fulvia, Liliana, Alfredo, Desi, Mauro... e in particolare a Massimo, che finalmente la smetterà di chiedermi quando finisco gli studi e vado a lavorare.

A tutti gli altri amici che mi hanno accompagnato al di fuori di queste quattro mura. A Maria “Eco” Elisa, e ai nostri drammi sentimentali. A Valentina, e ai nostri viaggi a Wacken. A Diego e Alice, grazie ai quali sono ancora più nerd. A Mara e Saverio, e alle nostre uscite. Alla mia sceneggia-trice Paola, che mi coinvolge in progetti mastodontici.

Ad Antonella, che ha fatto avanti e indietro innumerevoli volte per aprirci la porta del laboratorio. Al JJ e alla Latteria, per la loro fornitura quotidiana di piadine e focaccine. Alle ferrovie italiane, che non mancano mai di movi-mentarti la mattinata. A chi stamperà questa tesi, se mi farà uno sconto. Al Politecnico, che mi ha tenuto qui e mi ha istruito per tutti questi anni.

(19)

(20)

(21)

Chapter 1

Introduction

“A good question is, of course, the key by which infinite answers can be educed.”

Isaac Asimov, Foundation’s Edge

During their life, from childhood through adolescence to adulthood, hu-mans develop their mental capabilities: this process is called cognitive de-velopment, and concerns how a person perceives, thinks, and gains under-standing of his or her world through the interaction of genetic and learned factors [1]. A fundamental aspect in the cognitive development of a per-son is the autonomous generation of new goals and behaviors, which allows the individual to adapt to the various situations he faces every day. How humans can develop autonomously new goals during their existence is not completely understood. In order to realize agents capable to interact in an effective way with humans and integrate in their life, robotics should study the processes of human brain which allow the cognitive development of the individual, as well as the modalities underlying the generation of new goals and behaviors.

Our thesis gives a contribution to the achievement of this objective: its purpose is to create a bio-inspired robotic model based on human brain pro-cesses, that must be able to allow the agent to autonomously develop new goals as well as new behaviors that could be consistent with these goals.

There are different approaches in robotics to the problem of adapta-tion and self-development of motivaadapta-tions. Behavior-based robotics allow the agent to adapt its behavior to changes in the environment, in order to accomplish its goals [2]. In this approach goals are hard-coded into the robot, which cannot develop new ones. Developmental robotics (also called

(22)

2 Chapter 1. Introduction

“epigenetic robotics”) aims at modeling the development of cognition in nat-ural and artificial systems [3]. Developmental robotics leads to the cognitive development of the agent, making it able to adapt to the environment and autonomously develop new motivations and goals, that were not present at design time. Unlike these approaches, the system we propose addresses an intermediate level of cognition that allows mammals and humans to be aware of the surrounding environment and then interact with it for most basic tasks.

This capability is an essential precondition to enable the robot to fit into the human’s everyday life. A robot must be able not just to act in a consistent manner to the changes in the surrounding environment, but also to develop goals that can emerge from the situation in which it is located, in order to interact effectively with people with whom it would come into contact. Such robot would also develop a unique personality, depending on the experiences that contributed to the creation of its new goals and behaviors. These fea-tures would make it the perfect robot for advanced applications in the world of entertainment, either at home and not, pushing robotics beyond common application fields.

We present a system that allows the robot to develop new objectives, in addition to the hard-coded ones, and to adapt its own behavior to these objectives. This system is inspired by the mechanics of the human brain, in particular the communication of three different areas of the brain: the cere-bral cortex, thalamus and amygdala. The interaction of these three areas is a key element in human cognitive development (i.e. the development of intellectual abilities), and it is therefore regarded as a good reference point for the implementation of our system [4, 5]. On this basis the Intentional Architecture has been created: it is a network of elementary units, called Intentional Modules, that enables the development of new goals. In addi-tion to Intenaddi-tional Modules in the network is also present a Phylogenetic Module, containing the hard-coded objectives, i.e. the “innate instincts” of the agent; this module performs the functions of the amygdala in biologi-cal systems. Through the action of the Phylogenetic Module, the more the current state of the robot meets the objectives already set, the higher is the signal coming out from the module.

Each Intentional Module consists of two internal modules: the Categoriza-tion Module, and the Ontogenetic Module. The CategorizaCategoriza-tion Module is the cerebral cortex of our system, and its role is to return a vector that represents the neural activation of the cortex in response to the input state provided.

(23)

3

The Ontogenetic Module is the basis of the development of new objectives; it receives the vector of neural activations from the Categorization Module, and employs a Hebbian learning function to develop new goals. After its processing is done, it returns a signal indicating whether the current state meets the new goals defined.

The Intentional Module analyzes signals from the Phylogenetic and Onto-genetic Modules, returning the most relevant of the two, and returning also the vector of neural activation computed by the Categorization Module. Therefore, the Intentional Architecture execution flow starts when the sen-sory input is filtered and then sent to the Intentional Modules network, which processes the received information; each module returns an vector containing information about the state of the environment, and a signal indicating how much the actual state is satisfying the actual goals. The network can be composed by several layers, each one containing several In-tentional Modules, that can be connected to each other in straightforward or feedback mode. The use of different levels of Intentional Modules allows the user to abstract from the raw sensory input.

The vector of neural activations and the signal computed by the Intentional Architecture are then used by a Motor System to generate movements that can be consistent with the goals of the agent. Each movement is meant to be formed by a series of elementary components called motor primitives, which are on the basis of animal and human motion alike: they represent the muscles activations over time, and their composition (i.e. muscular synergy) leads to the execution of complex movements [6, 7, 8, 9, 10]. In order to ob-tain this result also in the system we propose, the informations coming from the Intentional Architecture are used to adapt the behavior of the agent to the new goals and situations: each Dynamic Behavior analyzes the input data to suggest to the agent the best movement to apply to that specific situation, that is, which action will lead to the fulfillment of its goals, either learned or innate. The Dynamic Behavior returns a set of muscle activations (namely a composition of motor primitives), each one referring to each joint of the agent connected to that specific Behavior. These muscle activations are then used by the agent to perform the correct movement.

The Intentional Architecture therefore is concerning with the cognitive de-velopment of the robot, by input analysis and the generation of new goals, while the Motor System allow the robot to move and take actions in order to accomplish these goals.

The presented system has been tested with two main experiments, which let us verify the goals development skills as well as the ability to adapt its

(24)

4 Chapter 1. Introduction

behavior to its goals.

In a first experiment the Intentional Architecture is put under test. With a simple network of a single Intentional Module, the agent (a NAO robot, a humanoid robot produced by Aldebaran Robotics) learns a new goal from a hard-coded one: starting from an innate instinct related to figures with high-saturated colors, the agent autonomously develops an interest to a par-ticular shape of the figures.

In a second experiment instead, the motor capabilities are tested. This time the network has two layers and three Intentional Modules, and NAO robot moves to maximize the relevant signal coming form the Intentional Architec-ture; furthermore, movements are generated by linear combination of motor primitives.

These experimental results show the efficacy of the system, and they pro-vided several clues about possible improvements and future developments of the project. The architecture has proven to learn new goals, as well to act in a smart way to satisfy these goals. However the system is still un-der development and a lot of improvements could be made, starting from the capability to develop more complex goals and behaviors, to technical improvements in order to obtain better performances. In addition to im-provements of the system, it will be of interest the implementation of the architecture on several different agents, in order to study their interaction and the possible development of team goals.

The main contributions of this work are:

• the design and implementation of a Cognitive Architecture based on an amygdala-thalamo-cortical model;

• the validation of the architecture itself, by testing the goals genera-tion feature of the architecture with the replicagenera-tion of an experiment performed with an early implementation of the Intentional Module structure;

• the extension of the Cognitive Architecture with the implementation of a Motor System, which allow the creation of movements by linear composition of motor primitives.

This thesis is structured as follows.

In chapter 2 we present the state of the art, with various references concern-ing the biological aspect related to our system, and then different technical applications where these concepts have been used.

(25)

5

In chapter 3 we discuss the Cognitive Architecture, composed by the Inten-tional Architecture and the Motor System. First the IntenInten-tional Architec-ture is described with all of its components: the Intentional Module and the Phylogenetic Module. For each one of these parts we provide a detailed description of its functionalities. Then we show the functioning of the Motor System with dynamic behaviors, illustrating how the output of the Inten-tional Architecture is converted for the generation of a movement that could be consistent with the goals of the agent, by means of composition of motor primitives.

In chapter 4 we describe the Intentional Distributed Robotic Architecture (IDRA) project, the implementation of the Cognitive Architecture described in chapter 3. A technical description of the program is provided, as well as the guidelines to use the software.

In chapter 5 we describe the experimental activities and results obtained by the presented system. Two experiments are described, each testing the main features of the architecture: goals development and complex move-ments generation.

Chapter 6 contains the conclusions and future development: we summarize our goals, our evaluations of these results and possible future developments of the project.

In appendix A we present some code excerpts of IDRA, with a brief descrip-tion.

In appendix B we show the library that we implemented for the execution of the Independent Component Analysis (ICA) algorithm, as well as a brief theoretical description of the algorithm.

In appendix C we show the library that we implemented for the execution of the K-Means clustering algorithm, as well as a brief theoretical description of the algorithm.

In appendix D we briefly show the Behavior Simulation project, a software we developed to test the functionalities of Dynamic Behaviors before their implementation in the Motor System.

(26)

(27)

Chapter 2

State of the art

“Artificial Intelligence is whatever hasn’t been done yet.”

Larry Tesler

2.1 The Amygdala-Thalamo-Cortical model

Several studies have shown the importance of the amygdala-talamo-cortical model in the cognitive development [11, 12].

The cortex is the external part of the brain; it is divided in several sec-tors (primary visual cortex, posterior parietal cortex, primary motor cortex, etc.), and it receives signals from the sensory organs. Although most of the sectors of the cortex receive input from a specific source, different stud-ies have proven that different areas can properly react to different stimuli sources [5]. Therefore the whole cortex is composed from the same kind of cells, and it is able to respond, store and adapt to different kind of stimuli. Moreover, the generic mechanism used by the cortex is what allows the hu-man beings to adapt to problems that did not exist in nature, and to learn how to deal with them.

The main difference between an usual computer system and the brain, and the reason why the latter is more efficient at solving some kind of problems - e.g. ball catching, face recognition, etc. - lies in the fact that the brain does not compute everytime for this kind of problems. Instead, the cortex has been proven to act as a memory bank, and the brain use to compare the incoming data with memories from past situation, in order to find analogies, patterns, and invariant representations, and act accordingly [4]. Moreover,

(28)

8 Chapter 2. State of the art

it has been proven how the cortex stores these patterns in sequences, and how sequences can be recalled in an auto-associative fashion, thus creating the so-called stream of thoughts.

Figure 2.1: The brain areas of interest.

The thalamus is the largest component of the diencephalon, one of the most internal regions of the brain. It is the primary site of relay for all of the sensory pathways, except olfaction, on their way to the cortex. The thalamus plays a central role for the mammals in the development of new motivations and goals, as well in the choice of what goal to pursue. The thalamus is “a central, convergent, compact miniature map of the cortex” [13]. The thalamus is partitioned into about fifty segments, which do not communicate directly with each other. Instead, each one is in synchronized projection to a specific segment of the cortex, and it is receiving a projection from the same segment. Therefore, while the cortex is concerned with data processing, storing and distribution, the thalamus determines which goals have to be pursued [14]. Furthermore, each pair of cortex and thalamus sections seem to be intensively communicating. The cortex is believed to be

(29)

2.2. The movement generation via motor primitives 9

the main memory storage allowing consciousness and cognition, while the thalamus would determine how incoming stimuli are relevant with respect to actual goals and to the developing of new goals. Thanks to the thalamo-cortical model the brain is able both to learn how to achieve goals and which goals have to be pursued.

The amygdala is an almond-shaped group of nuclei located deep within the medial temporal lobes of the brain of complex vertebrates. It seems to be heavily connected to the cortex area; it is involved in the generation of somatosensory response on the basis both of innate and previously developed goals, and of sensory informations [15]. In particular, the amygdala seems to be an essential part in social and environment cognition in order to guide social behaviors.

The amygdala has a key role in the recognition of the emotions and in the generation of an adequate response. Evidence shows that this response is independent from the test subject [16]. Thus one of the principal tasks of the amygdala is to generate new goals taking advantage of hardwired criteria.

2.2 The movement generation via motor

primi-tives

The cerebellum is supposed to have functionalities and structures similar to the classical perceptron pattern of a classification device. The cerebellum has an extended network of various types of neurons, giving different abili-ties including motor learning and motor coordination, thanks in particular to the Purkinje response cells, which can learn fast and that are able to easily distinguish patterns [17].

Other studies have shown the importance of the basal ganglia in the mo-tor generation of the movement [18]. Basal ganglia and cerebellum seem to create two different sets of loop circuits with the cortex, both dedicated to different features of motor learning; both are independent and in differ-ent coordinates. Furthermore, there has been recdiffer-ently proposed a functional dissociation between the basal ganglia and the cerebellum: the former is im-plicated in optimal control of movements, that is optimization of costs and rewards obtained by the execution of a specific movement, and the latter seems to be able to predict the sensory consequences of a specific movement through the use of internal models [19].

The spinal cord is the lower caudal part of the nervous system; it receives and processes sensory information from the various parts of the body, and

(30)

controls the movement of the muscles and the joints of the main parts of the body [11]. It acts as a bridge between the body and the mind: it receives electrical signals input from the peripheral nervous system, and sends them to the central nervous system. In the same way, receives signals from the neurons in the cerebellum and the cortex and processes them before sending instructions to the specific muscles.

Additional studies show how the cortex is involved in motor generation, in particular the cortico-striatal circuit [20].

Several studies have shown how complex movements are generated from the combination of a limited number of waveform modules, which are inde-pendent from the considered muscle, its speed, and its gravitational load; these slighting varying modules control the active muscles in order to pro-duce kinematic changes [21]. These studies have found how just five com-ponents can account for 90% of the muscles activations.

It was suggested that the nervous system do not need to generate all the mus-cles activity patterns, but to generate only a few basic patterns and combine them accordingly, to generate a specific muscle activation, through modula-tion of phasic component and magnitude of these primitives [22]. Different muscles performing different tasks showed different changes in patterns ac-tivities. This model can be represented by an oscillator that produces the output frequency basing on the input signal, a set of nonlinear functions that shape the oscillator output into the set of patterns, and a weighting functions for the generation of the muscle activity pattern [23].

Figure 2.2: Motor primitives combination

A motor primitive is a specific neuronal network, found in the spinal cord, that generates a specific motor output by sending a definite signal to muscles for activations [24]; each primitive has a basic activation pattern (in function of time), with different weights and distributions for each muscle;

(31)

2.3. An intermediate level of cognition 11

thus, the muscular activity during movement is determined by few basic patterns, independent from locomotion mode, direction or speed [25].

The Dynamic Movement Primitives - that is, a formulation of movement primitives via autonomous nonlinear differential equations - have been suc-cessfully used in a series of robotic applications, and they can be employed with supervised learning and reinforcement learning techniques in order to optimize trajectory and energy criteria [26]. Various application involved skill learning from supervised learning on a robot, or biped walking from imitation of an human, or the generation of rhythmic movement patterns using a nonlinear oscillator control [27].

However, we separate ourselves from this kind of experiments, which use learning methods commonly applicable to task-based robots; we will in fact use motor primitives in motivation-based robots, as we will see in the next chapters.

2.3 An intermediate level of cognition

The amygdala-thalamo-cortical model we addressed so far is the basis of the biological inspiration of the first part of our architecture. In this part we do not focus on high-level motor skills, nor on high level of reasoning and planning. We will instead focus on an intermediate level of cognition that allows mammals and humans to be aware of the surrounding environment and then interact with it for most basic behaviors.

Consciousness has been already suggested to be a product of an intermedi-ate level of cognition [28]. The awareness of the surrounding environment is supposed to be not a direct product of sensations and other sense-data, the “phenomenological mind”, nor a product of a high level conceptual thoughts, the “computational mind”, but to be a product from several intermediate levels of representation [29].

This middle level has some interesting features related to consciousness: it underlines how we can interpret the surrounding environment and react to this awareness without the need for high-level conceptualizations nor com-plex motor controls, therefore solving the grounding problem of a semantic interpretation of a formal symbol system that is intrinsic to the system itself [30]. In particular, as we will see in the next chapters, we will deal primar-ily with the categorical representation, that are described as “learned and innate feature detectors that pick out the invariant features of object and event categories from their sensory projections”.

(32)

Most robots - and their related intelligence - are designed with a specific task or a set of tasks. Some robots are equipped with some sort of skills that allow them to interact with the environment and in particular with humans. Most of these robots have a selection of preset behaviors that allow them to tune their actions according to past experience; they are reactive to their surroundings in order to reach their goals [2].

Some of these robots are not just able to tune their behaviors to the situa-tion: they are also able to act on their own. While the behavior-based robots explore the environment in order to optimize their actions for reaching a pre-defined goal, these kind of robots must be able to explore the environment in order to find new goals on their own. They must be curious to explore their environment, and once explored, they must be able to do something new according to their abilities as well as to their experience. We call this kind motivational-based robotics [31], also known as epigenetic robotics, or developmental robotics [3].

Behaviors-based robots make use of motivations as well, but these are fixed, hardwired at design time; on the contrary, developmental robots must be able to perform actions following goals that were not present at design time. To make a comparison with human beings, the need for food is a hardwired criteria, and even a child will try to obtain it in the ways his known behav-iors allow him to. At first he will scream until his mother will feed him; eventually he will learn to reach cutlery with his arm in order to bring food to his mouth.

(33)

Chapter 3

Cognitive Architecture

“They learn to speak, write, and do arithmetic. They have a phenomenal memory. If one read them the Encyclopedia Britannica they could repeat everything back in order, but they never think up anything original. They’d make fine university professors.”

Karel Čapek, R.U.R.

3.1 Intentional Architecture:

cognitive

develop-ment

Cognitive development is the construction of thought processes, including re-membering, problem solving, and decision making, from childhood through adolescence to adulthood. It refers to how a person perceives, thinks, and gains understanding of his or her world through the interaction of genetic and learned factors. Among the areas of cognitive development we can find information processing, intelligence, reasoning, language development, and memory [1].

The Intentional Architecture deals with the cognitive development of the agent, analyzing inputs and allowing it to develop new goals. The ar-chitecture is basically a net of linked modules, simulating connections and interactions between the three cerebral areas which allows the cognitive de-velopment in humans: the cerebral cortex, the thalamus and the amygdala. The modules composing the net are the Phylogenetic Module (amygdala) and the Intentional Module, which in turn is composed by the Categoriza-tion Module (cerebral cortex) and the Ontogenetic Module (thalamus). The

(34)

14 Chapter 3. Cognitive Architecture

Intentional Architecture is composed by one Phylogenetic Module and sev-eral layers of Intentional Modules. Intentional Modules are linked in various ways, also with feedback connections, while the Phylogenetic Module broad-casts its signal to all the Intentional Modules, without receiving data back by them. This kind of structure can simulate the interaction of the three areas in human brain: different areas of thalamus can communicate with some respective areas of the cerebral cortex, which collects all the informa-tion coming from the thalamus, but different areas of the thalamus cannot communicate between them. The amygdala sends its information both to the thalamus and the cerebral cortex (Figure 3.1).

The input of the net is composed by various sensor types (video, audio, tactile, etc.), elaborated by some filters and sent to the Intentional Modules of the first layer. The output of the net is composed by a vector, repre-senting the neural activation generated by sensory input, and by a signal, representing how much the actual input satisfies both hard-coded goals and new developed goals.

Thanks to the net of Intentional Modules, the Intentional Architecture can autonomously develop new goals (through the Ontogenetic Module) start-ing from hard-coded ones (provided by the Phylogenetic Module), which represent the “innate instincts” of the agent.

Figure 3.1: From the brain to our architecture

3.1.1 Input processing: sensory integration

Sensory integration is the neurological process that organizes sensation from a person’s own body and the environment, thus making it possible to use the body effectively within the environment. Specifically, it deals with how the brain processes multiple sensory modality inputs into usable functional out-puts. For some time it has been believed that inputs from different sensory

(35)

3.1. Intentional Architecture: cognitive development 15

organs are processed in different areas of the brain [32]. The communi-cation within and among these specialized areas of the brain is known as functional integration [33, 34, 35]. Newer researches had shown that these different regions of the brain may not be solely responsible for just one sen-sory modality, but they could use multiple inputs to perceive what the body senses about its environment. Sensory integration is necessary for almost every activity that we perform since the combination of multiple sensory inputs is essential for us to comprehend our surroundings.

The input of the Intentional Architecture comes from several sensors of the agent. Just like the information that are sent to our brain, the inputs of the agent are processed to give the architecture the same type of data and to perform sensory integration. In this way, the architecture does not depend on the type of input, while every kind of sensory information (video, audio, tactile, etc.) can be processed by the architecture in the same way. Furthermore, the Intentional Architecture can perform sensory integration by sending multiple sensory data to the same Intentional Module.

The data coming from every sensor is converted into arrays of different size and enclosed in a standard structure; every block of information is then sent to the first layer of Intentional Module to start the computation (Figure 3.2).

(36)

3.1.2 Phylogenetic Module: innate instincts

In biology, phylogenetics is the study of evolutionary relation among groups of organisms, which is discovered through molecular sequencing data and morphological data matrices. The result of phylogenetic studies is the evo-lutionary history of taxonomic groups: their phylogeny.

Phylogenetic processes contribute to the adaptation of an organism behav-iors to the environment through the production of instincts [36]. An Instinct is the inherent inclination of a living organism toward a particular behavior, i.e. an impulse or powerful motivation from a subconscious source. The part of the brain that is associated with these types of reactions is the amygdala [37]. Different parts of the brain receive different signals and make them consciously known. The auditory cortex, for example, is responsible for hearing. The amygdala has its own set of “receivers” for sensory intake, and can retrieve information from the environment and take a decision about what to do, before a person could consciously think about it [38].

We implemented instincts in the system as hard-coded goals, in the Phy-logenetic Module. this module contains embedded information and criteria in order to permit the bootstrap of the system. Phylogenetic Module tells the agent what is relevant according to its hard-coded instinctive functions. The input of this module comes from sensors; each sensory information is then processed by instinctive functions; each function processes only a cer-tain input type, according to the instincts associated to that specific type (e.g. input from video sensors is processed only by instinctive functions re-lated to video input). The output of this module is the phylogenetic signal, which tells how much the incoming stimulus is important according to the a priori stored criteria. The more considerable the stimulus, the higher the generated phylogenetic signal (for our purposes, as we will see in the next chapters, this value will always be normalized between zero and one).

3.1.3 Intentional Module: neuroplasticity

Plasticity, or neuroplasticity, is the lifelong ability of the brain to reorganize neural pathways based on new experiences.

Neuroplasticity can occur in different levels, ranging from cellular changes (e.g. during learning) to large-scale changes involved in cortical remap-ping (e.g. as consequence of response to injury). As we learn, we acquire new knowledge and skills through instruction or experience. Scientific re-search has now demonstrated how substantial changes can occur in the low-est neocortical processing areas, and how these changes can profoundly alter

(37)

the pattern of neuronal activation in response to experience [39]. Further-more, experience can actually change both the brain’s physical structure (anatomy) and functional organization (physiology) [40]. Neuroplasticity has replaced the formerly-held position that the brain is a physiologically static organ, and explores how and in which ways the brain changes through-out life into adulthood [39]. In order to learn or memorize a fact or skill, there must be persistent functional changes in the brain, and these changes represent the new knowledge. The ability of the brain to change with learn-ing is what is known as neuroplasticity.

Neuroplasticity is a fundamental feature to give an agent the ability to learn new goals, so we designed the Intentional Architecture as a net of basic modules which can adapt to changes in sensory inputs and connections. The Intentional Module is the basic building block of the Intentional Archi-tecture. This module has to be able to adapt to various sensor modalities, as well as to combine with other instances of the same kind in some simple way. The Intentional Module can easily adapt to changes in sensory input. If we send to an Intentional Module some input from a video sensor, the In-tentional Module will specialize to that type of input; however, if we decide to change some connections and send to this specialized Intentional Module a different type of input, e.g. an audio input, it will lose its specialization in video input and gradually adapt to the new sensory input.

An Intentional Module has other important features: since it is the basic unit of the architecture, it must be able to self-develop new goals and mo-tivations. This feature of the Intentional Module has already been put to test in a very simplified robotic setup, aiming at developing new motivations and controlling the gaze of a camera towards unexpected classes of visual stimuli [31, 41, 42].

According to what we said, we can summarize the main objectives of the module:

• it must be able to adapt to any kind of input;

• it must be able to learn to categorize incoming stimuli;

• it must be able to use acquired categories to develop new criteria to categorize;

• it must be able to interface smoothly with similar modules and give rise to a hierarchical structure.

(38)

by two simple structures: the Categorization Module, which performs cate-gorization, and the Ontogenetic Module, which can develop new goals. The Intentional Module works as follows. At the beginning, both the Onto-genetic Module and the Categorization Module are empty. Incoming data, of any type and size they are, are sent to the Categorization Module, which has the important function of receive raw data and then to develop higher order categories. Once the categories have been created, they are sent to the Ontogenetic Module. This module perform Hebbian learning on incoming data to develop new goals, and returns a signal based on how much this new goals are satisfied. This signal is called ontogenetic signal; a high value of ontogenetic signal corresponds to high satisfaction of the developed goals. The Intentional Module receives as input also the signal from the Phylo-genetic Module, and returns as output the maximum between the signal of the Phylogenetic Module and the Ontogenetic Module. This outgoing signal is called relevant signal; the Intentional Modules also sends as output the categories created by the Categorization Module (Figure 3.3).

Figure 3.3: Intentional Module

Categorization Module: cognitive autonomy

Most of the actual information processing in the brain takes place in the cerebral cortex. The cerebral cortex is divided into lobes, and each of them has a specific function [11]. For example, there are specific areas involved

(39)

in vision, hearing, touch, movement, and smell. Other areas are critical for thinking and reasoning. Although many functions, such as touch, are found in both the right and left cerebral hemispheres, some functions are found only in one of the two cerebral hemisphere. For example, in most people, language abilities are found in the left hemisphere. The sensory areas are the regions that receive and process information from the senses [11]. Parts of the cortex that receive sensory inputs from the thalamus are called pri-mary sensory areas. One of the main feature of the cerebral cortex is the ability to adapt to stimuli, whatever is their nature [5, 4]. Cerebral cortex presents a high degree of cognitive autonomy, so each cortical area is capable to compute any type of incoming data [43, 44, 45].

The Categorization Module represents the cerebral cortex of the tional Architecture. It receives the input from sensors or from other Inten-tional Modules and perform categorization. in order to do this operation, the input is elaborated twice: first with Independent Component Analysis (ICA, see Appendix B), then with a clustering algorithm such as K-Means (see appendix C).

Independent Component Analysis allows the module to generalize the input representation regardless to the type of incoming stimuli. In an early-development stage, independent components are extracted from a series of input through the ICA algorithm. After this training stage, the input is projected in the bases space (i.e. the previously extracted independent com-ponents), in order to reduce the dimension of the data and to get a general representation:

W = IC × I (3.1)

Where W is the resulting vector of weights, IC is the matrix of independent components and I is the input vector.

This results in a vector of weights where clustering is performed, using K-Means algorithm. Clustering is a good and simple way to get the neural code of the information. Neural coding consists in the translation of a stimulus into a neural activation; according to this, a net of neurons can represent every kind of incoming information. Basing on the theory that sensory and other information is represented within the brain by networks of neurons, it is supposed that neurons can encode any type of informations [46]. During clustering, each vector is assigned to an existing cluster, if the distance from existing clusters is below a previously-set threshold, otherwise a new cluster is created, using the newly acquired vector.

(40)

The output of the Categorization Module is a vector containing the activa-tions of clusters, which depend on the distances of the input data from the center of each cluster (namely a category). This vector corresponds to the activation of a neuron centered in each cluster:

yi = ρ(x, Ci) (3.2)

Where y_i is the distance of the actual input from the center of the cluster i, x is the input and Ci is the center of the cluster i. For our purposes, the

values are normalized between zero and one.

A new category is created by the categorization module depending on the value of the relevant signal computed by the Intentional Module. This way, only relevant inputs are categorized, so that the module saves only mean-ingful information (Figure 3.4).

Figure 3.4: Categorization Module

Ontogenetic Module: goals generation

The thalamus is a structure of the brain composed by four parts, or nu-clei, situated between cerebral cortex and midbrain. Functionalities of the thalamus include elaboration and relay of input and motor signals to cere-bral cortex, as well as regulation of consciousness, sleep, and alertness [47]. Furthermore, thalamus is a “miniature map” of the cerebral cortex [13]. All thalamic nuclei, with the exception of the reticular thalamic nucleus, project primarily to the cerebral cortex. Additionally, each portion of the thalamus receives a reciprocal connection from the same portions of the cerebral cortex whereby the cortex can modify thalamic functions. These connections are more data intensive from cerebral cortex to thalamaus, while backward con-nection going from thalamus to cortex are weaker [48]. This close concon-nection between thalamus and cortex and their interplay, as well as the implication

(41)

of thalamus in consciousness, led to the idea that goals generation is indeed spread everywhere in cerebral cortex and it is obtained by the interaction between thalamus and cortex.

The Ontogenetic Module represents the thalamus of our system and it is the core part of the goals development of the Intentional Module. the name ontogenetic derives from ontogeny, namely the origin and the development of an organism; it covers, in essence, the study of an organism lifespan, unlike phylogeny which covers instead the evolution of the entire species of an organism. Just like in the brain thalamus and cerebral cortex present strong connections, Ontogenetic Module is closely connected to the Categorization Module. It uses the categories computed by the Categorization Module and a Hebbian learning function to develop new goals.

The values of neural activations, provided by the Categorization Module, are evaluated using a vector of weights, and the resulting ontogenetic signal is the maximum value between the evaluated neural activations:

os= max

i (yiwi) (3.3)

Where o_s is the resulting ontogenetic signal, y_i is the activation of neuron i and wi is the vector of weights associated to neuron i. Each weight is

normalized between zero and one (Figure 3.5).

Figure 3.5: Ontogenetic Module

(42)

to evaluate the input. These weights are updated at each iteration, using a Hebbian learning function:

wi = wi+ η(hsyi− (wiyi2)) (3.4)

In this equation η is the learning rate, while hsstands for the hebbian signal,

which is a control signal coming from the Intentional Module (it could be the relevant signal itself). A threshold is fixed, so that if a weight is beyond or equal the threshold value, its value is always set to one. This is done to strenghten signals that have already proven to be good enough.

The output of the Ontogenetic Module is the ontogenetic signal, whose value represents how much the actual input state satisfies the new goals developed through the hebbian learning process.

3.2 Motor System: movement generations

In order to accomplish its goals, the agent should be able to move and in-teract with the environment. The problem of moving and acting in a smart way, according not only to hard-coded goals, but also to new developed goals, is not trivial. We suggest a solution to this problem based on Dy-namic Behaviors (for movement evaluation) and on the concept of motor primitives (for movement generation).

The input of this motor part comes from the Intentional Architecture, and it is composed by a vector of neural activations and a relevant signal. The vector, representing the state of the environment in a high level of abstrac-tion, is clustered using K-Means algorithm. The output of the clustering is the cluster corresponding to the current state; this information is used with a State-Action table to choose the best movement to do, according to the relevant signal. The movement is composed by a linear combination of primitives, and it is sent as output to the agent. In order to execute the movement, the agent uses the output of this motor part to compute joints values of its actuators.

3.2.1 Motor primitives: elementary movements

In recent years different lines of evidence have led to the idea that motor ac-tions and movements, in both vertebrates and invertebrates, are composed of elementary building blocks, called motor primitives. Motor primitives might be equivalent to “motor schemas”, “prototypes” or “control modules” [49, 50, 51].

(43)

3.2. Motor System: movement generations 23

Motor primitives could be transformed with a set of operation and combined in different ways, according to well defined syntactic rules, in order to obtain the entire motor repertoire of motor actions of an organism. At neuronal level, a primitive corresponds to a neuron assembly, for example, of spinal or cortical neurons [9, 52].

Studies on the motor system suggests that voluntary actions are composed by movement primitives, that are bonded to each other either simultane-ously or serially in time [6, 7, 8, 9, 10].

Following this idea, we use this concept of motor primitives to create muscular activations that allow the agent to perform a complex movement. A motor primitive could be seen as the activation of a muscle during time. The higher the value of the primitive, the stronger the muscle activation, which will bring to a faster execution of the movement. Activating different muscles in time, a complex movement can be performed. We implement primitives as Gaussian functions delayed in time:

p = e

−(x−c)2

2σ2 (3.5)

Where c is the center of the muscular activation of the primitive p. We chose the bell-shaped profiles of Gaussian function for primitives according to biological evidences: when humans move their limbs from one position to another they generally change joint angles in a smooth manner, such that angular velocity follows a symmetrical, bell-shaped profile [53]. To generate a complex movement, primitives are linearly combined, producing a muscular sinergy:

m =X

i

wipi (3.6)

Each weight w is initially randomly generated during movement generation, thus creating a a random complex movement to be performed (See Figure 2.2).

3.2.2 Dynamic Behaviors: complex movements

In human brain motor cortex is involved in planning, control and execu-tion of movements. Motor cortex is composed by several parts, each one contributing in generation of movements; however, the main contributor to generating neural impulses to pass down to the spinal cord and control the execution of movement is the primary motor cortex. Scientific evidence sug-gests that each neuron in the primary motor cortex contributes to the force

(44)

in a muscle: the more activity in the motor cortex neuron, the more muscle force [54]. Furthermore, the primary motor cortex is somatotopically orga-nized, which means that stimulation of a specific part of the primary motor cortex elicits a response from a specific body region (Figure 3.6).

Figure 3.6: The Somatosensory Cortex and the Motor Cortex

According to the fundamental role of the motor cortex in movement generation, the approach to this task should be similar to the one used for categorization; more precisely, we need a neural code of information like the one computed by the categorization module. Following this idea, the first step in order to perform a movement is clustering (Figure 3.7), like in Categorization Module. Clustering is performed by K-Means algorithm (see Appendix C), the same used in the Intentional Module. However, in this case we don’t need to create new categories, so clusters are defined in a previous training phase. We need the cluster representing the current state, namely the part of the primary motor cortex that is stimulated by the current state. This approach respects the idea that the same neural activation produces the same muscular response.

Movement generation is not the only goal to be accomplished; we need something able to select the best movement according to the current state of the environment, depending on the goals we want to satisfy. Dynamic

(45)

3.2. Motor System: movement generations 25

Figure 3.7: An example of clustering

Behaviors allow the agent to perform the best movement, given the state of the environment and the relevant signal coming from the Intentional Archi-tecture, as well as the ability to learn the best movement to execute in an unknown situation. The relevant signal depends on the ontogenetic signal, coming from the thalamus of the system, and the phylogenetic signal, com-puted by the amygdala of the system; its use in the motor development of the agent is based on scientific evidences, confirming the fundamental role of the basal ganglia, a group of nuclei in the brain containing the amygdala, and of the thalamus in the process of motor learning [20]. Moreover, studies on monkeys prove that lesions on the thalamus seriously impair their motor learning capabilities [55].

Each Dynamic Behavior is composed by a list of actuators of the agent that have to be moved in order to satisfy a goal. If we want the agent to look to a ball, for example, we can create a Behavior linked to the actuators which control head yaw and pitch angles. The Dynamic Behavior selects the set of movements to be executed from the actuators in order to get the best relevant signal as a response from the Intentional Architecture. The com-putation of the best movement to be performed is based on a State-Action table (Figure 3.8); the table associates a state and a movement to a relevant

(46)

signal: when the system is in a certain state and performs a movement, the relevant signal generated by this state-performed movement combination is stored in the table. The policy for movement selection for each input state follow these rules:

• if there is a movement associated with a relevant signal above a defined threshold, that movement is selected;

• otherwise, if there is a movement not performed yet, that movement is selected;

• otherwise, if all movements have already been performed at least once, and no one is associated with a relevant signal above a defined thresh-old, a new random movement is added to the list.

Each movement is a set of weights to apply to primitives in order to obtain a muscular activation. Once a movement is selected, it is used to linearly combine primitives, and the resulting vector is sent to the agent. The agent can use this output to compute the correct joint values and move according to the information it has received.