Analysis of vertebrates’ activity by machine
learning
Thesis for the Degree of Doctor of Philosophy
Rita Pucci
Academic advisors:
Alessio Micheli
1, & Stefano Chessa
11
Department of Computer Science
University of Pisa
Italy
Prof. GianLuigi Ferrari
Department of Computer Science, University of Pisa, Italy
Prof. Roberto Barbuti
Department of Computer Science, University of Pisa, Italy
Abstract:
Activity Recognition (AR) is nowadays a fervent research area which gives many new challenges to deal with. In particular, we focus our attention on the development and analysis of classifiers based on Machine Learning (ML) approaches in the areas of Human Activity Recognition (HAR) and biologging. The literature already presents many different approaches forMLclassifiers for HARproblems, especially in monitoring of humans in domestic environments. Less common is the application ofML classifiers in biologging, which is still commonly studied through traditional methods.
In our research we applied classifiers implemented byMLapproaches from the classes of Neural Network (NN) and of Support Vector Machine (SVM). The classification of the activities was performed over time−series collected by sensory devices. The devices were worn by each subject of the case studies considered. The classifiers were configured and tuned specifically for each case study at hand. In this thesis we dealt with four cases of study: with humans to identify daily activities, with tortoises to identify the digging activity, and with penguins and seals to identify the prey handling activity. These case studies covered an heterogeneous set of both HAR and biologging problems. The classifier applied via shift-window over the time and specifically tuned by input sequences (windows shifted over the input time−series) is implemented by Input Delay Neural Network (IDNN), Convolutional Neural Network (CNN), andSVMwhich naturally deal with these input. For the same case studies, we implemented the classifier by models from the Recurrent Neural Networks (RNN) class, which naturally apply over streams by taking advantage from their internal memory. We evaluated each implementation of the classifier by means of its accuracy and F1 score reached in classification, and by assessing its feasibility for its use into embedded devices in terms of memory space. We demonstrated that with sequences as input, theIDNN model provides a good trade off between performance (accuracy of the model) and feasibility (memory footprint of the model). Instead with streams we observed that the Echo State Network (ESN) reaches a good performance and it is feasible as well because the reservoir can be kept small without a significant penalty in terms of performance of classification.
The results of this analysis would contribute to improve future activity recognition methods. We showed that it was possible to implement efficient classifiers in selected real−world case studies. In particular, such efficiency of the classifiers allows to meet the performance requirements of real applications enabling the embedding of the classifiers into low−power devices. In perspective, we believe that this research will support future research directions with the focus on stimulating research in the directions of animals’ monitoring/protection.
Don’t cry for me please don’t be sad
Hold on to the memories of the times we both had don’t dwell on dark thoughts hold on tight to your wishes Sending you hugs and butterfly kisses
I walk beside you I am there all day long I am right here but you think I am gone You don’t see me but I can see you
What ever the problems I will help get you through I am the wind in your hair the sand in your toes Butterfly kisses that you feel on your nose I am with you at sunrise and in sunset But you can’t see me is my one regret I sit right beside you when you are sad
As you look through the photos of times we both had I watch you sleeping I hold you so tight
Before I go I kiss you goodnight I will watch over you from heaven above Forever you will be my one true love Hold on to your dreams and all your wishes Sending you hugs and butterfly kisses.
iii
Acknowledgments:
"We are such stuff As dreams are made on, and our little life Is rounded with a sleep."
Shakespeare
Alla fine ce l’ho fatta. Dicono che quando concludi qualcosa ti senti leggero e carico per la prossima avventura, così sembra essere anche in questo momento. Che dire di questi anni, sono stati anni di grandi soddisfazioni ma anche di forti rancori, alcune volte sono stati bellissimi e pieni di emozioni, altre volte sono stati orribili, ma come dice mia mamma "sono uno schiaccia sassi" e quindi si va avanti.
Ma passiamo ai ringraziamenti, non tutti possono essere ringraziati in una sola pagina ma si fa quello che si può. Prima di tutto i miei supervisori, Alessio e Stefano. Visto? alla fine ce l’ho fatta! Grazie per esserci sempre stati, per aver cercato di capire il Rita-mondo, per avermi spinto a valorizzarmi invece di screditarmi. Dirò solo che sono stati anni divertenti e formativi per me, con un sacco di risate e di commenti su pdf. Voglio anche ringraziare il Prof Degano, che mette tutto a posto anche quando non sembra possibile, ed Enrico sempre allegro e disponibile.
Un grazie enorme va a mia mamma che si è subita tutti i miei malumori da ansia e le mie gioie da conquista. A lei, la mia mamma, con i suoi "e poi si risolve" e "che vuoi che sia", voglio dire grazie perché sono stati tre anni complicati, mammetta, ma ce la si fa, ce la si è sempre fatta. Grazie a mia sorella, l’unica che riesce a farmi stare calma quando mi prende l’ansia e che trova sempre una lettura pragmatica ad ogni situazione. Alla fine forse con calma "ce sto".
Grazie Fede, grazie per tutto. Hai ragione, bisogna sempre pensare che è una cosa bella sennò diventa solo un brutto ricordo. Non saprei nemmeno dirti a parole quanto "grazie" è questo "grazie" quindi finisco qua senza altre parole perché in fondo non serve scriverle.
Adesso iniziamo i ringraziamenti agli amici, quelle personcine con cui ho fatto un saccone di risate, tutte quelle persone un po’ strane e sbilenche che hanno popolato la mia vita in questi anni. Grazie a Silvia, a Patrizia e a Giulia le mie tre donne che mi conoscono in ogni mia brutta e bella sfaccettatura. Grazie a Luigi, mio vecchio amico di universitá, che, anche se ci dividono terre e mari, non si è mai dimenticato di ascoltarmi e ridere con me. Grazie ad Andrea, un super amico che mi insegue per il mondo, ma come si fa a non voler bene ad un
capoccetto così? una personcina speciale. Grazie alle mie amiche di sorellanza Marta e Chiara, nei momenti belli e nei momenti brutti le uniche che ridono e piangono con me. Mai come in questo anno ho capito quanto sia importante conoscervi. Grazie a Tiziano, che ha ascoltato tanto di me e silenziosamente mi ha fatto capire che parlare ascoltando fa la differenza. Mi hai fatto ridere tanto e pensare che ero io quella che portava allegria. Grazie al bar Cornolti, grazie a Francesco, sempre solare e radioso a suo modo, grazie a Marco e a Lillo, con cui ho passato un sacco di pause "video divertente?". Grazie alla mia donno, così spietatamente divertente. Grazie a Barbara e a Laura, che donne ragazzi. Grazie a Daniele, Mary, Lucia, Alessio, Jacopo, Gaggo, Michael, Roby, Walter, Lorenzo. Grazie ai miei amici lontanissimi Hamed, Vikram. Grazie a un sacco di gente ma tanta tanta che in questi anni c’è stata e che in questi anni ho vissuto.
Ho imparato molto in questo dottorato, ne sono uscita ammaccata ma ne sono uscita e la me risultate è quella che ho e anche se non è forte come mi aspettavo e brava come volevo è quella che ho e quindi le voglio bene uguale. L’ultimo grazie va a me, perché ho lottato con le unghie e con i denti per fare quello che volevo e un primo passo l’ho fatto. Grazie.
Contents
Acronym ix
I
Basics
1
1 Introduction 3
1.1 Objectives of the Thesis . . . 5
1.2 Methodology . . . 6
1.3 Main achievements of the thesis. . . 8
1.4 Plan of the Thesis . . . 10
1.5 Relevant publications. . . 11
2 State of the art 13 2.1 Activity recognition . . . 13
2.2 Machine learning for human activity recognition . . . 15
2.3 Machine learning for animal activity recognition . . . 18
3 Background 27 3.1 Machine Learning. . . 27
3.1.1 Multi Layer Perceptron . . . 28
3.1.2 Recurrent Neural Network: Echo State Network . . . . 33
3.1.3 Support Vector Machine . . . 36
3.2 Validation technique . . . 37
4 Methodology 39 4.1 Hardware material . . . 39
4.2 Dataset . . . 40
4.3 Configuration of classifier input/output . . . 41
4.4 Activity recognition through an automatic classifier. . . 43
4.4.1 Filter stage . . . 44
4.4.2 Classifier stage . . . 45
4.5 Validation schema for the experimentation. . . 46
4.6 Statistical significance tests . . . 49
4.7 Summary of terminology . . . 50
activ-ity
53
5 Daily activities in humans 55
5.1 Introduction. . . 55 5.2 Dataset . . . 56 5.3 Methods . . . 57 5.3.1 Filter stage . . . 58 5.3.2 Classifier stage . . . 59 5.3.3 Validation schema . . . 62 5.4 Model validation . . . 63 5.5 Experimental analysis . . . 65 5.6 Discussion . . . 70
6 Nesting activity in tortoises 71 6.1 Introduction. . . 71
6.2 Tortoise@ system . . . 73
6.3 Dataset . . . 76
6.3.1 Collection hardware . . . 77
6.3.2 Data collection protocol . . . 78
6.4 Methods . . . 79
6.4.1 Filter stage . . . 79
6.4.2 Classifier stage . . . 80
6.5 Experimental Analysis . . . 90
6.5.1 Identification of digging activity in sequences . . . 90
6.5.2 Identification of digging activity in a stream. . . 104
6.6 Discussion . . . 107
7 Prey handling activity in little penguins 111 7.1 Introduction. . . 111 7.2 Dataset . . . 113 7.3 Methods . . . 114 7.3.1 Filter stage . . . 115 7.3.2 Classifier stage . . . 117 7.3.3 Validation schema . . . 118 7.4 Experimental analysis . . . 119
7.4.1 Identification of prey handling activity in a sequence . . 119
7.4.2 Identification of prey handling activity in a stream . . . 125
7.5 Discussion . . . 126
8 Prey handling activity in seals 129 8.1 Introduction. . . 129
Contents vii 8.3 Methods . . . 132 8.3.1 Filter stage . . . 133 8.3.2 Classifier stage . . . 133 8.3.3 Validation schema . . . 134 8.4 Experimental analysis . . . 135
8.4.1 Identification of prey handling activity in sequence . . . 135
8.4.2 Identification of prey handling activity in a stream . . . 141
8.5 Discussion . . . 142
III
Discussions and Conclusions
145
9 Discussions and Conclusions 147 9.1 Summary of case studies . . . 1479.2 Discussion . . . 149
9.3 Conclusions . . . 151
IV
Bibliography
155
Acronym
AAL Ambient Assisted Living
AAR Animal Activity Recognition
AR Activity Recognition
AReM Activity Recognition system based on Multisensor data fusion
ARS Activity Recognition System
CNN Convolutional Neural Network
CSV Comma−separated values
CV Cross−Validation
ESN Echo State Network
ESP Echo State Property
EvAAL Evaluating Ambient Assisted Living (AAL)
GPS Global Positioning System
GDM Gradient Descent with Momentum
HAR Human Activity Recognition
HC Health Care
IDNN Input Delay Neural Network
LM Levenberg-Marquardt
LoRa Low Power Long Range
LRF Local Receptive Fields
MEMS Micro−Electrical and Mechanical Systems
MLP Multi Layer Perceptron
MSE Mean Squared Error
ML Machine Learning
NN Neural Network
RBF Radial Basis Function
RC Reservoir Computing
RNN Recurrent Neural Networks
RP Resilient Propagation
SVM Support Vector Machine
TDNN Time Delay Neural Network
WS Weight Sharing
Part
I
1
Introduction
Nowadays systems able to "figure out" activities performed by a subject, are one of the main component for many projects addressed to the improvement of human life and/or to the protection of animals. The Activity Recognition(AR) research field, investigate automatic methods to recognise activities from data collected by sensors. In this thesis we focus on theARapplied with humans and animals in daily activities through data collected by individual-attached recording devices, such as wearable devices.
In last four decades several different approaches have been proposed with this purpose and recentMLapproaches demonstrate to be valid forARin specific tasks, such as mothorial activities. The use of MLappriaches is studied in this thesis by considering some, meaningful cases of study, and by taking into consideration both performance and feasibility. The term performance in this context is used with the meaning of: the capacity of a technique to identify properly an activity" and it is evaluated by the accuracy and the F1score values. The feasibility term is the application of a Machine Learning MLapproach within the specific constraints arising in a real-world application, which may include physical constraints (e.g. size or weight) on the wearable devices and constraints on the communication, storage and/or processing capabilities of the devices.
In the following, we introduce the human’s and animal’s activity recognition areas and we conclude this introduction by summarizing the main objectives and contributions of the thesis in these areas.
In literature for both applicative areas, data are collected by means of the sensors in the wearable devices following either bio-logging or biotelemetry collecting procedure. In the first procedure, during the activity the device logs data and stores them in the internal memory, hence the data are not available till retrieval of the device. The second procedure takes advantage of wireless communications, then during an activity the device logs data and immediately sends them to a remote station without storing it. We take in consideration application of ML approaches in systems that apply mainly the bio-logging procedure and occasionally they can use also biotelemetry.
Hereafter in this thesis we refer toARfor humans with the name of the related research area Human Activity Recognition (HAR). With animals there is no a specific name introduced to identify the ARresearch area releated to animals then we use the name Animal Activity Recognition (AAR) for sake of clarity to referARwith animals performed by bio-logging collecting procedure. Started in the 90s with [1], AR applied for humans spans several, heteroge-neous applicative areas, ranging from medical, military, security applications, entertainment, Ambient Assisted Living(AAL) etc. [2]. The thrust given by these applications to the research of sensors and techniques for HAR, have brought an exceptional development of off-the-shelves, cheap and miniaturized devices that opened opportunities for new applications and business. All this gave particular benefits to theAALapplications, for which cost and ease of use are essential features. Under this respect, the widespread use of smart phones and wearable devices enable an even more pervasive monitoring of activities. In this application HARis a key element, since inferring the activities of the users in their own environments (either at home, work or outdoor) it allows the offering services to the users in the right place and at the right time, thus providing assistance, care and ensuring safety.
The other area of interest of AR is the monitoring of wildlife, which gives a significant contribution since the 60s to a deeper understanding of hidden aspects of the animal’s behavior, and, in turn, opens new perspectives to the campaigns supporting biodiversity, helping ethologists worldwide in direct monitoring. The methods of observation in ethology are traditionally achieved by the presence on the field of the researcher (observer). However, while it allows biologists to obtain a clear description of the animal life, it is particularly arduous in the case of most wildlife due to their high mobility, nocturnal life, and particular habitat that can be dangerous for observers. Furthermore, the observer itself is an unusual presence in the habitat and his presence induces the animal to modify its natural behaviour.
Both applications of AR (for humans and animals) share the same main challenges introduced in [3]:
i Construction of a portable, and unobtrusive data acquisition devices and analysis systems;
ii Collection of data under realistic conditions and selection of features; iii Data analysis approaches.
The first challenge concerns the limitation of the invasiveness of the device on the life of the observed subject. The invariveness is due to the dimension and the weight of the device, then it is important to consider device proportioned to the body of the subject and light enough to do not limit his movement. The
1.1. Objectives of the Thesis 5
device has also to be configured appropriately to provide coherent and consistent data. Data has to be realistic then it is important that they are collected in field from the subject under study, and that such data are associated to the corresponding activities of the subject. Since the device design respects the first challenge about dimension and the second challenge about the consistency of data the third challenge points the attention to data analysis approaches. In particular, most recent developments of technology in wireless sensors enabled the embedding of sophisticated algorithms for the analysis of sensed data directly on board[4]. This, in turn, allows such devices to operate autonomously for longer periods because they make a more efficient use of their energy, communication and storage resources as described in Tortoise system [5].
1.1
Objectives of the Thesis
Our general objective is to develop and apply state-of-the-artMLapproachess forARproblems addressed to humans and animals.
We focus our attention to four cases, one concerning humans and three con-cerning animals. The reason for selecting different cases and different ML approaches is two-fold: to obtain a good coverage of heterogeneous case studies and to analyse different approaches for theARproblems presented. However, the choice of the case studies is limited by the availability of datasets suit-able for our studies. Furthermore the complexity in collecting phase of the bio-logging problems have driven first our selection of case studies. In our case we have the opportunity to use the datasets from [6], [7], and [8] to test anAALscenario in humans. For wild animals, we use a dataset on tortoises that we built in our previous studies [9], and [5]. Furthermore, through a collaboration with the Macquarie University and the University of Queensland (both in Australia), we have access to two rich datasets on penguins and seals, respectively [10]. Although these datasets are available mostly for contingent situations, they met our main requirement of heterogeneity in terms of different subjects (both humans and animals are present), different activities (daily activities for humans, nesting for tortoises, prey handling for both penguins and seals), different environment (troposphere for humans and tortoises and hydrosphere for penguins and seals), different devices and sensors and different position of the sensors on the subject’s bodies. As all the considered animals are wild rather than pets, this brought a more challenging analysis of data, since the datasets are obtained in a less controlled environment.
We decided for theIDNNandCNNmodels for theNN, for theESNmodel for theRNN, and theSVMfor the kernel based. These approaches are the most commonly used in literature to deal with theARin bothHARand bio-logging as described widely in Chapter 2. Further characteristics and motivations to their use will be provided in Chapter 3.
In our study we consider each case of study individually with the following objectives at hand:
• With the humans’ dataset we analysedARin daily activity recognition. In particular we deal with the identification of lying, sitting, standing, walking, falling, and cycling activities (activities that are known to be present in the dataset), by means of wearable sensors. In this case the sensors used were IRIS devices [8] worn by the users on chest and on the ankles;
• With the tortoises’ dataset we worked towards the identification of nesting activities. In this case, the dataset contains time series recorded by accelerometer sensors embedded in a MicaZ device [5];
• With the penguins’ dataset the aim is to identify the prey handling activ-ities. The penguins’ movements time series are provided by a waterproof accelerometer data logger [10];
• With seals’ dataset we also identify the prey handling activity. This dataset was produced by using a similar accelerometer data logger as that used for penguins.
1.2
Methodology
To provide theARof a subject we analyse time series of data produced by one or several wearable and heterogeneous sensors. In the case of humans such sensors are worn on the wrist, on the ankles, or on the chest and they measure 3D acceleration and proximity of the subject to a token. In the case of animals, we consider time series generated by 2D accelerometer with tortoises and 3D accelerometer with penguins and seals which are placed on the back of the animals.
Since we have the time series the ARproblem is specified as in [2]. TheAR problem is a mapping function that provides for each pattern a classification as similar as possible to the actual activity performed. Where we identify with the name pattern a temporal interval extracted from the time series and each
1.2. Methodology 7
pattern has to be labeled with the activity performed by the subject. The temporal data of each pattern have to be consecutive.
The mapping function is here implemented by a two stage automatic classifier structure composed by a filter stage and a classifier stage. The filter stage applies several known techniques over the pattern (such as sampling, feature extraction, and normalization) to increase the performance of the subsequent classifier stage. The classifier stage provides the mapping function by using a supervisedMLapproach for each "filtered" pattern. The overall automatic classifier structure is evaluated in terms of the performance obtained (accuracy of the classifier stage) and of its feasibility (memory footprint of the entire structure). This analysis also provides a trade off between the performance and feasibility.
TheMLapproaches can be effortlessly developed on low-power devices. This meets the most recent trends of research on wireless sensors that exploit on-board data analysis to reduce the amount of data to be stored and/or , if it is necessary, transmitted, thus improving the efficiency and lifetime of the sensors. Furthermore, this also enables a new generation of systems that can remain operative for long periods while worn by the animals and that require sporadic communications, thus resulting in less obtrusive devices. All this induced us to chooseMLapproaches that require low resources. This led us to consider IDNNs,CNNs, andESNs.
The IDNN is considered the base line for the HAR systems. In fact, this model is the natural reference in the NN for HAR. We used this model as ground to develop customized models for the problem at hand. The CNN model is a natural evolution ofIDNN. This model addresses theHARproblem ensuring the degree of shift and distortion invariance in pattern recognition. An alternative approach is represented byRNNarchitectures that deal with sequential data. In particular we analyse theESNmodel due to its efficiency, which has recently obtained promising results withHAR applications [11]. TheSVMmodel, which is not suitable for embedding in low-power sensors, is applied in this thesis at the current state of the art in off-line analysis ofAR data, and it is thus an important term of comparison for the other models.
1.3
Main achievements of the thesis
This section summarizes the contributions for each case of study.
- Chapter5, Human activity recognition using multisensor data fusion based
on Reservoir Computing: Activity recognition plays a key role in providing
activity assistance and care for users in smart homes. In this case study, we present an activity recognition system that classifies in the near real-time a set of common daily activities exploiting both the data sampled by sensors embedded in a smartphone carried out by the user and the reciprocal Remote Sensing Systems (RSS) values coming from worn wireless sensor devices and from sensors deployed in the environment. In order to achieve an effective and responsive classification, a decision tree based on multisensor data-stream is applied fusing data coming from embedded sensors on the smartphone and environmental sensors before processing theRSSstream. To this end, we model theRSSstream, obtained from a Wireless Sensors Network (WSN), usingRNNimplemented as efficient ESN, within the Reservoir Computing (RC) paradigm. We targeted the system for the EvAAL scenario, an international competition that aims at establishing benchmarks and evaluation metrics for comparing AALsolutions. In this paper, the performance of the proposed activity recognition system is assessed on a purposely collected real-world dataset, taking also into account a competitive neural network approach, an IDNN, for performance comparison. Our results show that, with an appropriate configuration of the information fusion chain, the proposed system reaches a very good accuracy with a low deployment cost.This research was published on the Journal of Ambient Intelligence and Smart Environments [8]. This paper also provides a dataset of accelerometer andRSSdata.
- Chapter 6, Nesting activity in tortoisesTestudo hermanni: The goal of this research is to recognize the nest digging activity of tortoises using a device mounted atop the tortoise carapace. The device classifies tortoise movements in order to discriminate between nest digging, and non-digging activity (specifically walking and eating). Accelerometer data was collected from devices attached to the carapace of a number of tortoises during their two-month nesting period. Our system uses an accelerometer and an activity recognition system (ARS) which is modularly structured using an artificial neural network and an output filter. For the purpose of experiment and comparison, and with the aim of minimizing the computational cost, the artificial neural network has
1.3. Main achievements of the thesis 9
been modelled according to three different architectures based on the input delay neural network (IDNN). We show that the ARS can achieve very high accuracy on sequences, with an extremely small neural network that can be embedded in programmable low power devices. Given that digging is typically a long activity (up to two hours), the application of ARS on data sequences can be repeated over time to set up a reliable and efficient system, called Tortoise@, for digging activity recognition. This research was published in PlosOne [5]. This paper also provides an original dataset of accelerometer data recorded with mediterranean tortoises. Furthermore, in this thesis we present new results obtained with the application of anESNmodel. TheESNobtained high performances on time-series with small structure and introduced the classification of data on data streams of accelerometer.
- Chapter7, Prey handling activity in Little Penguins(Eudyptula minor): The focus is to identify where, when and how much animals eat the Little Penguins. This analysis could provide valuable insights into their ecology, and could improve the supportive programs for their protection. We present a comparative analysis betweenSVMandIDNNmodels to identify prey capture events from penguin accelerometry data. A pre-classified dataset of 3D time-series data from back-mounted 3D accelerometer is used. We train both the models to classify the penguins’ activity at intervals as either ’prey handling’ or ’swimming’. We analyse different settings for both models and we determine whetherIDNN achieves the same level of classification accuracy asSVM. We prove that for the same level of accuracy, theIDNNhas a lower memory footprint than theSVM. This enables the IDNN model to be embedded on the accelerometer micro-system itself, and hence reduce the magnitude of the output data to be uploaded. Based on the classification results, this section provides an analysis of the two models from both an accuracy and feasibility point of view. These results are submitted for journal publication. We developed the analysis applying theESNfor the same task. The ESN provided high performances with small memory footprint.
- Chapter8, Prey handling activity in seals (Otariid pinnipeds): Foraging behaviour in seals is a topic widely investigated because of its importance to understand the ecology of these animals. The analysis of data in order to identify the prey handling activities could provide important information to improve supportive programs. In particular, the use of an embedded system to identify such activity can provide an unobtrusive solution to monitor seals in wild. We developed and analysedMLmodels in order to discern the ’prey handling’ activity from the ’swimming’ activity. Models were trained on dataset of tri-axial accelerometer and
pressure data from back-mounted device. We appliedIDNN model and SVMon tailored time windows and we observed a similar performance with both IDNNandSVM with a reduced memory occupations. This allows to identify in anIDNNmodel a good trade-off between performance and memory required for an embedded solution on the micro-system itself. We appliedESNon streams of data and observed high performance with small reservoir.
1.4
Plan of the Thesis
The thesis is organised in three parts:
• The first part is entitled "Basics"; it investigates the background of knowledge needed for this thesis. This part is organized in the following chapters:
– "State of the art": it presents the works that describe the evolution of ARrelated to the use of wearable sensors for humans. In this chapter we introduce the concept of Human Activity Recognition (HAR). In particular, the use of MLapproaches to deal with time-series inHAR systems is investigated. In the second subsection, we analyse the evolution of AR with animals by describing some of the works presented in literature. We describe the evolution of data anlysis system based onMLapproaches based on bio-logging collecting procedure.
– "Background": it describes the models and the validation schema thus presents the common structure used for every case of studies. In particular, it presents theMLapproaches for pattern classification, and its application on time-series of sensed data. It introduces the MLapproaches ofNN,RNN, andSVM, which are the models used in this thesis for the experimentation. Finally, the validation schema is presented followed in each cases of study.
– "Methodology": it presents the common structure used in the applica-tive case-studies. In particular, it introduces the datasets structure, configuration of classifier input/output, the structure of the auto-matic classifier, and, for each considered technique and model, it introduces its method of application.
• The second part, which is entitled "Machine Learning for metazoa activity
1.5. Relevant publications 11
describes the specific dataset, methodology, model validation technique, and experimentation used in its case study.
• The last part, entitled "Discussion and Conclusions", comprises one chapter that synthesizes all the results obtained from each case of study.Furthermore, we present the goals reached with this thesis and the future goals we will pursue.
1.5
Relevant publications
This thesis is based on the following publications:
1. 2012: Journal Article: Barbuti R., Chessa S., Micheli A., Pallini D.,
Pucci R. and Anastasi G., “Tortoise@: a system for localizing tortoises during the eggs deposition phase.", Atti Societa Toscana Scienze Naturali, 2012, memorie B., pii: 119:89-95.
2. 2014: Workshop Article: Barbuti R., Chessa S., Micheli A. and Pucci
R., “Identification of nesting phase in tortoise populations by neural networks. Extended Abstract.", The 50th Anniversary Convention of the AISB, 2014, Selected papers at workshop Intelligent System for Animal Welfare (ISAWEL), pii: 62-65.
3. 2015: Journal Article: Palumbo F., Gallicchio C., Pucci R., and Micheli
A., “Human activity recognition using multisensory data fusion based on reservoir computing", Journal of Ambient Intelligence and Smart Environments (JAISE), 2016, 8(2), pii:87-107.
4. 2016: Journal Article: Barbuti R., Chessa S., Micheli A. and Pucci
R., “Localizing tortoise nests by neural networks", PlosOne, 2016, 11(3), pii:e0151168.
5. 2016: Proceedings of Doctoral Consortium: Pucci R., “Evaluation and
Deployment of Models for Activity Recognition”, Doctoral Consortium of the 15th Edition of AI*IA. 2016, pii: 64-69
6. 2017: Journal Article: Chessa S., Micheli A., Pucci R., Carroll G.,
Harcourt R., Hunter J., “A comparative analysis of SVM and IDNN for identifying penguins activities", Applied Artificial Intelligence: An International Journal, 2017.
7. 2017: Journal Article: Chessa S., Micheli A., and Pucci R., “Wild
8. 2017: Journal Article: Pucci R., Chessa S., Micheli A., Hunter J.,
“Machine learning approaches for identifying prey handling activity in otariid pinnipeds”, Submitted for journal publication.
9. 2017: Poster session: Pucci R., Micheli A., Chessa S., “Machine
Learn-ing approaches to analyse data collected via bio-loggLearn-ing procedure in Tortoise@”, Animal-Computer Interaction. 2017.
10. 2017: Poster session: Pucci R., “Advanced Biologging by Machine
2
State of the art
In this chapter we will present the development of the AR research area in last four decades. We will first analyse the development in Human Activity Recognition (HAR), in particular in the application toAAL, and the evolution of Animal Activity Recognition Animal Activity Recognition (AAR).
2.1
Activity recognition
The activity recognition is based on data collected through sensors embedded on individual-attached recording device worn by a subject (hereafter referred as device).Usually, most of the data measured by sensors (i.e. accelerometer axes, depth values etc.) are related with the movements of the subject, the physiological conditions, or the environmental conditions, and they are all recorded as continuous raw time series data.
The collection data procedures applied in activity recognition are either the bio-logging, the biotelemetry, and an hybrid solutions. The bio-logging indicates the log of signals (time series data) collected by embedded sensors on board of a device. Data are stored on the flash memory of the device and kept in memory until the device is retrieved by a researcher. In biotelemetry instead, the collect data are immediately sent to a remote base station, without storage. This distinction was presented in 2004 by Robert-Coudert and Wailson in [12]. Hybrid solutions perform both procedures, data are stored on the device and sent to a remote data collection station.
The type of sensors used for a data collection campaign depend on the mature of data in which the researchers are interested. For example accelerometer measure, depth measure, and temporal degrees deliver: environmental sensors and wearable sensors. The former provides context information (e.g. temperature, humidity, light, etc.). The latter provides biological information (such as body movement, heart beat, etc.). In this thesis we consider data collection
campaigns (either biologging, biotelemetry or hybrid)conducted with wearable devices. This solution uses sensors attached to the subject to measure directly his/her physical, environmental or biological parameters. This method could be perceived as intrusive but with the recent advances in technologies of embedded systems, sensors tend to be small enough and lightweight. Furthermore, they can embed a large number of heterogeneous sensors that can fit several, different purposes.
Given S a set of time series collected through bio-logging we consider the Activity Recognition Problem with the followed definition proposed in [2]:
Activity Recognition problem:
Given:
• A set S = S1...Sk of k time series, each one from a particular measured
attribute of each sensor, and all defined within time interval I = [tα; tω].
Where α is the start time and ω is the end time.
The goal is to find a temporal partition < I0...Ir−1> of I, based on the data S,
and a set of labels representing the activity performed during each jth interval
Ij where 0 < j < r − 1. This implies that the time intervals Ij are consecutive,
non-empty, non-overlapping.
By this definition, which does not limit the number of attributes and the length of the time series to be analysed, the task of AR becomes hard to deal with. A relaxed version of this definition assumes the time series divided into fixed length time windows. Specifically as defined in [2]:
Activity Recognition problem relaxed:
Given:
• A set W = W1...Wmof m equally sized time windows, totally or partially
labeled. Each Wi contains a set of time series S = Si1...Sik from each of
k measured attributes; A set A = A1...An of activity labels.
The goal of the relaxed problem is to find a mapping function f = Si→ A that
can be evaluate Si in order to obtain that f = (Si) is as similar as possible to
the actual activity performed in Wi.
In the literature, different solutions are proposed to face this defini-tion of the problem. Over time, due to the interest of research and the consequent improvement of technologies in this field, the solutions become progressively more accurate and customised.
2.2. Machine learning for human activity recognition 15
2.2
Machine learning for human activity recognition
HARresearch field investigates how to provide accurate information on humans’ behaviour by recognizing their activities. Obtaining these information have become a task of interest mainly for all the disciplines and services by being automatized. HARis founded on established research fields such as ubiquitous computing, context-aware computing and multimedia, and machine learning for pattern recognition. Recognizing everyday life activities is a challenging application in HAR, with a lot of interesting developments in the health care domain, the human behaviour modeling domain and the human-machine interaction domain [13]. Both classes, environmental and wearable, of sensors are used simultaneously in the HAR systems. In fact, the combination of the two enriches the information, thus augmenting the capability of theHAR systems in recognising activities. Most of the measured attributes are naturally indexed over the time dimension. In particular, cameras installed in domestic environments are probably the most common [14] as solutions based exclusively on environmental devices. However, even though this approach is physically less intrusive for the user, it suffers from several issues: low image resolution, target occlusion and time-consuming processing, which is still a challenge for real-time activity recognition systems. Furthermore, user’s privacy is also an important issue, especially if cameras are used for continuous monitoring. A thorough analysis of the use of environmental sensors is presented in liter-ature in [15], [16], and [17]. In these solutions the aim is to recognize more complex activities by a network of environmental sensors. More recently, a new generation of non-wearable solution has emerged. An example is given by solutions that exploit the implicit alteration of the wireless channel due to the movements of the user, which is measured by devices placed in the environment; an example is the measure of theRSSof the beacon packets that they exchange among themselves [18], [19].
However, solutions that make use of wearable devices, as in [20], [21], and [22], are the more common. In this case, the required sensors are embedded on a wearable device, which, in turn, should be placed on the subject in the most appropriate way. Note that the positioning of the device on the body of the subject is quite important as it affects the performance of the entire system. In [23], authors investigate the position impact for wearable devices. They consider devices with accelerometer and light sensors embedded and demonstrate that some positions are better for particular activities, and they also confirm that there is not a generally valid position for all the possible activities to be recognized. In [24], authors suggest that adding more sensors
improves the recognition of some activities. Hybrid solution with environmental and wearable devices are the main idea ofAALsystem to detect human activity in domestic ambient, [25], [26].
The kind of activity to be recognized in AAL applications depends on the problem at hand. An activity can be represented by a single movement, or a single action (such as sitting, running, jumping), by a group of activity (cooking, watching television, going to work), or by a complex, social activities (such as dancing). Hence, HAR systems range from systems able to recognize simple activities (e.g. lying, sitting, standing, walking, bending, cycling, falling), up to more complex systems for more complex activities.
When theHARproblem concerns the recognition of simple activities, theHAR system usually has to process sensed data in form of time series, and thus their processing involves computational learning tasks characterized by a sequential nature. Furthermore, the various estimations o the activity performed by the user can be considered to be discernible basing on specific patterns of activations/values from a typically heterogeneous set of possibly noisy sensor sources (with potentially both continuous and discrete values), and based on the temporal order of such series.
In this scenario, some approaches to the analysis of time series are based on the use of multiple classification systems within a hierarchical activity recognition model (e.g. [27], [28]). However, all these aforementioned discriminative approaches are based on learning models suitable for flat data domains, and therefore they are often restricted in the processing of sequential/temporal information. Other approaches which are also limited in the processing of dynamical information are based on decision tree models (e.g. [20], [2], [23]), instance based learning (e.g. [29]) or linear discriminant analysis (e.g. [30]). To analyse the time series,ML approaches have found wide applications in buildingHAR systems based on data generated from sensors. Depending on the nature of the treated data, of the specific scenario considered and of the admissible trade-off among efficiency, flexibility and performance, different ML models have been applied [2]. Also examples of Evolving Fuzzy System with respect to differentMLmodels can be found in [31]. In the neurocomputing area, neural network models for sequential domains processing represent good candidates for applications in human activity recognition problems, as they are characterized by the ability to effectively learn input-output temporal relations from a potentially huge set of noisy and imprecise heterogeneous streams of sensed data. In theMLcontext, we consider the solutions based on supervised learning models, of which NN represent a first example. The NN provides discriminative models interesting from the point of view of the computational effort and predictive accuracy, such as [32] and [33]. In this context, delay neural
2.2. Machine learning for human activity recognition 17
networks are a popular class of models [34], which represent temporal context using windowing/buffering techniques in conjunction with feed-forward neural architectures. In this concern, theIDNNmodel in [35] and [36] is a paradigmatic approach, and it is therefore considered for performance comparison. Several works in literature report examples of applications of delay neural networks to problems in the field of human activity recognition, taking into consideration temporal windows with length typically in the range of 2–10 seconds. For example, in [37] and [38], delay networks are used to classify daily-life activities from accelerometer data, whereas a similar approach is used in [39] for human gesture recognition, and in [40] for recognition of workers activities. In [41] a delay network based approach is used to recognize human postures and activities from data collected by a smart-shoe device. More recently, delay neural networks are used for daily activity monitoring using accelerometer and gyroscope data generated by a smartphone [42].
The versatility of IDNN based approaches in this context is also illustrated by applications in the area of animal activity recognition as described in the following section. Note, however, that windowing strategies adopted in delay neural networks may often imply several drawbacks, mainly related to the fact that the window length is a fixed a-priori defined parameter of the model, determining its memory length, i.e. the limit of the length of the longest sequences that can be successfully discriminated. An alternative approach is represented byRNNs architectures [34], [43] with explicit recurrent connections, which are capable of dealing with sequential/temporal data through recursive processing based on the presence of feedback delay connections. A particularly efficient approach toRNN training is represented byRC [44], [45] networks, and in particular byESNs [46], [47]. Recently, promising results have been reported in applications of RC networks in the fields of AAL and human activity recognition. In this context, the RC approach has been introduced and successfully experimentally assessed in tasks related to robot localization [48] and indoor user context localization in real-world environments fromRSS data streams [49],[50],[51],[52]. A further application of RC models for real-time user localization in indoor environments is presented in [53], in which by adopting a hybrid approach, RC networks have been showed to provide significant benefits to the accuracy ofRSSbased localization systems. It is also worth mentioning the results of the European FP7 Project RUBICON [54],[49], whose goal was to design and develop a self-learning robotic ecology made up of sensors, actuators and robotic devices. Within the aims of the RUBICON project, ESNs distributed on the low powerful nodes of a WSN have been used to approach supervised computational tasks in the field of AAL, and pertaining to the classification of human activities, using input data streams coming from sensorized environments. The work in [4] describes the application ofESNs to a set of supervised tasks pertaining to the recognition of
user daily−life activities, within an AALreal−life test bed scenario, including more than 50 sensors of different types. Moreover, an application of ESNs to adaptive planning for context-aware robot navigation is presented in [51]. Further applications of ESNs in the area ofAALare reported in [55], in which ESNs are used to classify physical activities in equestrian sports from a tri−axial accelerometer, a gyroscope and a magnetic sensor stripped to a horse rider wrist, and in [56], in which standardESNs are used to estimate people count from multiple PIRs in indoor corridors. Overall, the aforementioned applications of the RC approach to problems in the area HARshow the potentiality of this methodology, which still deserves further investigations and deep experimental assessment. In the area ofHARis widely adopted theSVM[57]. In [58],SVMs are used to approach an activity recognition problem considering a user in a smart environment with a set of sensors. In [59], [60],[61],SVMs are used for problems of activity recognition using input from tri-axial accelerometer data.
2.3
Machine learning for animal activity recognition
In recent decades, the number of species in the Red List - which classifies globally endangered plant and animal taxa - is increasingly enlarged highlighting a severe crisis of the biodiversity [62]. Projects of animal conservation are developed as remote assessments in favor of the biological studies. The variety of plant and animal life in the world or in a particular habitat is usually considered to be important and desirable and it has to be protected to maintain the balance between the species. An aid in understanding is considerably represented by the analysis of the animal behaviour. In fact, the information extracted by the observation of animals’ behaviour are useful to identify how to intervene for a supportive program, [63]. A technique to survey the behaviour of wild animals is the direct observation on field. It allows biologists to obtain a quantitative description of the animal activities, but it can be challenging in cases of wildlife where the animals have high mobility, nocturnal life, or they life in dangerous wild habitat. An example is the hydrosphere, which hosts 71% of Earth’s fauna, and it represents the most important habitat from the point of view of biodiversity. This habitat is difficult to explore by humans, hence the information about the behaviour of marine life is not often observable directly on field. Devices are applied in such scenarios to provide a direct observation through sensors. The information obtained through sensors are collected via bio-logging on board or they are sent via biotelemetry.
2.3. Machine learning for animal activity recognition 19
on animals mainly focused on the extension of recording period time. The revised architecture of time depth sensor made possible for researchers to record signals from 14 days up to 3 months. The recording time period is a crucial pre-requisite to study animals that spend long period far away from the land, such as seals and big marine mammals. At that time, this pre-requisite was concerning mainly the energy available on the device. An example is the first device applied in underwater in 1965, this device was equipped with a time−depth sensor and it was used in measuring the diving capacity for one hour of Weddell Seals (Leptonychotes weddelli) [64] [65]. Researchers attached the device on seals which were used to visit often the same holes. This characteristic made possible the retrieve of the device to download the data collected via bio-logging.
In [66], [67], the wearable devices were applied to monitor elephant seals (Mirounga angustirostris). As the application of devices on animals matured, the researchers were interested in devices small enough to be worn by small animals without hindering their movements. This advancement was made possible by taking advantage from the MEMS (micro-electrical and mechanical systems) [68] technologies. MEMS provides sensors and antennas embedded in microsystems. An example is the study of hawksbill turtle (Eretmochelys imbricata) with a time-depth data-logger presented in [69]. Turtles worn devices on their carapace and the devices were small enough (it was less than 10 cm) to allow the turtles to dive naturally. In this work, turtles were let swim in a restricted area for a settled period of time, after which devices were retrieved. Since 2003 in [70], [71], [72], [12], [62], [73], [74], [75], [63], and [76]) major reviews on bio-logging and biotelemetry with animals, there are explained further refinements and advancements in devices (such as storage capacity, lifetime, and number and types of sensors on board). In fact, the innovation provided increasingly complex devices that incorporate a range of sensors aimed at detecting more complex information. An example, Ropert-Coudert, and Kato in [77] monitored seabirds, penguins, and seals equipped with devices and the animals are used to construct living observatories to study predators in remote marine locations. Other relevant examples are presented in [78], [79], [80], [81], [82], [83], and [84].
We focus our attention on devices equipped with accelerometers. These sensors are widely applied forAARsuch as in [85], [86], [5], [87], and [63]. The use of an accelerometer introduced the possibility to detect specific movements of an animal’s body. The accelerometer has a significantly low intrusiveness, great energy efficiency, and much small size. Accelerometry data are not influenced by weather conditions or the inclination of the sensor and they can be used for many different tasks. Examples are the use of 3D accelerometer with elephant seals as in [88]. In [89] accelerometer data are used to detect when southern
elephant seals (Mirounga leonina) open and close their mouths to monitor foraging activity.
Table 2.1: Selected examples of bio-logging studies. The first column lists animals
subjects of studies, the second column lists sensors used to collect behavioural data and the third column is the description of the activities recognised.
Animal Sensors Activities Citation
Elephant seals
Mirounga leonina Time-depth, 3D acc.
Depth, movement of body, time immersion mouth opening.
[66], [80]
Weddell seals
Leptonychotes weddellii Time-depth Depth, time immersion. [64]
Antarctic fur seals
(Arctocephalus gazella) Time-depth Foraging activity. [82] Hawksbill turtles
(Eretmochelys imbricata) Time-depth Depth, time immersion. [69] Hermanni turtles
(Testudo hermanni) 2D acc. Digging. [5] Little Penguins
(Eudyptula minor ) 3D acc. Prey capture activity. [90] Adelie Penguins
(Pygoscelis adeliae) 3D acc.
Walk, toboggan, stand on land, lay on land and rest.
[78]
Cape gannets (Morus capensis)
Time-depth,3D acc., tilt sensor
Take-off, flapp, glid flight,
plunge dive, and land. [79] Red Foxes
(Vulpes vulpes) 3D acc.,magnetometer
Hunting movement,
magnetic alignment. [91] Leopards
(Panthera pardus) 3D acc., gyroscope Energy consumed. [92] Domestic dogs
(Canis familiaris) 3D acc.
Walk, run, sit,
lie-down, and stand. [87] Green Turtles
(Chelonia mydas) Time-depth Depth, Time immersion. [93] Rockfish
(Scorpaena onaria) Ultrasonic transmitter Diving activity [94]
Inspired by the application of devices to study animals of the underwater world, researchers applied devices also for the animals on land. The usefulness of collecting information via bio-logging on land is evident when we consider predators. For example, pumas (Puma concolor), which are predators difficult to study due to their speed. The first study of pumas with devices is performed in California with collars equipped with accelerometers [92]. By this device, it is possible to monitor movements of pumas during the hunt and, by these information, to infer an estimate of energy expenditures. In [91], the hunting behaviour studied is from semi-domestic red foxes (Vulpes vulpes) equipped with collars. The collar contains a device with accelerometers and magnetometers that are retrieved at the end of the experiment. Domestic dogs became subject of study as a tame surrogate instead of predators. In [95], authors present an exploratory study to record daily activities of dogs, using collars with embedded
2.3. Machine learning for animal activity recognition 21
accelerometers. In general AAR can be applied in many field, for example, to identify when a tortoise is digging a nest [5], or when a fish opens and closes its mouth [89]. Table 2.1 presents a summary of some of the case of studies highlighted here but does not intend to be an exhaustive illustration of all the types of information that can be obtained inAARstudies. Activities depend on the species studies and the aim of the research. Accelerometers, magnetometers, and gyroscopes, allowed researchers to study animals’ body movements at a fine scale.
In all these studies, data are collected via bio-logging or biotelemetry procedures. The bio-logging and biotelemetry procedures are widely applied to obtain information not directly available by observation on field, but by applying bio-logging procedure here means that the animal is recaptured after the collecting phase to download data and by applying biotelemetry procedure means that it is necessary for the collecting phase that the researchers are in the connection range with the animal. The first is stressful for the animal and the second could be no possible for the researchers.As such, recapture of the animal is imperative which means that the animal must be limited to a certain space or easy to find. In addition, most analyses are done once the logger has been retrieved. To address this latter limitation, one direction pursued by developers is to provide a standard and autonomous behavioural annotation system able to recognise activity from data in automatic way, [96], [87].
Furthermore, by combining technology with field observations, a vast magnitude of behavioural data can be collected. We take into consideration the Machine Learning (ML) approaches to analyse these data. The machine learning models can be used to autonomously identify a pattern of animal activity within streams of accelerometer data. This pattern can be learned by models through a training phase exploiting samples of classified data gathered in the field, allowing us to adapt the model to different tasks and contexts, and to make it robust enough to cope with noisy data. It is worth noting that by ML approaches the identification of the activities is completely autonomous, and in some cases it is possible to customise the ML approach in order to obtain a light software which can be upload on board. These two aspects cope with the retrieve of the device and the analysis of data providing a solution for both of them.
As proof of concept, we use machine learning approaches to analyse data in a set of different applications. The first one is the Tortoise@ project. Tortoise@ is an autonomous system for large scale applications aimed at identifying tortoise nests and rescuing eggs. It is a system based on bio-logged data preliminarily presented in [97] and again later in [5]. The software enables an autonomous and assistive observation of tortoise digging behaviour. The data are collected in Toscana at the ”Centro di protezione delle tartarughe mediterranee” (CTM,
Protection centre for Mediterranean tortoises) in Massa Marittima. The data acquired by the accelerometer sensor are used to infer behaviour of the nest digging activity through machine learning techniques and to transmit the location of the nest to a monitoring center - if and only if the activity has been identified as a digging activity. The entire system is designed to be contained within the device attached to the carapace. This means that it is possible to monitor the nesting activity of a large number of tortoises without the necessity of capturing the animals, thus reducing human interference. Other cases of our research were focused on comparing different machine learning techniques being applied to different data from wild animals. In 2016, we collaborated with Macquarie University (Sydney, Australia) to study the accuracy of different machine learning techniques in identifying the prey capture activity in penguins and seals, then quantifying food intake remotely. An initial analysis of the categorization of the behaviour of captive penguins was presented in [84] from HD video and data from back-mounted accelerometers. Like for the tortoises, we examined the possibility of obtaining competitive results with systems that could be used on board, letting us obtain real time identification of the activities. A similar analysis could be presented for the seals’ dataset. The results obtained for the Tortoise@ system, as well as for the penguins and seals datasets, outline the capabilities of the combination of technology with biological fields. This collaboration enabled bio-logging and biotelemetry to evolve with the understanding of animals’ and environmental conservation necessities.
The inspiration for these approaches came from theHAR, which growing up faster due to the request of automatic tools for data analysis in medical area as described in Section2.2. WhileAARtakes a great influence fromHAR, it is worth noting that activity recognition in humans is deeply different from activity recognition in animals. This difference is in three main aspects: (i) the sensors used to record movement information may be the same (accelerometers, and magnetometers) but the device and the location on body could be different both for physiological differences (for example the lack of paws for fishes or the lack of wings for humans); (ii) the sensors themselves may be different (for example, depth sensors are meaningful for fishes and marine mammals but usually not for humans); (iii) the observed activities may be completely different. Furthermore, the data collecting procedure may be completely different for humans and animal, because do not follow a pre-defined schema of activities or a plan of movements. The collection campaign with animals is more complex to perform due to lack of volunteers and to the danger of the environment. The result is that the analysis of data of animal activities represents a new challenge for activity recognition due to the heterogeneity of datasets and to a variable quality of data.
2.3. Machine learning for animal activity recognition 23
In [88] data are analysed for diving behaviour in elephant seals including stroking rates and three-dimensional movements. Data are analysed observing the signal stream to identify spikes that make evident the rotation movement used for hunting by the elephant seal. This procedure is one of the first approaches to the analysis of bio-logging data and it needs a long time working. In last decades, the observation of data stream become obsolete and it is substituted by the signal analysis. In [80], authors present an example of signal analysis of behavioural data in seals. Devices include accelerometer and depth sensors, and each device is head-mounted to the seal. An initial set of data is used to link sensors’ data profiles (exemplar trend of the signal) to each activity. The accelerometer and depth data are compared with dive depth profiles and accelerometer profiles. In [78] depth and accelerometer profiles are used to classify Adelie penguins’ activities. In all these studies the profiles are calibrated in the aquarium on a restricted set of samples. This can lead to an over fitting behaviour on samples data, a strong sensibility to signals used as profiles. Although the classification results were good on data from wild animals. However the exemplar profiles are in bio-logging more convenient than direct observation. Statistics analysis is a method commonly used through tools as R [98]. Statistics provide summarized data about individual behaviour of animals and are used to write customized routine. An example of this technique is presented in [82] for fur seals. A custom written routine in R inspired by [89] is used to classify bio-logging data. Recently the same technique has been used for pumas, [92]. These methods hereafter are referred as traditional methods. Modern data analysis adopts methods based onML to study complex interaction between biotic and abiotic systems. ML can outperform the traditional methods of analysis in classification task as reported in [99]. In the following section we analyse the advantage of the machine learning in bio-logging.
MLmethods are used to analyse the data recorded by low power device. The robustness and the ability for universal approximation of these methods (e.g. Neural Networks) provide the basis for the flexibility of the approach for pattern recognition. In particular, they allow the approximation of arbitrary classification functions from experimental data, despite not having a theory of the pattern characteristics. This is particularly advantageous for non-linear classification/pattern recognition tasks, which can be difficult to be addressed by the traditional approaches. AMLclassifier is inferred by data: it is able to classify new data after being properly trained with known instances. In this context, ML models show high classification capabilities. In [91], k-Nearest Neighbours (K-NN) model is applied to identify magnetic alignment responses during the hunting event in red foxes. In this case the training dataset consists of the k closest examples used as seeds to classify future data streams. A data stream is classified by a majority behaviour referred by its neighbours’
seeds. In particular, the activity identified of a current data stream is assigned considering the most common activity identified by its k nearest neighbours. This model is applied in [100] and in [101], where theMLmodels are proposed to classify the daily activity of several species. On such application, the K-NN provides good performance over the dataset. It is worth noting that this model is tailored on dataset at hand, which reduces its usability when the application is in the wild. To address this problem, the main focus changes from the K-NN to SVM. TheSVM uses a set of seeds,which are the support vectors used to classify each data stream and the SVMbenefits of error tolerance due to its architecture. The SVM is applied to identify the prey capture event in little penguins(Eudyptula minor ) as explained in [84]. It is trained over a sampled dataset recorded in Taroonga Zoo during a prey capture activity by penguins. The tunedSVMwill be able to identify prey capture activities in wildlife of penguin to help and support them against the lack of food. In [95] the SVMis applied to identify the activities of domestic dogs. It distinguishes between five different activities and classifies each activity individually. In both this cases, data are analysed after the recording phase. To analyse data in real-time is a more interesting challenge that can be a changing view. But to this aim the model needs to be embedded on a low power device. Both K-NN andSVM need to maintain information about the seeds because this information is necessary to classify the data streams but requires a certain amount of memory space. For this reason different MLtechniques are recently considered for the classification of data. This is the main aim addressed with the feasibility analysis in [5]. Authors present ML for classification of nest digging activity in tortoises (Hermanni hermanni). The device is equipped with accelerometer, light, and temperature sensors used to identify the activity of the tortoises and the surrounding environment. The ML proposed is a customisedIDNN. This model provides a solution that is a trade-off between the generality of the ML model in classification and memory space needed. The result obtained is a model with high performance that can be embeddable on a device. The possibility of installing the system on the device is a big improvement for behavioural analysis. In this way devices on animals do not have to be retrieved after the recording season.
We can identify two main approaches by analysing the different approaches proposed in the papers taken in consideration both based on data collected via bio-logging procedure. The first main approach is the log of data on the device without any pre-processing on the daa. In this case, to analyse data it is necessary the retrieval of the device and the statistical analysis on a remote base. The second main approach is by using a ML approach on board. In this case it is not necessary to retrieve the device to be able to analyse data. Data are analysed on board by the ML approach and only the information of classification is stored. If on one hand the statistical analysis provides good
2.3. Machine learning for animal activity recognition 25
results over data samples, on the other hand the solutions reached are difficult to apply in wild habitats. Instead,ML-models are intrinsically more flexible and can be used to deal with nonlinear relation between data and classification. The machine learning provides methodologies to build a model directly from data. It is worth noting that not all the bio-logging problems can be resolved with both traditional methods and MLmodels with the same way and that MLmodels in many cases provide better results than the traditional methods for classification and identification of activities.
3
Background
The primary objective of Machine Learning models for activity recognition is to learn discriminating knowledge from data in order to build a classifier for the task at hand. We consider the supervised learning models; the “supervised” specifies the presence of the outcome information to guide the learning process. In this section we introduce state-of-the-art of the models taken in consideration for this thesis. In particular in the following we introduce models from the Neural Networks learning class: theIDNN, theCNN, and, in particular in the Recurrent Neural Network class, theESN. We consider also theSVM.
3.1
Machine Learning
In this thesis, we distinguish between two different classes of task: asynchronous, and synchronous tasks. This distinction is based on the output of the automatic classifier and how it is provided in the case study at hand. With the term asynchronous we indicate that the automatic classifier provides an outcome on demand, the output is provided only in a certain moment or for certain conditions. With the term synchronous, we indicate an outcome provided continuously (in real time). The outcome of the automatic classifier is the identified class of the input, in this case the term class is referred to the activity performed by the subject. Dealing with these two tasks we take into consideration two different type of inputs: sequences, and streams. The sequences is extracted by the time-series. The dimension of the sequence is chosen taking into consideration the activity performed, and the subject. Each sequence is labeled individually then each sequence concern an activity. Each stream is the time-series data recorded on real-time by the sensors each time step.
The case studies are been characterised by the task required. The input type affected the development of the more suited model. In particular, such as the IDNN, theCNN, and the SVMare naturally thought to deal with sequences,
