Deep Reservoir Computing: A Novel Class of Deep Recurrent Neural Networks

(1)

Title:

_{Deep Reservoir Computing: A Novel Class of Deep Recurrent Neural Networks}

Ph.D. candidate:

_{Luca Pedrelli}

Supervisors:

_{Alessio Micheli, Claudio Gallicchio}

Internal commission:

_{Salvatore Ruggieri, Stefano Chessa}

International reviewers:

_{Igor Farkaš, Peter Tiño}

My Ph.D. thesis is focused on the analysis and the development of deep Recurrent Neural

Networks (RNNs) for time-series processing. In particular, we introduced the Deep Echo State

Network (DeepESN) models to study the intrinsic role of layering in deep recurrent architectures

and to provide, at the same time, efficient tools to address real-world tasks composed by time

series characterized by multiple time-scales dynamics. Through empirical analysis performed on

DeepESN models we highlighted the intrinsic ability of deep RNNs to develop a multiple

time-scales representation in the layers dynamics. Then, we introduced the a DeepESN with

linear activations (L-DeepESN) to analyze the spectral properties of the state developed by

layers starting from a signal composed by superimposed oscillators. The analysis highlighted

that the layering in recurrent architecture is intrinsically able to develop hierarchical and

distributed temporal features.

Finally, we developed an efficient design tool based on the DeepESN architecture able to

choose the number of recurrent layers in deep RNNs through quantitative and qualitative

measures.

We applied the developed approaches to address real-world tasks in the field of Health

Informatics and prediction of multivariate time-series. Moreover, we thoroughly compared

DeepESN and ESN models with the state-of-the-art RNNs. Overall, DeepESN outperformed the

others approaches in terms of efficiency and compared well with the state-of-the-art results on

challenging real-world tasks.

The list of journal papers and proceedings published within my thesis work is reported below.

Journal Papers

[1] C. Gallicchio, A. Micheli, L. Pedrelli, "Design of deep echo state networks", Neural networks: the official journal of the International Neural Network Society 108 (2018), vol. 108, pp. 33-47, DOI:10.1016/j.neunet.2018.08.002, ISSN: 0893-6080

Neural Networks - Journal, Impact Factor: 7.197

[2] C. Gallicchio, A. Micheli, L. Pedrelli, "Deep Reservoir Computing: A Critical Experimental

Analysis", Neurocomputing (2017), vol. 268, pp. 87-99, DOI:

10.1016/j.neucom.2016.12.089, ISSN: 0925-2312 Neurocomputing - Journal, Impact Factor: 3.241

[3] D. Bacciu, S. Chessa, C. Gallicchio, A. Micheli, L. Pedrelli, E. Ferro, L. Fortunati, D. La Rosa, F. Palumbo, F. Vozzi, O. Parodi. "A learning system for automatic Berg Balance Scale score estimation", Engineering Applications of Artificial Intelligence (2017), vol. 66 pp. 60-74, DOI:10.1016/j.engappai.2017.08.018, ISSN 0952-1976

(2)

Proceedings

[4] C. Gallicchio, A. Micheli, and L. Pedrelli. "Comparison between DeepESNs and gated RNNs on multivariate time-series prediction”. In: Proceedings of the 27th European Symposium on Artificial Neural Networks (ESANN), Bruges, Belgium, 24-26 April 2019, (accepted)

[5] C. Gallicchio, A. Micheli, and L. Pedrelli. "Fast Spectral Radius Initialization for Recurrent Neural Networks”. In: Proceedings of the INNS Big Data and Deep Learning (INNSBDDL), Sestri Levante, Genoa, Italy, 16-18 April 2019, (accepted)

[6] C. Gallicchio, A. Micheli, L. Pedrelli, "Deep Echo State Networks for Diagnosis of Parkinson's Disease" (2018), Proceedings of the 26th European Symposium on Artificial Neural Networks (ESANN), Bruges, Belgium, 25-27 April 2018, i6doc.com, pp. 397-402, ISBN: 978-2875870476

[7] C. Gallicchio, A. Micheli, L. Pedrelli, “Hierarchical Temporal Representation in Linear Reservoir Computing”, in: A. Esposito, M. Faundez-Zanuy, F. C. Morabito, E. Pasero (Eds.), Neural Advances in Processing Nonlinear Dynamic Signals, Springer International Publishing, Cham,2019, pp. 119–129, WIRN 2017, arXiv preprint arXiv:1705.05782. doi:10.1007/978-3-319-95098-3_11

[8] C. Gallicchio, A. Micheli, L. Pedrelli, L. Fortunati, F. Vozzi, and O. Parodi. "A reservoir computing approach for balance assessment" (2016), In A. Douzal-Chouakria, J.A. Vilar, and P.-F. Marteau, editors,Advanced Analysis and Learning on Temporal Data: First ECML PKDD Workshop, AALTD 2015, Porto,Portugal, September 11, 2015, Revised Selected Papers, volume 9785 of Lecture Notes in Computer Science, pages 65–77.

Springer International Publishing, DOI: 10.1007/978-3-319-44412-3_5, ISBN:

978-3-319-44412-3

[9] C. Gallicchio, A. Micheli, L. Pedrelli, F. Vozzi, O. Parodi, "Preliminary Experimental Analysis of Reservoir Computing Approach for Balance Assessment" (2015), Proceedings of the 1st International Workshop on Advanced Analytics and Learning on Temporal Data (AALTD), co-located with The European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases(ECML PKDD), Porto, Portugal, 11 September 2015, A. Douzal-Chouakria, J.A. Vilar Fernandez, P.-F.Marteau, A.E. Maharaj, A.M. Alonso Fernandez, E. Otranto, M.-I. Nicolae Editors, CEUR Workshop Proceedings, pp. 57-62

Deep Reservoir Computing: A Novel Class of Deep Recurrent Neural Networks

Title: