• Non ci sono risultati.

Open-Source Frameworks for Deep Learning: an Overview Vincenzo Lomonaco

N/A
N/A
Info
Scarica
Protected

Academic year: 2021

Condividi "Open-Source Frameworks for Deep Learning: an Overview Vincenzo Lomonaco"

Copied!
37
0
0

Caricamento.... (Visualizza il testo completo ora)

Scarica adesso ( 37 pagine )

Testo completo

(1)

Open-Source Frameworks for Deep Learning: an Overview

Vincenzo Lomonaco

University of Bologna ML Course - 2020

Post-Doc @ University of Bologna

President & Co-Founder of ContinualAI.org vincenzolomonaco.com

(2)

About Me

• Post-Doc @ University of Bologna

• Visiting Scholar @ ENSTA ParisTech, Purdue University and Numenta

• Teaching Assistant of the courses Machine Learning and Computer Architectures

• Author and Technical reviewer of the online course Deep Learning with R and book R Deep Learning Essentials

• Co-Founder & President of ContinualAI

• Co-Founder & Board Member of AIforPeople

https://www.vincenzolomonaco.com

(3)

Outline

1. Introduction

a. From Machine Learning to Deep Learning b. History of Deep Learning Frameworks

2. Open-Source Deep Learning Frameworks a. Main Features and Distinctions

b. Pytorch and Tensorflow 3. Practical Advices

4. What next?

(4)

What’s ContinualAI?

● ContinualAI is a non-profit research organization and the largest research community on Continual Learning for AI.

● It counts more than 1000+ members in 19 different time-zones and from top-notch research institutions.

● Learn more about ContinualAI at www.continualai.org

(5)

The Machine Learning Software Stack

“Hidden Technical Debt in Machine Learning Systems” by D. Sculley et al.

(6)

The Machine Learning Software Stack

For Prototyping and R&D

● Scikit-learn (Python), Caret (R), Weka (Java)

For Production

● Kubernetes, KubeFlow, Hadoop

● H2O, MLlib, SparkML, FlinkML

Setting up a Machine Learning Framework for Production: http://code.hootsuite.com/setting-up-a-machine-learning-framework-for-production/

(7)

From Machine Learning to Deep Learning

For Deep Learning the line between Prototyping and Production is more blurred

● Caffe / Caffe 2

● Torch / PyTorch

● Tensorflow

● CNTK, MxNet, Gluon, Chainer, etc.

(8)

From Machine Learning to Deep Learning

Why

● Prototyping with neural nets on massive datasets needs efficiency

● Limited focus on:

1. (Deep) Neural Networks.

2. Gradient-Based Optimization.

3. Transparent (multi) CPU/GPU acceleration

(9)

History of Deep Learning Frameworks

2010 2013 2014 2015 2016 2017 2018

...

Comparison of deep-learning software: https://en.wikipedia.org/wiki/Comparison_of_deep-learning_software

(10)

History of Deep Learning Frameworks

(11)

History of Deep Learning Frameworks

Andrej Karpathy, 10 mar 2018: https://twitter.com/karpathy/status/972295865187512320/

(12)

Deep Learning Frameworks: 2018 Figures

Deep Learning Framework Power Scores 2018: https://towardsdatascience.com/deep-learning-framework-power-scores-2018-23607ddf297a/

(13)

Deep Learning Frameworks: 2018 Figures

Deep Learning Framework Power Scores 2018: https://towardsdatascience.com/deep-learning-framework-power-scores-2018-23607ddf297a/

(14)

Is PyTorch Catching TensorFlow? 2020 Update

Is PyTorch Catching TensorFlow? https://towardsdatascience.com/is-pytorch-catching-tensorflow-ca88f9128304

(15)

Is PyTorch Catching TensorFlow? 2020 Update

Is PyTorch Catching TensorFlow? https://towardsdatascience.com/is-pytorch-catching-tensorflow-ca88f9128304

(16)

Outline

1. Introduction

a. From Machine Learning to Deep Learning b. History of Deep Learning Frameworks

2. Open-Source Deep Learning Frameworks a. Main Features and Distinctions

b. Tensorflow and Pytorch 3. Practical Advices

4. What next?

(17)

Main Features and Distinctions

1. Static vs Dynamic Graph 2. Differentiation Support

3. Hardware / Platform Support

4. Model / Utils Support and Performance 5. Multi-CPU/GPU support

6. API level

7. License and Community Support

Comparison of deep learning software: https://en.wikipedia.org/wiki/Comparison_of_deep_learning_software

(18)

Static vs Dynamic Graph

Static (define-AND-run) Dynamic (define-BY-run)

(19)

Differentiation Support

Types of Differentiation

1. Numerical Differentiation 2. Symbolic Differentiation 3. Automatic Differentiation 4. Hard-coded Differentiation

“Automatic Differentiation in Machine Learning: a Survey”: http://www.jmlr.org/papers/volume18/17-468/17-468.pdf

(20)

Hardware / Platform Support

Platform Support

● Windows, Unix (MAC OS, Linux)

● Android OS, iOS, Embedded Systems

Hardware Acceleration

● Cuda / OpenCL Support

● OpenMP, MPI Support

● AI Chips

(21)

Model / Utils Support and Performance

Types of models supported

● FC-NNs, CNNs, RNNs, etc.

● General purpose algebraic functions

Utils support

● Pre-trained models Zoo

● Data format and loading

● Monitoring / Visualization tools

Performance

● Very different from task to task and model to model, etc.

(22)

Multi-CPU / GPU support

Types of parallelization

● Data Parallelism

● Model Parallelism

Hardware for parallelization

● CPUs

● GPUs

● Distributed clusters

“Large Scale Distributed Deep Networks”: http://papers.nips.cc/paper/4687-large-scale-distributed-deep-networks.pdf

(23)

API Level

Important features

● Usability

● Flexibility

● Expandability

● Easy to Debug

(24)

License and Community Support

Licence

● Apache 2.0, MIT, BSD, GNU GPL, Freemium, etc.

● Most of them allow also commercial use (but look at patent licence too)

Community Support

● Contribution Diversity

● Active Development

● Supportive Q&A

● Number of users

(25)

Tensorflow and Pytorch

● Static/Dynamic Graph

● Automatic Differentiation

● Windows, Unix, Embedded

● General purpose graph

● Multi-CPU/GPUs

● C++ and (many) Python API

● Apache 2.0

● High community support

● Dynamic Graph

● Automatic Differentiation

● Windows, Unix, Embedded

● General purpose graph

● Multi-CPU/GPUs

● C++ and Python API

● BSD Licence

● High community support

(26)

Tensorflow: Deep learning framework by Google

“Tensorflow: a system for large-scale machine learning.”: https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf

(27)

Tensorflow: Deep learning framework by

Google

(28)

Tensorflow: Deep learning framework by Google

Key features

● Great models / utils support

● Large API variants at different levels

● Solid codebase

● Amazing community support and permissive Apache 2.0 licence

Tensorflow course: https://ekababisong.org/gcp-ml-seminar/tensorflow/

(29)

Pytorch: Deep learning framework by Facebook

“Automatic differentiation in PyTorch”: https://openreview.net/pdf?id=BJJsrmfCZ

(30)

Pytorch: Deep learning framework by

Facebook

(31)

Pytorch: Deep learning framework by Facebook

Key features

● Great models / utils support

● Nice integration with ONNX and Caffe2

● Great for highly-dynamic graphs

● Simple, neat API

● Easy to learn

Python API C++ API

(32)

[Hands-on (30 minutes)] Training a

ConvNet with Pytorch

(33)

Outline

1. Introduction

a. From Machine Learning to Deep Learning b. History of Deep Learning Frameworks

2. Open-Source Deep Learning Frameworks a. Main Features and Distinctions

b. Tensorflow and Pytorch 3. Practical Advices

4. What next?

(34)

Practical Advices

(35)

Practical Advices

● Your don’t need to choose one framework for life

● Be flexible, understand the core and common ideas

● On-the-run conversion may be painful!

● Choose the best depending on the task at hand

and the already existing resources

(36)

What Next?

(37)

Questions?

Vincenzo Lomonaco

University of Bologna ML Course - 2020

Post-Doc @ University of Bologna

President & Co-Founder of ContinualAI.org vincenzolomonaco.com

Riferimenti

Scarica adesso ( PDF - 37 pagine - 3.15 MB )

Documenti correlati

Index of /files/CoursesDev/public/2018/Deep Learning

[r]

Deep Learning Libraries

 Keras is a high-level neural networks API, written in Python and capable of running on top of. TensorFlow, CNTK,

Deep Learning

The rest of the thesis is organized as follows: in chapter 2, we briefly describe the most important elements of CNNs, well-known architectures and also the training process of

Introduction to Deep Learning

– While standard NN nodes take input from all nodes in the previous layer, CNNs enforce that a node receives only a small set of features which are spatially or temporally close

Deep Learning in Matlab

Introducing Deep Learning with MATLAB.. • https://it.mathworks.com/videos/transfer-

Deep Learning: Biometric Applications

Scotti, A novel pore extraction method for heterogeneous fingerprint images using Convolutional Neural Networks, Pattern Recognition Letters,

Deep Learning in Biometrics

– In current models layers often learn in an unsupervised mode and discover general features of the input space – serving multiple tasks related to the unsupervised instances (image

Deep Learning for NL

 Both are 2 layer neural networks that learn to model their inputs. 