by deep learning

(1)

Image segmentation:

from the Mumford-Shah conjecture to more global variational formulations

by deep learning

A tribute to Gianni Dal Maso

(2)

(3)

Plan:

• The Mumford-Shah conjecture and our (small) contribution

• The three ways to segment an image according to Mumford-Shah: some

algorithms

• A real image segmentation problem: manipulating cardboxes

• Another one : crack detection in walls

(4)

How I woke up every morning in

this room

(5)

Optimal Approximations by Piecewise Smooth Functions and Associated Variational Problems’ DAVID MUMFORD

Harvard University AND

JAYANT SHAH

(6)

(7)

(8)

(9)

LEMENANT, Antoine. A selective review on Mumford–Shah minimizers. Bollettino dell'Unione Matematica Italiana, 2016, vol. 9, no 1, p. 69-113.

(10)

(11)

(12)

(13)

(14)

(15)

“Short list” of authors who dedicated time to the Mumford-Shah conjecture:

Giovanni Alberti, Guy Bouchitté, Gianni Dal Maso, Luigi Ambrosio, Nicola Fusco, John E. Hutchinson, Diego Pallara, Vincenzo Maria Tortorelli, Giovanni Bellettini, Alessandra Coscia, Blaise Bourdin, Antonin

Chambolle, Jean-François Babadjian, Antoine Lemenant, Alexis Bonnet, Guy David, Dorin Bucur, Stephan Luckhaus, Gilles A. Francfort, Jean-Jacques Marigo, Ennio De Giorgi, Michele Carriero, Antonio Leaci, Françoise Dibos, Georges Koepfler, Jean-Christophe Léger, Camillo De Lellis, Matteo Focardi, Berardo Ruffini, Rodica Toader, Sergio Solimini, Guido De Philippis, Alessio Figalli, Thierry De Pauw, Didier Smets, Stephen Semmes, Nicola Fusco Matteo Focardi, Berardo Ruffini, Benoît Merlet, Maria Giovanna Mora, Massimiliano Morini, Séverine Rigot, Anthony Siaudeau.

(16)

What was done to tackle (define, solve) the “real” image segmentation

problem?

The three ways to segment an image according

to Mumford-Shah: some algorithms

(17)

Antoine Lemenant. A selective review on Mumford-Shah minimizers. Bollettino dell’Unione Matematica Italiana, Springer Verlag, 2016, 9, pp.69 - 113.

(18)

VON GIOI, Rafael Grompone et RANDALL, Gregory. A Sub-Pixel Edge Detector: an Implementation of the Canny/Devernay

Algorithm. IPOL Journal, 2017, vol. 7, p. 347-372.

(19)

VON GIOI, Rafael Grompone et RANDALL, Gregory. A Sub-Pixel Edge Detector: an Implementation of the Canny/Devernay Algorithm. IPOL Journal, 2017, vol. 7, p. 347-372.

(20)

Von Gioi, R. G., Jakubowicz, J., M., J. M., & Randall, G. (2012): LSD: a line segment detector. Image Processing On Line, 2, 35-55.

(21)

Von Gioi, R. G, Jakubowicz, J., M., J. M., & Randall, G. (2012). LSD: a line segment detector. Image Processing On Line, 2, 35-55.

(22)

LSD: a Line Segment Detector

Piecewise Affine Image Segmentation Based on Mumford-Shah Functional (4 regions) G. Facciolo Algorithm: Koepfler, G., Lopez, C., & Morel, J. M. (1994). A multiscale algorithm for image segmentation by variational method. SIAM journal on numerical analysis, 31(1), 282-299.

(23)

Piecewise Affine Image Segmentation Based on Mumford-Shah Functional (4 regions) G. Facciolo

Algorithm: Koepfler, G., Lopez, C., & Morel, J. M. (1994). A multiscale algorithm for image segmentation by variational method. SIAM journal on numerical analysis, 31(1), 282-299.

(24)

Piecewise Affine Image Segmentation Based on Mumford-Shah Functional (4 regions) G. Facciolo

Algorithm: Koepfler, G., Lopez, C., & Morel, J. M. (1994). A multiscale algorithm for image segmentation by variational method. SIAM journal on numerical analysis, 31(1), 282-299.

(25)

A jump in time: A real segmentation problem in

2020: Cardbox detection

(26)

Cardbox detection

Due to labor shortage, warehouses in Japan need to automate their

processes.

(27)

Cardbox detection

Due to labor shortage, warehouses in Japan need to automate their

processes.

(28)

Cardbox detection

Due to labor shortage, warehouses in Japan need to automate their

processes.

Not pre-arranged or

ordered and can be

inclined

(29)

Cardbox detection

(30)

Cardbox detection

(31)

Cardbox detection

(32)

Cardbox detection

(33)

(34)

(35)

Cardbox detection

(36)

Cardbox detection

(37)

Cardbox detection

(38)

Cardbox detection

(39)

Cardbox detection

(40)

Cardbox detection

→ Deep learning

Main idea:

●

Use a first segmentation obtained by 3D

(41)

Cardbox detection

(42)

Cardbox detection

(43)

Cardbox detection

(44)

Cardbox detection

(45)

Cardbox detection

One « semantic boundary » is missing

(46)

Cardbox detection

First approach:

Border detection

U-Net

(47)

(48)

Lu, Z., Pu, H., Wang, F., Hu, Z., & Wang, L. (2017). The expressive power of neural networks: A view from the width.

Advances in neural information processing systems (pp. 6231-6239).

« All » functionals are by now approximable by a neural network, provided

enough examples are given. The only left variational problem is the minimization

of the loss functional of the neural network. This loss has two terms: the fidelity

term and a regularizing term (typically L2 norm of the coefficients)

(49)

Cardbox detection

First approach:

Border detection

U-Net

(50)

(51)

(52)

(53)

(54)

(55)

Cardbox detection

→ Detection algorithm should be as robust as possible to changes in scale,

rotation, contrast, illumination, shadows…

Making the learning data:

●

Acquiring and labeling as many images as possible:

○

~ 2000 images

○

Limitations: diversity!

●

_{Choosing adequate data augmentation.}

●

Choosing adequate loss function / neural network structure.

●

Adding simulation images for increasing diversity.

(56)

Cardbox detection

Unity3D

Texture of cardboxes taken from our existing dataset (+ add changes in the

future).

Random texture in background.

(57)

Cardbox detection

(58)

Cardbox detection

(59)

Cardbox detection

(60)

Cardbox detection

(61)

Cardbox detection

Results on new

environment.

(62)

Cardbox detection

Results on new

environment.

(63)

Cardbox detection

Results on new

environment.

(64)

Crack Detection

(65)

An "all terrain" crack detector obtained by deep learning on available databases

(66)

(67)

(68)

(69)

(70)

(71)

(72)

(73)

S. Drouyer An 'All Terrain' Crack Detector Obtained by Deep Learning on Available Databases IPOL 2019 http://www.ipol.im/pub/pre/282/

(74)

S. Drouyer An 'All Terrain' Crack Detector Obtained by Deep Learning on Available Databases IPOL 2019 http://www.ipol.im/pub/pre/282/

(75)

Discussion: Bayesian

framework? Calculus of

Variations?

(76)

(77)

Discussion

(78)

Discussion

The « general » functional often has a Bayesian interpretation

Example: denoising

(79)

A. Levin, B. Nadler. CVPR 2011. Natural image denoising: Optimality and inherent bounds Zoran, D., & Weiss, Y. ICCV 2011. From learning models of natural image patches to whole image restoration.

4-The Bayesian denoising paradigm from « non-local » to « global »

The Levin and Nadler optimal « global denoising algorithm » uses « all patches of the world »

(80)

An Analysis and Implementation of the FFDNet Image Denoising Method

(81)

(82)

(83)

(84)