Definition of the models
Listing C.1: Definition of the VGG16 Teacher model.
c l a s s VGG1 6( nn . Module ) : d e f i n i t ( s e l f ) :
s u p e r (VGG1 6, s e l f ) . i n i t ( )
s e l f . f e a t u r e s0 = nn . Conv2d ( i n c h a n n e l s=3, o u t c h a n n e l s=6 4, k e r n e l s i z e=3, p a d d i n g=1, b i a s=F a l s e )
5 s e l f . f e a t u r e s2 = nn . Conv2d ( i n c h a n n e l s=6 4, o u t c h a n n e l s=6 4, k e r n e l s i z e=3, p a d d i n g=1, b i a s=F a l s e )
s e l f . f e a t u r e s5 = nn . Conv2d ( i n c h a n n e l s=6 4, o u t c h a n n e l s=1 2 8, k e r n e l s i z e=3, p a d d i n g=1, b i a s=F a l s e )
s e l f . f e a t u r e s7 = nn . Conv2d ( i n c h a n n e l s=1 2 8, o u t c h a n n e l s=1 2 8, k e r n e l s i z e=3, p a d d i n g=1, b i a s=F a l s e )
10 s e l f . f e a t u r e s1 0 = nn . Conv2d ( i n c h a n n e l s=1 2 8, o u t c h a n n e l s=2 5 6, k e r n e l s i z e=3, p a d d i n g=1, b i a s=F a l s e )
s e l f . f e a t u r e s1 2 = nn . Conv2d ( i n c h a n n e l s=2 5 6, o u t c h a n n e l s=2 5 6, k e r n e l s i z e=3, p a d d i n g=1, b i a s=F a l s e )
s e l f . f e a t u r e s1 4 = nn . Conv2d ( i n c h a n n e l s=2 5 6, o u t c h a n n e l s=2 5 6, k e r n e l s i z e=3, p a d d i n g=1, b i a s=F a l s e )
s e l f . f e a t u r e s1 7 = nn . Conv2d ( i n c h a n n e l s=2 5 6, o u t c h a n n e l s=5 1 2, k e r n e l s i z e=3, p a d d i n g=1, b i a s=F a l s e )
15 s e l f . f e a t u r e s1 9 = nn . Conv2d ( i n c h a n n e l s=5 1 2, o u t c h a n n e l s=5 1 2, k e r n e l s i z e=3, p a d d i n g=1, b i a s=F a l s e )
s e l f . f e a t u r e s2 1= nn . Conv2d ( i n c h a n n e l s=5 1 2, o u t c h a n n e l s=5 1 2, k e r n e l s i z e=3, p a d d i n g=1, b i a s=F a l s e )
s e l f . f e a t u r e s2 4 = nn . Conv2d ( i n c h a n n e l s=5 1 2, o u t c h a n n e l s=5 1 2, k e r n e l s i z e=3, p a d d i n g=1, b i a s=F a l s e )
s e l f . f e a t u r e s2 6 = nn . Conv2d ( i n c h a n n e l s=5 1 2, o u t c h a n n e l s=5 1 2, k e r n e l s i z e=3, p a d d i n g=1, b i a s=F a l s e )
20 s e l f . f e a t u r e s2 8 = nn . Conv2d ( i n c h a n n e l s=5 1 2, o u t c h a n n e l s=5 1 2, k e r n e l s i z e=3, p a d d i n g=1, b i a s=F a l s e )
s e l f . maxpool = nn . MaxPool2d ( k e r n e l s i z e=2, s t r i d e=2) s e l f . a v g p o o l = nn . AdaptiveAvgPool2d ( (7, 7) )
25 s e l f . c l a s s i f i e r0 = nn . L i n e a r (2 5 0 8 8, 4 0 9 6, b i a s=F a l s e ) s e l f . c l a s s i f i e r3 = nn . L i n e a r (4 0 9 6, 4 0 9 6, b i a s=F a l s e ) s e l f . c l a s s i f i e r6 = nn . L i n e a r (4 0 9 6, 1 0, b i a s=F a l s e )
d e f f o r w a r d ( s e l f , x ) :
30 x = F . r e l u ( s e l f . f e a t u r e s0( x ) ) x = F . r e l u ( s e l f . f e a t u r e s2( x ) ) x = s e l f . maxpool ( x )
x = F . r e l u ( s e l f . f e a t u r e s5( x ) ) x = F . r e l u ( s e l f . f e a t u r e s7( x ) )
35 x = s e l f . maxpool ( x )
x = F . r e l u ( s e l f . f e a t u r e s1 0( x ) ) x = F . r e l u ( s e l f . f e a t u r e s1 2( x ) ) x = F . r e l u ( s e l f . f e a t u r e s1 4( x ) ) x = s e l f . maxpool ( x )
40 x = F . r e l u ( s e l f . f e a t u r e s1 7( x ) ) x = F . r e l u ( s e l f . f e a t u r e s1 9( x ) ) x = F . r e l u ( s e l f . f e a t u r e s2 1( x ) ) x = s e l f . maxpool ( x )
x = F . r e l u ( s e l f . f e a t u r e s2 4( x ) )
45 x = F . r e l u ( s e l f . f e a t u r e s2 6( x ) ) x = F . r e l u ( s e l f . f e a t u r e s2 8( x ) ) x = s e l f . maxpool ( x )
x = s e l f . a v g p o o l ( x ) x = t o r c h . f l a t t e n ( x , 1)
50 x = F . r e l u ( s e l f . c l a s s i f i e r0( x ) ) x = F . d r o p o u t ( x , 0.5 5)
x = F . r e l u ( s e l f . c l a s s i f i e r3( x ) ) x = F . d r o p o u t ( x , 0.5 5)
x = s e l f . c l a s s i f i e r6( x )
55 r e t u r n x
Listing C.2: Definitioon of the Student model.
c l a s s S t u d e n t N e t ( nn . Module ) : d e f i n i t ( s e l f ) :
s u p e r ( StudentNet , s e l f ) . i n i t ( )
s e l f . conv1=nn . Conv2d ( i n c h a n n e l s=3, o u t c h a n n e l s=8, k e r n e l s i z e=3, p a d d i n g=1, s t r i d e=1, b i a s=F a l s e )
5 s e l f . conv2=nn . Conv2d ( i n c h a n n e l s=8, o u t c h a n n e l s=1 6, k e r n e l s i z e=3, p a d d i n g=1, s t r i d e=1, b i a s=F a l s e )
s e l f . conv3=nn . Conv2d ( i n c h a n n e l s=1 6, o u t c h a n n e l s=3 2, k e r n e l s i z e=3, p a d d i n g=1, s t r i d e=1, b i a s=F a l s e )
s e l f . conv4=nn . Conv2d ( i n c h a n n e l s=3 2, o u t c h a n n e l s=6 4, k e r n e l s i z e=3, p a d d i n g=1, s t r i d e=1, b i a s=F a l s e )
s e l f . conv4 1=nn . Conv2d ( i n c h a n n e l s=6 4, o u t c h a n n e l s=6 4, k e r n e l s i z e=3, p a d d i n g=1, s t r i d e=1, b i a s=F a l s e )
s e l f . conv5=nn . Conv2d ( i n c h a n n e l s=6 4, o u t c h a n n e l s=1 2 8, k e r n e l s i z e=3, p a d d i n g=1, s t r i d e=1, b i a s=F a l s e )
10 s e l f . conv5 1=nn . Conv2d ( i n c h a n n e l s=1 2 8, o u t c h a n n e l s=1 2 8, k e r n e l s i z e=
3, p a d d i n g=1, s t r i d e=1, b i a s=F a l s e )
s e l f . a v g p o o l = nn . AdaptiveAvgPool2d ( (7, 7) ) s e l f . p o o l = nn . MaxPool2d ( k e r n e l s i z e=2, s t r i d e=2)
15 s e l f . f c1=nn . L i n e a r (7✯7✯1 2 8,1 0 0, b i a s=F a l s e ) s e l f . f c2=nn . L i n e a r (1 0 0,1 0 0, b i a s=F a l s e ) s e l f . f c2 1=nn . L i n e a r (1 0 0,1 0 0, b i a s=F a l s e ) s e l f . f c3=nn . L i n e a r (1 0 0,5 0, b i a s=F a l s e ) s e l f . f c4=nn . L i n e a r (5 0,1 0, b i a s=F a l s e )
20
d e f f o r w a r d ( s e l f , x ) :
x= s e l f . p o o l (F . r e l u ( s e l f . conv1( x ) ) ) x= s e l f . p o o l (F . r e l u ( s e l f . conv2( x ) ) ) x= s e l f . p o o l (F . r e l u ( s e l f . conv3( x ) ) )
25 x=F . r e l u ( s e l f . conv4( x ) )
x= s e l f . p o o l (F . r e l u ( s e l f . conv4 1( x ) ) ) x=F . r e l u ( s e l f . conv5( x ) )
x= s e l f . p o o l (F . r e l u ( s e l f . conv5 1( x ) ) ) x = s e l f . a v g p o o l ( x )
30 x = t o r c h . f l a t t e n ( x , 1) x=F . r e l u ( s e l f . f c1( x ) ) x = F . d r o p o u t ( x , 0.5) x=F . r e l u ( s e l f . f c2( x ) ) x = F . d r o p o u t ( x , 0.5)
35 x=F . r e l u ( s e l f . f c2 1( x ) ) x = F . d r o p o u t ( x , 0.5) x=F . r e l u ( s e l f . f c3( x ) ) x = F . d r o p o u t ( x , 0.5) x= s e l f . f c4( x )
40 r e t u r n x
Main code relating the Teacher model
Listing D.1: Tranfer learning of the VGG16 model.
b i a s t e a c h e r m o d e l = models . vgg1 6( p r e t r a i n e d=True ) n o b i a s t e a c h e r m o d e l = VGG1 6( )
#We a d j u s t t h e f i n a l l a y e r o f t h e vgg1 6 p r e t r a i n e d model
5 i n p u t l a s t l a y e r = b i a s t e a c h e r m o d e l . c l a s s i f i e r [6] . i n f e a t u r e s b i a s t e a c h e r m o d e l . c l a s s i f i e r [6] = nn . L i n e a r ( i n p u t l a s t l a y e r , 1 0)
#We copy t h e s t a t e d i c t i o n a r y o f t h e o r i g i n a l model .
b i a s s t a t e d i c t = copy . deepcopy ( b i a s t e a c h e r m o d e l . s t a t e d i c t ( ) )
10 n o b i a s s t a t e d i c t = {}
#We need t o t u n e t h e name o f t h e modules i n o r d e r t o copy t h e s t a t e d i c t i o n a r y w i t h o u t b i a s e s i n t o t h e new VGG1 6 model
f o r key , v a l u e i n b i a s s t a t e d i c t . i t e m s ( ) : i f ’ b i a s ’ n o t i n key :
15 i f ’ f e a t u r e s ’ i n key :
n o b i a s s t a t e d i c t [ key . r e p l a c e (” s . ”,” s ”) ] = v a l u e e l i f ’ c l a s s i f i e r ’ i n key :
n o b i a s s t a t e d i c t [ key . r e p l a c e (” r . ”,” r ”) ] = v a l u e
20 n o b i a s t e a c h e r m o d e l . l o a d s t a t e d i c t ( n o b i a s s t a t e d i c t )
#F i n a l l y , h e r e we have t h e VGG1 6 t e a c h e r model w i t h o u t b i a s e s t e a c h e r m o d e l = n o b i a s t e a c h e r m o d e l
25 d e l b i a s t e a c h e r m o d e l d e l b i a s s t a t e d i c t
Listing D.2: Structured pruning loop using the percentage of pruned nodes obtained from the unstructured pruning experiment.
i f epoch > 1 0:
f o r name , module i n model . named modules ( ) :
i f i s i n s t a n c e ( module , nn . Conv2d ) : i f name == ’ f e a t u r e s2 8’:
5 prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 3, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’) e l i f name == ’ f e a t u r e s2 6’:
prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 1, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’)
10 e l i f i s i n s t a n c e ( module , nn . L i n e a r ) : i f name == ’ c l a s s i f i e r0’:
prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 5, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’) e l i f name == ’ c l a s s i f i e r3’:
15 prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 1, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’) i f epoch % 2 == 0:
i f i s i n s t a n c e ( module , nn . Conv2d ) : i f epoch < 2 0:
20 i f name == ’ f e a t u r e s2 4’:
prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0. 0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’) i f epoch < 1 8:
i f name == ’ f e a t u r e s2 1’:
25 prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0. 0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’) i f epoch < 1 6:
i f name == ’ f e a t u r e s1 9’ o r name == ’ f e a t u r e s1 7’ o r name
== ’ f e a t u r e s1 4’ o r name == ’ f e a t u r e s1 2’ o r name == ’ f e a t u r e s1 0’ o r name
== ’ f e a t u r e s7’ o r name == ’ f e a t u r e s5’ o r name == ’ f e a t u r e s2’ o r name == ’ f e a t u r e s0’:
prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0. 0 2, n=1, dim=0)
30 prune . remove ( module , ’ w e i g h t ’)
Main code relating the Student model.
Listing E.1: Distillation loss function of the Student model.
d e f d i s t i l l a t i o n l o s s ( s c o r e s , t a r g e t s , T=5) : s o f t p r e d=F . s o f t m a x ( s c o r e s /T, dim=1) s o f t t a r g e t s=F . s o f t m a x ( t a r g e t s /T, dim=1) l o s s=F . m s e l o s s ( s o f t p r e d , s o f t t a r g e t s )
5 r e t u r n l o s s
Listing E.2: Structured pruning of the Student model following the static approach.
i f epoch > 2 0 0 and epoch < 2 8 0: i f epoch % 7 == 0:
f o r name , module i n model . named modules ( ) :
i f i s i n s t a n c e ( module , nn . Conv2d ) :
5 prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0) #b e f o r e 0.0 1 4
prune . remove ( module , ’ w e i g h t ’) e l i f i s i n s t a n c e ( module , nn . L i n e a r ) :
i f name != ’ f c4’:
prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
10 prune . remove ( module , ’ w e i g h t ’)
Listing E.3: Early pruning strategy followed in the dynamic structured pruning approach.
i f epoch > 1 0 and epoch < 2 2 0: i f epoch % 1 0 == 0:
f o r name , module i n model . named modules ( ) :
i f i s i n s t a n c e ( module , nn . Conv2d ) :
5 i f name == ’ conv5 1’:
prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’) e l i f i s i n s t a n c e ( module , nn . L i n e a r ) :
i f name == ’ f c1’:
10 prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’) i f epoch == 5 0 o r epoch == 1 5 0:
f o r name , module i n model . named modules ( ) :
i f i s i n s t a n c e ( module , nn . Conv2d ) :
15 i f name == ’ conv4’ o r name == ’ conv5’:
prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’) e l i f i s i n s t a n c e ( module , nn . L i n e a r ) :
i f name == ’ f c2’:
20 prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’)
i f epoch == 6 0 o r epoch == 1 2 0 o r epoch == 1 8 0 o r epoch == 2 4 0:
f o r name , module i n model . named modules ( ) :
i f i s i n s t a n c e ( module , nn . Conv2d ) :
25 i f name == ’ conv3’:
prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’) e l i f i s i n s t a n c e ( module , nn . L i n e a r ) :
i f name == ’ f c2 1’ o r name == ’ f c3’:
30 prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’)
i f epoch == 5 0 o r epoch == 1 0 0 o r epoch == 1 5 0 o r epoch == 2 0 0 o r epoch == 2 5 0:
f o r name , module i n model . named modules ( ) :
i f i s i n s t a n c e ( module , nn . Conv2d ) :
35 i f name == ’ conv4 1’:
prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’)
Listing E.4: Middle pruning strategy followed in the dynamic structured pruning approach.
i f epoch > 1 0 0:
i f epoch % 1 0 == 0:
f o r name , module i n model . named modules ( ) :
i f i s i n s t a n c e ( module , nn . Conv2d ) :
5 i f name == ’ conv5 1’:
prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’) e l i f i s i n s t a n c e ( module , nn . L i n e a r ) :
i f name == ’ f c1’:
10 prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’) i f epoch == 1 0 1 o r epoch == 2 0 1:
f o r name , module i n model . named modules ( ) :
i f i s i n s t a n c e ( module , nn . Conv2d ) :
15 i f name == ’ conv4’ o r name == ’ conv5’:
prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’) e l i f i s i n s t a n c e ( module , nn . L i n e a r ) :
i f name == ’ f c2’:
20 prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’)
i f epoch == 1 2 0 o r epoch == 1 6 0 o r epoch == 2 0 0 o r epoch == 2 4 0:
f o r name , module i n model . named modules ( ) :
i f i s i n s t a n c e ( module , nn . Conv2d ) :
25 i f name == ’ conv3’:
prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’) e l i f i s i n s t a n c e ( module , nn . L i n e a r ) :
i f name == ’ f c2 1’ o r name == ’ f c3’:
30 prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’)
i f epoch == 1 1 0 o r epoch == 1 4 0 o r epoch == 1 7 0 o r epoch == 2 0 0 o r epoch
== 2 3 0:
f o r name , module i n model . named modules ( ) :
i f i s i n s t a n c e ( module , nn . Conv2d ) :
35 i f name == ’ conv4 1’:
prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’)
Listing E.5: Late pruning strategy followed in the dynamic structured pruning approach.
i f epoch > 2 0 0:
i f epoch % 5 == 0:
f o r name , module i n model . named modules ( ) :
i f i s i n s t a n c e ( module , nn . Conv2d ) :
5 i f name == ’ conv5 1’:
prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’) e l i f i s i n s t a n c e ( module , nn . L i n e a r ) :
i f name == ’ f c1’:
10 prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’) i f epoch == 2 1 0 o r epoch == 2 2 0:
f o r name , module i n model . named modules ( ) :
i f i s i n s t a n c e ( module , nn . Conv2d ) :
15 i f name == ’ conv4’ o r name == ’ conv5’:
prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’) e l i f i s i n s t a n c e ( module , nn . L i n e a r ) :
i f name == ’ f c2’:
20 prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’)
i f epoch == 2 1 0 o r epoch == 2 3 0 o r epoch == 2 5 0 o r epoch == 2 7 0:
f o r name , module i n model . named modules ( ) :
i f i s i n s t a n c e ( module , nn . Conv2d ) :
25 i f name == ’ conv3’:
prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’) e l i f i s i n s t a n c e ( module , nn . L i n e a r ) :
i f name == ’ f c2 1’ o r name == ’ f c3’:
30 prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’)
i f epoch == 2 2 0 o r epoch == 2 3 0 o r epoch == 2 4 0 o r epoch == 2 5 0 o r epoch
== 2 6 0:
f o r name , module i n model . named modules ( ) :
i f i s i n s t a n c e ( module , nn . Conv2d ) :
35 i f name == ’ conv4 1’:
prune . l n s t r u c t u r e d ( module , name=” w e i g h t ”, amount=0.0 2, n=1, dim=0)
prune . remove ( module , ’ w e i g h t ’)
S. Anwar, K. Hwang, and W. Sung. Structured pruning of deep convolutional neural networks, 2015. URL https://arxiv.org/abs/1512.08571.
P. Biondi and the Scapy community. Scapy. https://scapy.readthedocs.io/en/
latest/introduction.html, 2019. [Online; accessed 14-dec-2021].
D. Blalock, J. J. G. Ortiz, J. Frankle, and J. Guttag. What is the state of neural network pruning?, 2020. URL https://arxiv.org/abs/2003.03033.
A. Bragagnolo and C. A. Barbano. Simplify: A python library for optimizing pruned neural networks. SoftwareX, 17, 2022. URL https://doi.org/10.1016/j.softx.2021.
100907.
C. Bucilua, R. Caruana, and A. Niculescu-Mizil. Model compression. Association for Com-puting Machinery, 2006. URL https://doi.org/10.1145/1150402.1150464.
T. Chen, B. Ji, T. Ding, B. Fang, G. Wang, Z. Zhu, L. Liang, Y. Shi, S. Yi, and X. Tu. Only Train Once: A One-Shot Neural Network Training And Pruning Framework, volume 34.
Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/
2021/file/a376033f78e144f494bfc743c0be3330-Paper.pdf.
R. R. Fontes, S. Afzal, S. H. B. Brito, M. Santos, and C. E. Rothenberg. Mininet-wifi: Emulating software-defined wireless networks. 2nd International Workshop on Management of SDN and NFV Systems 2015, 2015. URL https://ieeexplore.
ieee.org/document/7367387.
A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer. A survey of quantization methods for efficient neural network inference, 2021. URL https:
//arxiv.org/abs/2103.13630.
F. Girosi, M. Jones, and T. Poggio. Regularization theory and neural networks architec-tures. Neural Computation, 7(2):219–269, 03 1995. ISSN 0899-7667. doi: 10.1162/neco.
1995.7.2.219. URL https://doi.org/10.1162/neco.1995.7.2.219.
J. Gou, B. Yu, S. J. Maybank, and D. Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789–1819, mar 2021. doi: 10.1007/s11263-021-01453-z. URL https://doi.org/10.1007/
2Fs11263-021-01453-z.
S. Han, J. Tran, and W. Dally. Learning both weights and connections for efficient neural networks. Journal of Systems Architecture, 122:357–362, 2022. URL https://doi.
org/10.1016/j.sysarc.2021.102336.
C. Harris, K. Millman, and S. Van der Walt. Array programming with numpy. Nature, 585:357–362, 2020. URL https://doi.org/10.1038/s41586-020-2649-2.
G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network, 2015.
URL https://doi.org/10.48550/arXiv.1503.02531.
Y. Huang and Y. Chen. Autonomous driving with deep learning: A survey of state-of-art technologies, 2020. URL https://arxiv.org/abs/2006.06091.
Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel.
Backpropagation applied to handwritten zip code recognition. Neural Computation, 1:
541–551, 1989.
X. Li, W. Hu, C. Li, T. Jiang, H. Sun, X. Li, X. Huang, and M. Grzegorzek. A state-of-the-art survey of artificial neural networks for whole-slide image analysis:from popular convolutional neural networks to potential visual transformers, 2021. URL https:
//arxiv.org/abs/2104.06243.
G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. van der Laak, B. van Ginneken, and C. I. S´anchez. A survey on deep learning in medical image analysis. Medical Image Analysis, 42:60–88, dec 2017. doi: 10.1016/j.media.2017.07.005.
URL https://doi.org/10.1016/2Fj.media.2017.07.005.
Y. Matsubara, M. Levorato, and F. Restuccia. Split computing and early exiting for deep learning applications: Survey and research challenges, 2021. URL https://arxiv.
org/abs/2103.04505.
T. Mitchell. Machine Learning. McGraw Hill, 1997. ISBN 0-07-042807-7.
P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz. Pruning convolutional neu-ral networks for resource efficient inference, 2016. URL https://arxiv.org/abs/
1611.06440.
M. Paganini and J. Forde. On iterative neural network pruning, reinitialization, and the similarity of masks, 2020. URL https://arxiv.org/abs/2001.05050.