Results
4.1 Models’ search with PIT
4.1.2 MobileNet
x y z phi 0.1
0.2 0.3 0.4 0.5
MAE
1
7FrontNet FP32 17FrontNet UINT8 ID 13FrontNet FP32
1
3FrontNet UINT8 ID FrontNet FP32
Figure 4.5: MAE of non-quantized vs quantized ID FrontNet models
x y z phi
0.2 0.4 0.6 0.8
R2score
1
7FrontNet FP32 17FrontNet UINT8 ID 13FrontNet FP32
1
3FrontNet UINT8 ID FrontNet FP32
Figure 4.6: R2 score of non-quantized vs quantized ID FrontNet models
the respective results are listed in table 4.5 for x and y while, table 4.6 for z and ϕ.
The values of λ used have been chosen to start from λ = 0 and iteratively increased in order to reach smaller networks. Starting from a total number of parameters equal to 213956 for the original MobileNet v1 with α = 0.25, the most compact reached has 5824 parameters.
In the following graphs, the networks with λ = 1 · 10−8, λ = 5 · 10−8,λ = 1 · 10−10 are referred respectively as 17 MobileNet, 15 MobileNet, 14 MobileNet in order to further stress the reduction in dimension with respect to the original model.
Table 4.5: Models with changing strength, x and y performance
Strength
x y
MSE MAE R2 MSE MAE R2
0 0.0459 0.1594 0.8546 0.1223 0.2125 0.4733 5 · 10−11 0.0529 0.1751 0.8326 0.0837 0.1916 0.6395 1 · 10−10 0.0510 0.1709 0.8384 0.0850 0.2018 0.6340 1 · 10−8 0.0604 0.1841 0.8086 0.0742 0.1822 0.6803 5 · 10−8 0.0373 0.1431 0.8819 0.0730 0.1917 0.6858 1 · 10−7 0.0566 0.1787 0.8209 0.0801 0.1858 0.6552 5 · 10−7 0.0404 0.1492 0.8720 0.1162 0.2007 0.4996 5 · 10−6 0.0539 0.1770 0.8294 0.0804 0.1923 0.6539 1 · 10−5 0.0459 0.1641 0.8548 0.0740 0.1904 0.6815 5 · 10−5 0.0689 0.1952 0.7817 0.0751 0.1886 0.6766
Table 4.6: Models with changing strength, z and ϕ performance
Strength
z ϕ
MSE MAE R2 MSE MAE R2
0 0.0145 0.0838 0.6038 0.3547 0.4548 0.1892 5 · 10−11 0.0170 0.0895 0.5365 0.3788 0.4738 0.1342 1 · 10−10 0.0139 0.0813 0.6198 0.3428 0.4401 0.2163 1 · 10−8 0.0210 0.1015 0.4272 0.4643 0.5330 -0.0612 5 · 10−8 0.0140 0.0797 0.6192 0.3986 0.4835 0.0889 1 · 10−7 0.0388 0.1424 -0.0591 0.3986 0.4835 0.0889 5 · 10−7 0.0134 0.0785 0.6334 0.4944 0.5473 -0.1301 5 · 10−6 0.0391 0.1374 -0.0667 0.3898 0.4826 0.1089 1 · 10−5 0.0394 0.1388 -0.0743 0.3821 0.4721 0.1265 5 · 10−5 0.0389 0.1339 -0.0611 0.3317 0.4459 0.2418
Quantization of MobileNet v1 0.25 models: Fake quantized
Table 4.7 and figures 4.7, 4.8 and 4.9 report the results for MobileNet v1 0.25.
Also, in this case, the lose in performance is sufficiently low and all the networks maintain the results of the FP32 test set.
Table 4.7: Non quantized vs quantized FQ MobileNet v1 0.25 models
Name 1 7Mobile
FP32 1 7Mobile
UINT8 1 5 Mobile
FP32 1 5Mobile
UINT8 1 4Mobile
FP32 1 4 Mobile
UINT8
FrontNet FP32
Strength 1E-8 1E-8 5E-8 5E-8 1E-10 1E-10 0
FLOPS 4.06 · 106 4.06 · 106 5.37 · 106 5.37 · 106 7.40 · 106 7.40 · 107 1.47 · 107
Par [MB] 0.11 0.11 0.14 0.14 0.21 0.21 1.14
# par 29281 29281 37709 37709 56302 56302 304356
MSE x 0.0604 0.0850 0.0373 0.0390 0.0510 0.0519 0.0621
MSE y 0.0742 0.0670 0.0730 0.0735 0.0850 0.0927 0.0805
MSE z 0.0210 0.0197 0.0140 0.0134 0.0139 0.0140 0.0166
MSE phi 0.4643 0.4587 0.4806 0.4739 0.3428 0.3644 0.3244
MAE x 0.1841 0.2282 0.1431 0.1478 0.1709 0.1714 0.187
MAE y 0.1822 0.1726 0.1917 0.1731 0.2018 0.2118 0.1837
MAE z 0.1015 0.1002 0.0797 0.0793 0.0813 0.0823 0.0927
MAE phi 0.5330 0.5298 0.5389 0.5353 0.4401 0.4526 0.4367
R2 x 0.8086 0.7309 0.8819 0.8765 0.8384 0.8356 0.8035
R2 y 0.6803 0.7117 0.6858 0.6835 0.6340 0.6010 0.6532
R2 z 0.4272 0.4628 0.6192 0.6351 0.6198 0.6192 0.5481
R2 phi -0.0612 -0.0486 -0.0985 -0.0832 0.2163 0.1670 0.2585
H
x y z phi
0 0.1 0.2 0.3 0.4 0.5
MSE
1
7Mobile FP32 17Mobile UINT8 15Mobile FP32 15Mobile UINT8
1
4Mobile FP32 14Mobile UINT8 FrontNet FP32
Figure 4.7: MSE of non-quantized vs quantized FQ Mobilenet v1 0.25 models
H
x y z phi
0.1 0.2 0.3 0.4 0.5 0.6
MAE
1
7Mobile FP32 17Mobile UINT8 15Mobile FP32 15Mobile UINT8
1
4Mobile FP32 14Mobile UINT8 FrontNet FP32
Figure 4.8: MAE of non-quantized vs quantized FQ Mobilenet v1 0.25 models
x y z phi
0.10 0.20.3 0.40.5 0.60.7 0.80.9
R2score
1
7Mobile FP32 17Mobile UINT8 15Mobile FP32 15Mobile UINT8
1
4Mobile FP32 14Mobile UINT8 FrontNet FP32
Figure 4.9: R2 score of non-quantized vs quantized FQ Mobilenet v1 0.25 models
Quantization of MobileNet v1 0.25 models: Integer deployable
Finally in the case of MobileNet v1 0.25 in the step from fake quantized to integer deployable there is no loss of performance in any of the cases reported below.
Details may be found in table 4.8 and graphs 4.10, 4.11 and 4.12.
Table 4.8: MobileNet v1 models, width multiplier = 0.25, Integer Deployable performance
Name 1 7Mobile
FP32 1 7Mobile
UINT8 1 5 Mobile
FP32 1 5Mobile
UINT8 1 4Mobile
FP32 1 4 Mobile
UINT8
FrontNet FP32
Strength 1E-8 1E-8 5E-8 5E-8 1E-10 1E-10 0
FLOPS 4.06 · 106 4.06 · 106 5.37 · 106 5.37 · 106 7.40 · 106 7.40 · 107 1.47 · 107
Par [MB] 0.11 0.11 0.14 0.14 0.21 0.21 1.14
# par 29281 29281 37709 37709 56302 56302 304356
MSE x 0.0604 0.0621 0.0373 0.0584 0.0510 0.0543 0.0621
MSE y 0.0742 0.0750 0.0730 0.0654 0.0850 0.0822 0.0805
MSE z 0.0210 0.0229 0.0140 0.0159 0.0139 0.0147 0.0166
MSE phi 0.4643 0.4633 0.4806 0.4759 0.3428 0.3428 0.3244
MAE x 0.1841 0.1900 0.1431 0.1838 0.1709 0.1778 0.187
MAE y 0.1822 0.1786 0.1917 0.1739 0.2018 0.1904 0.1837
MAE z 0.1015 0.1071 0.0797 0.0839 0.0813 0.0840 0.0927
MAE phi 0.5330 0.5332 0.5389 0.5369 0.4401 0.4415 0.4367
R2 x 0.8086 0.8032 0.8819 0.8152 0.8384 0.8280 0.8035
R2 y 0.6803 0.6769 0.6858 0.7184 0.6340 0.6460 0.6532
R2 z 0.4272 0.3763 0.6192 0.5661 0.6198 0.5990 0.5481
R2 phi -0.0612 -0.0591 -0.0985 -0.0878 0.2163 0.2163 0.2585
x y z phi
0 0.1 0.2 0.3 0.4 0.5
MSE
1
7Mobile FP32 17Mobile UINT8 15Mobile FP32 15Mobile UINT8
1
4Mobile FP32 14Mobile UINT8 FrontNet FP32
Figure 4.10: MSE of non quantized vs quantized ID Mobilenet v1 0.25 models
x y z phi 0.1
0.2 0.3 0.4 0.5 0.6
MAE
1
7Mobile FP32 17Mobile UINT8 15Mobile FP32 15Mobile UINT8
1
4Mobile FP32 14Mobile UINT8 FrontNet FP32
Figure 4.11: MAE of non quantized vs quantized ID Mobilenet v1 0.25 models
x y z phi
0.10 0.20.3 0.40.5 0.60.7 0.80.9
R2score
1
7Mobile FP32 17Mobile UINT8 15Mobile FP32 15Mobile UINT8
1
4Mobile FP32 14Mobile UINT8 FrontNet FP32
Figure 4.12: R2 score of non-quantized vs quantized ID Mobilenet v1 0.25 models
MobileNet v1 α= 1.0
In the case of MobileNet v1 with α = 1.0, the research with PIT has been done with 9 different values of strength (λ), in particular the values of λ employed and the respective results are listed in table 4.9 for x and y while, table 4.10 for z and ϕ.
The values of λ used have been chosen to start from λ = 0 and iteratively increased in order to reach smaller networks. Starting from a total number of parameters equal to 213956 for the original MobileNet v1 with α = 0.25, the most compact reached has 5824 parameters.
In the following graphs, the networks with λ = 1 · 10−8, λ = 5 · 10−8,λ = 1 · 10−10 are referred respectively as 17 MobileNet, 15 MobileNet, 14 MobileNet in order to further stress the reduction in dimension with respect to the original model.
Table 4.9: Models with changing strength, x and y performance
Strength
x y
MSE MAE R2 MSE MAE R2
0 0.0602 0.1818 0.8093 0.1466 0.2462 0.3686 1 · 10−11 0.0591 0.1817 0.8129 0.0887 0.1990 0.6182 1 · 10−10 0.0586 0.1786 0.8146 0.0598 0.1668 0.7425 1 · 10−9 0.0565 0.1802 0.8211 0.0705 0.1790 0.6965 1 · 10−8 0.0502 0.1692 0.8411 0.0852 0.2014 0.6330 5 · 10−8 0.0529 0.1754 0.8326 0.0786 0.1998 0.6617 5 · 10−7 0.0517 0.1769 0.8364 0.0721 0.1940 0.6895 1 · 10−6 0.0442 0.1632 0.8599 0.0894 0.2119 0.6151 1 · 10−5 0.0470 0.1660 0.8513 0.0955 0.1975 0.5887
Table 4.10: Models with changing strength, z and ϕ performance
Strength
z ϕ
MSE MAE R2 MSE MAE R2
0 0.0239 0.1092 0.3472 0.4472 0.5126 -0.0223 1 · 10−11 0.0171 0.0909 0.5335 0.3475 0.4506 0.2056 1 · 10−10 0.0151 0.0869 0.5885 0.3865 0.4717 0.1165 1 · 10−9 0.0142 0.0844 0.6132 0.3003 0.4190 0.3135 1 · 10−8 0.0159 0.0872 0.5664 0.2664 0.3944 0.3912 5 · 10−8 0.0141 0.0811 0.6154 0.2938 0.4205 0.3285 5 · 10−7 0.0144 0.0819 0.6076 0.3319 0.4438 0.2414 1 · 10−6 0.0146 0.0844 0.6016 0.3259 0.4333 0.2550 1 · 10−5 0.0172 0.0865 0.5308 0.3134 0.4291 0.2835
Quantization of MobileNet v1 1.0 models: Fake quantized
Table 4.11 and figures 4.13, 4.14 and 4.15 report the results for MobileNet v1 1.0.
Also, in this case, the loss in performance is sufficiently low and all the networks maintain the results of the FP32 test set.
Table 4.11: Non quantized vs quantized FQ MobileNet v1 1.0 models
Name 1851 Mobile 1851 Mobile 8 751 Mobile 751 Mobile 8 701 Mobile 701 Mobile 8 FrontNet
Strength 1E-5 1E-5 1E-6 1E-6 5E-7 5E-7 0
FLOPS 6.3 · 106 6.3 · 106 1.17 · 107 1.17 · 107 1.24 · 107 1.24 · 107 1.47 · 107
Par [MB] 0.07 0.07 0.16 0.16 0.18 0.18 1.14
# par 17603 17603 41566 41566 46256 46256 304356
MSE x 0.0470 0.0463 0.0442 0.0505 0.0517 0.0609 0.0621 MSE y 0.0955 0.0976 0.0894 0.1172 0.0721 0.0631 0.0805 MSE z 0.0172 0.0181 0.0146 0.0142 0.0144 0.0138 0.0166 MSE phi 0.3134 0.3112 0.3259 0.3230 0.3319 0.3307 0.3244
MAE x 0.1660 0.1664 0.1632 0.1729 0.1769 0.1956 0.187
MAE y 0.1975 0.1984 0.2119 0.2682 0.1940 0.1779 0.1837 MAE z 0.0865 0.0913 0.0844 0.0836 0.0819 0.0803 0.0927 MAE phi 0.4291 0.4225 0.4333 0.4312 0.4438 0.4399 0.4367
R2 x 0.8513 0.8535 0.8599 0.8401 0.8364 0.8070 0.8035
R2 y 0.5887 0.5799 0.6151 0.4953 0.6895 0.7283 0.6532
R2 z 0.5308 0.5066 0.6016 0.6114 0.6076 0.6239 0.5481
R2 phi 0.2835 0.2887 0.2550 0.2618 0.2414 0.2440 0.2585
H
x y z phi
0 0.1 0.2 0.3
MSE
1
185Mobile FP32 1851 Mobile UINT8 751 Mobile FP32 751 Mobile UINT8
1
70Mobile FP32 701 Mobile UINT8 FrontNet FP32
Figure 4.13: MSE of non quantized vs quantized FQ Mobilenet v1 1.0 models
x y z phi 0.1
0.2 0.3 0.4 0.5
MAE
1
185Mobile FP32 1851 Mobile UINT8 751 Mobile FP32 751 Mobile UINT8
1
70Mobile FP32 701 Mobile UINT8 FrontNet FP32
Figure 4.14: MAE of non quantized vs quantized FQ Mobilenet v1 1.0 models
x y z phi
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
R2score
1
185Mobile FP32 1851 Mobile UINT8 751 Mobile FP32 751 Mobile UINT8
1
70Mobile FP32 701 Mobile UINT8 FrontNet FP32
Figure 4.15: R2 score of non-quantized vs quantized FQ Mobilenet v1 1.0 models
Quantization of MobileNet v1 1.0 models: Integer deployable
Even in the case of MobileNet v1 1.0 in the step from fake quantized to integer deployable there is no loss of performance in any of the cases reported below.
Details may be found in table 4.12 and graphs 4.16, 4.17 and 4.18.
Table 4.12: MobileNet v1 models, width multiplier = 1, Integer deployable
Name 1851 Mobile 1851 Mobile 8 751 Mobile 751 Mobile 8 701 Mobile 701 Mobile 8 FrontNet
Strength 1E-5 1E-5 1E-6 1E-6 5E-7 5E-7 0
FLOPS 6.3 · 106 6.3 · 106 1.17 · 107 1.17 · 107 1.24 · 107 1.24 · 107 1.47 · 107
Par [MB] 0.07 0.07 0.16 0.16 0.18 0.18 1.14
# par 17603 17603 41566 41566 46256 46256 304356
MSE x 0.0470 0.0461 0.0442 0.0580 0.0517 0.0613 0.0621 MSE y 0.0955 0.0818 0.0894 0.0790 0.0721 0.0761 0.0805 MSE z 0.0172 0.0173 0.0146 0.0148 0.0144 0.0148 0.0166 MSE phi 0.3134 0.3037 0.3259 0.3344 0.3319 0.3153 0.3244
MAE x 0.1660 0.1650 0.1632 0.1886 0.1769 0.1935 0.187
MAE y 0.1975 0.1791 0.2119 0.1965 0.1940 0.1912 0.1837 MAE z 0.0865 0.0885 0.0844 0.0851 0.0819 0.0824 0.0927 MAE phi 0.4291 0.4214 0.4333 0.4407 0.4438 0.4301 0.4367
R2 x 0.8513 0.8540 0.8599 0.8164 0.8364 0.8060 0.8035
R2 y 0.5887 0.6476 0.6151 0.6600 0.6895 0.6723 0.6532
R2 z 0.5308 0.5286 0.6016 0.5974 0.6076 0.5968 0.5481
R2 phi 0.2835 0.3058 0.2550 0.2357 0.2414 0.2793 0.2585
x y z phi
0 0.1 0.2 0.3
MSE
1
185Mobile FP32 1851 Mobile UINT8 751 Mobile FP32 751 Mobile UINT8
1
70Mobile FP32 701 Mobile UINT8 FrontNet FP32
Figure 4.16: MSE of non quantized vs quantized ID Mobilenet v1 1.0 models
x y z phi 0.1
0.2 0.3 0.4 0.5
MAE
1
185Mobile FP32 1851 Mobile UINT8 751 Mobile FP32 751 Mobile UINT8
1
70Mobile FP32 701 Mobile UINT8 FrontNet FP32
Figure 4.17: MAE of non-quantized vs quantized ID Mobilenet v1 1.0 models
x y z phi
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
R2score
1
185Mobile FP32 1851 Mobile UINT8 751 Mobile FP32 751 Mobile UINT8
1
70Mobile FP32 701 Mobile UINT8 FrontNet FP32
Figure 4.18: R2 score of non-quantized vs quantized ID Mobilenet v1 1.0 models