Squeezenet / squeezenext | lightweight network

Time:2021-7-30

Squeezenet series is an early and classic lightweight network. Squeezenet uses fire module for parameter compression, and squeezenext adds separation convolution to improve it. Although the squeezenet series is not as widely used as mobinet, its architecture idea and experimental conclusions can be used for reference.

Source: Xiaofei’s algorithm Engineering Notes official account

SqueezeNet


Paper: squeezenet: alexnet level accuracy with 50x fewer parameters and < 0.5Mb model size

Squeezenet / squeezenext | lightweight network

Introduction

When deep learning emerged, many studies focused on how to improve the accuracy of the network, and squeezenet was one of the early studies that began to focus on lightweight networks. The original intention of this paper is to greatly reduce the parameters of the model by optimizing the structure of the network.

SqueezeNet: Preserving Accuracy with Few Parameters

  • Architectural Design Strategies

The lightweight network design of this paper mainly includes the following strategies:

  1. Replace $3 \ times 3 $convolution with $1 \ times 1 $convolution, because $1 \ times 1 $convolution has 9 times less parameters than $3 \ times 3 $convolution.
  2. Reduce the input dimension of $3 \ times $convolution. The total parameter of $3 \ times $convolution is (number of input channels)(number of filters)(3 * 3). Using squeeze layers to reduce the input dimension can reduce the overall amount of calculation.
  3. The down sampling operation shall be arranged in the later stage of the network as far as possible, so that the convolution layer can have a larger characteristic map and save more information, so as to improve the accuracy.

Strategy 1 and strategy 2 are mainly to reduce network parameters but maintain accuracy, while strategy 3 is to maximize accuracy under limited parameters.

  • The Fire Module

Squeezenet / squeezenext | lightweight network

The core module of squeezenet is the fire module. The structure is shown in Figure 1. The input layer first compresses the dimension through the squeeze convolution layer ($1 \ times 1 $convolution), and then expands the dimension through the expand convolution layer ($1 \ times 1 $convolution and $3 \ times 3 $convolution). The fire module contains three parameters, which are $1 \ times $convolution kernels of the squeeze layer $s_{ 1×1} $, expansion layer $1 \ times 1 $convolution kernels $e_{ 1×1} $and $3 \ times 3 $convolution kernels of expand layer $e_{ 3×3} $, general $s_{ 1×1}<(e_{ 1×1}+e_{ 3×3})$

  • The SqueezeNet Architecture

Squeezenet / squeezenext | lightweight network

The structure of squeezenet is shown in Table 1. After conv1, fire4, fire8 and conv10, a pooling layer is added for pooling operation, and the output dimension in the network is gradually improved. The output of each squeeze layer and expand layer is activated through relu, and 50% dropout will be connected behind the fire9 module.

Evaluation of SqueezeNet

Squeezenet / squeezenext | lightweight network

Compared with alexnet, squeezenet only needs 1 / 50 of the parameter quantity under the same accuracy. After quantization, it can be reduced to 1 / 510 of the parameter quantity at most.

CNN Microarchitecture Design Space Exploration

Squeezenet / squeezenext | lightweight network

This paper explores the setting of fire module, mainly comparing the compression ratio of squeeze layer and the convolution ratio of $3 \ times 3 $in expand layer.

CNN Macroarchitecture Design Space Exploration

Squeezenet / squeezenext | lightweight network

Squeezenet / squeezenext | lightweight network

This paper has carried out exploratory experiments on the network architecture, mainly to study the impact of short-circuit connection on the network. The comparative network structure is shown in Figure 2.

Conclusion

As an early lightweight network research work, although the accuracy of squeezenet is compared with that of alexnet, its network compression ratio is considerable, and the design of fire module is also very novel.

SqueezeNext


Paper: squeezenext: Hardware aware neural network design

Squeezenet / squeezenext | lightweight network

Introduction

Squeezenext is an upgrade of the practical version of squeezenet, which directly compares the performance with mobilenet. Squeezenext uses standard convolution to analyze the actual reasoning speed. The optimization means focus on the optimization of the overall network structure.

SqueezeNext Design

Squeezenet / squeezenext | lightweight network

The design of squeezenext follows the residual structure. Instead of using the popular depth convolution at that time, it directly uses the separation convolution. The design is mainly based on the following strategies:

  • Low Rank Filters

The core idea of low rank decomposition is to decompose a large matrix into multiple small matrices. Here, CP decomposition is used to decompose the convolution of $K \ times K $into separate convolutions of $K \ times 1 $and $1 \ times K $. The parameter quantity can be reduced from $k ^ 2 $to $2K $.

  • Bottleneck Module

The parameter quantity is related to the input-output dimension. Although deep convolution can be used to reduce the amount of calculation, the calculation of deep convolution in the terminal system is not efficient. Therefore, the squeeze layer of squeezenet is used to compress the input dimension. Two consecutive squeeze layers are used at the beginning of each block, and each layer is reduced by 1 / 2 dimension.

  • Fully Connected Layers

In alexnet, the parameters of the full connection layer account for 96% of the total model. Squeezenext uses the bottleneck layer to reduce the input dimension of the full connection layer, so as to reduce the amount of network parameters.

Squeezenet / squeezenext | lightweight network

The basic 1.0-sqnxt-23 structure is shown in Figure 3. The blocks in the middle are squeezenext blocks, the first block is normal convolution, and the last two blocks are bottneck module and full connection layer respectively.

Result

Squeezenet / squeezenext | lightweight network

This paper compares different networks and different versions of squeezenext, including different network lengths and different structures. 1.0 in the table represents the basic layer dimension setting, G represents the convolution group size set to 2, the following number is the total number of layers, and Ida represents the use of iterative deep aggregation to fuse multiple layers for output.

Squeezenet / squeezenext | lightweight network

In addition, the paper also compares the performance of a wider network and enlarges the dimension.

Squeezenet / squeezenext | lightweight network

The structure of V5 is shown in Figure 9. In the simulation hardware performance experiment results, it is found that the lower the dimension is, the less efficient the computing performance is. Therefore, more layer operations are concentrated on blocks with higher dimensions.

Conclusion

On the compression idea of squeezenet, squeezenext combines separation convolution to improve parameter compression. The experiment of simulating hardware reasoning performance in this paper is very wonderful. You can see how the author improves the overall structure of the network step by step. If you are interested, you can see the original text.

CONCLUSION


Squeezenet series is an early and classic lightweight network. Squeezenet uses fire module for parameter compression, and squeezenext adds separation convolution to improve it. Although the squeezenet series is not as widely used as mobinet, its architecture idea and experimental conclusions can be used for reference.



If this article is helpful to you, please like it or read it
More content, please pay attention to WeChat official account.

Squeezenet / squeezenext | lightweight network