Jedi technique for reducing convolutional networks — pruning

You are once again faced with the task of object detection. The priority is the speed of operation with acceptable accuracy. You take the YOLOv3 architecture and fine-tune it. The accuracy (mAP75) should be above 0.95. However, the processing speed is still low. Damn.

Today we will bypass quantization. Below, we will consider Model Pruning — trimming redundant parts of the network to speed up inference without losing accuracy. Clearly showing where, how much, and how you can cut. We will discuss how to do this manually and where automation can be applied. At the end — a repository on Keras.

Introduction

At my last job, at Macroscop in Perm, I acquired a habit — always monitoring the execution time of algorithms. And always checking the networks' processing times through a sanity filter. Usually, state-of-the-art models don’t pass this filter in production, which led me to Pruning.

Pruning is an old topic, discussed in Stanford lectures in 2017. The main idea is to reduce the size of the trained network without losing accuracy by removing different nodes. It sounds great, but I rarely hear about its application. Perhaps, there’s a lack of implementations, no articles in Russian, or maybe everyone considers pruning a proprietary technique and remains silent.
But let's break it down

A glance at biology

I love when ideas from biology peek into Deep Learning. You can trust them, just like evolution (did you know that ReLU is quite similar to the activation function of neurons in the brain??)

The Model Pruning process is also close to biology. The network's reaction can be compared to the plasticity of the brain. There are a couple of interesting examples in the book by Norman Doidge.:

1. A woman's brain, which was born with only one hemisphere, reprogrammed itself to perform the functions of the missing hemisphere.
2. A guy shot himself in the part of the brain responsible for vision. Over time, other parts of the brain took over these functions. (We are not trying to repeat this)

Just as you can cut parts of weak convolutions from your model. In the worst case, the remaining convolutions will help compensate for the ones that were cut.

Do you prefer Transfer Learning or start from scratch?

Option number one. You are using Transfer Learning with Yolov3, Retina, Mask-RCNN, or U-Net. But most often we don't need to recognize 80 object classes like in COCO. In my experience, it's usually limited to 1-2 classes. One might assume that the architecture for 80 classes is excessive here. It raises the question of reducing the architecture. Moreover, I would like to do this without losing the existing pretrained weights.

Option two. Maybe you have a lot of data and computational resources or just need a highly customized architecture. It doesn't matter. But you are training the network from scratch. The usual order is to look at the data structure, select an OVERLY powerful architecture, and apply dropout to combat overfitting. I have seen dropouts of 0.6, Carl.

In both cases, the network can be reduced. You motivated me. Now let's figure out what pruning is.

General algorithm

We decided that we can remove convolutions. This looks quite simple:

Removing any convolution is a stress for the network, which usually leads to some increase in error. On one hand, this increase in error indicates how correctly we are removing convolutions (for example, a significant increase suggests that we are doing something wrong). However, a small increase is quite acceptable and is often mitigated by subsequent light retraining with a small LR. We add a retraining step:

Now we need to understand when we want to stop our LearningPruning cycle. There can be exotic cases where we need to reduce the network to a specific size and speed of execution (for example, for mobile devices). However, the most common scenario is to continue the cycle until the error exceeds the acceptable level. We add a condition:

So, the algorithm becomes clear. We just need to figure out how to identify removable convolutions.

Finding removable convolutions

We need to remove some convolutions. Blasting through and "shooting" any is a bad idea, even though it might work. But since we have brains, we can think and try to identify "weak" convolutions for removal. There are several options:

1. Smallest L1 measure or low_magnitude_pruning. The idea here is that convolutions with small weight values contribute little to the final decision-making.
2. Smallest L1 measure considering the mean and standard deviation. We complement with an evaluation of the distribution characteristics.
3. Masking of convolutions and excluding those with minimal impact on the final accuracy. A more accurate definition of insignificant convolutions, but quite time-consuming and resource-intensive.
4. Others

Each option has its place and specific implementation characteristics. Here, we will consider the option with the least L1 measure

Manual process for YOLOv3

The original architecture contains residual blocks. But no matter how effective they are for deep networks, they are somewhat obstructive for us. The complexity lies in the fact that we cannot remove convolutions with different indices in these layers:

Therefore, we will highlight the layers from which we can freely remove convolutions:

Now let's construct the workflow loop:

1. Dumping activations
2. Assessing how much to cut
3. Cutting
4. Training for 10 epochs with LR=1e-4
5. Testing

Dumping convolutions is useful to estimate what part we can remove at a given step. Examples of dumping:

We see that almost everywhere 5% of the convolutions have a very low L1 norm, and we can remove them. This dumping was repeated at every step, and assessments were made regarding which layers and how much could be cut.

The entire process was completed in 4 steps (here and everywhere numbers are for RTX 2060 Super):

By the second step, one positive effect emerged — the batch size of 4 fit into memory, significantly speeding up the retraining process.
At the fourth step, the process was halted, as even prolonged retraining did not raise mAp75 to previous values.
In the end, inference was accelerated by 15%, reduced size by 35% and accuracy was not lost.

Automation for simpler architectures

For simpler network architectures (without conditional add, concatenate, and residual blocks), it's quite feasible to focus on processing all convolutional layers and automate the process of trimming convolutions.

I implemented this approach here.
It's simple: you only need to provide the loss function, optimizer, and batch generators:

import pruning
from keras.optimizers import Adam
from keras.utils import Sequence

train_batch_generator = BatchGenerator...
score_batch_generator = BatchGenerator...

opt = Adam(lr=1e-4)
pruner = pruning.Pruner("config.json", "categorical_crossentropy", opt)

pruner.prune(train_batch, valid_batch)

If necessary, configuration parameters can be adjusted:

{
    "input_model_path": "model.h5",
    "output_model_path": "model_pruned.h5",
    "finetuning_epochs": 10, # the number of epochs for train between pruning steps
    "stop_loss": 0.1, # loss for stopping process
    "pruning_percent_step": 0.05, # part of convs for delete on every pruning step
    "pruning_standart_deviation_part": 0.2 # shift for limit pruning part
}

Additionally, a restriction based on standard deviation has been implemented. The goal is to limit the part being removed, excluding convolutions with already 'sufficient' L1 measures:

Thus, we allow only weak convolutions to be removed from distributions similar to the right and not affect the removal from distributions similar to the left:

As the distribution approaches normality, the coefficient pruning_standart_deviation_part can be selected from:

I recommend a tolerance of 2 sigma. Or you can disregard this feature, leaving the value < 1.0.

The output generates a graph of the network size, loss, and execution time throughout the test, normalized to 1.0. For example, here the network size was reduced almost by half without loss in quality (a small convolutional network with 100k weights):

The execution speed is subject to normal fluctuations and has changed little. There is an explanation for this:

1. The number of convolutions changes from convenient (32, 64, 128) to less convenient for graphics cards—27, 51, etc. I might be mistaken here, but this likely has an impact.
2. The architecture isn’t wide, but it is sequential. By decreasing the width, we do not touch the depth. Thus, we reduce the load without changing the speed.

Therefore, the improvement manifested as a 20-30% decrease in CUDA load during execution, but not in the execution time.

Summary

Let's reflect. We examined 2 pruning options— for YOLOv3 (when manual intervention is needed) and for simpler architecture networks. It is evident that in both cases, a reduction in network size and acceleration can be achieved without loss of accuracy. Results:

• Reduction in size
• Acceleration of execution
• Reduction in CUDA load
• Consequently, environmental friendliness (We optimize the future use of computational resources. Somewhere, a Greta Thunberg)

Appendix

• After the pruning step, quantization can also be fine-tuned (for example, with TensorRT)
• TensorFlow provides opportunities for low_magnitude_pruning. It works.
• Repository I want to develop this and will be glad for assistance.

Source: habr.com