I didn't get a reasonable speed-up when applying depthwise convolution to VGG16

[lawrencewxj]

Description

It should speed up for about 8x in theory，but it seems get a little bit lower in practice. Is there something wrong in my code? @piiswrong @crazy-cat
the original VGG convolution:

internel_layer = mx.sym.Convolution(data = internel_layer, kernel=(3, 3), pad=(1, 1), num_filter=filters[i], name="conv%s_%s" %(i + 1, j + 1))
internel_layer = mx.symbol.BatchNorm(data=internel_layer, name="bn%s_%s" %(i + 1, j + 1))
internel_layer = mx.sym.Activation(data=internel_layer, act_type="relu", name="relu%s_%s" %(i + 1, j + 1))

after apply depthwise convolution

internel_layer = mx.sym.Convolution(data = internel_layer, kernel=(3, 3), pad=(1, 1),num_filter=input_featuremap_channel, num_group=input_featuremap_channel, name="a_conv%s_%s" %(i + 1, j + 1))
internel_layer = mx.sym.Convolution(data=internel_layer, kernel=(1, 1), num_filter=filters[i], name="c_conv%s_%s" % (i + 1, j + 1))
internel_layer = mx.symbol.BatchNorm(data=internel_layer, name="c_bn%s_%s" % (i + 1, j + 1))
internel_layer = mx.sym.Activation(data=internel_layer, act_type="relu", name="c_relu%s_%s" % (i + 1, j + 1))

the original VGG convolution speed:

INFO:root:start with arguments Namespace(batch_size=32, benchmark=0, data_nthreads=8, data_train='/home/code/wangxijun/data/webface/scene_train.rec', data_val=None, disp_batches=20, dtype='float32', gpus='2', image_shape='3,224,224', kv_store='device', load_epoch=None, lr=0.01, lr_factor=0.1, lr_step_epochs='20,50,90', max_random_aspect_ratio=0.25, max_random_h=36, max_random_l=50, max_random_rotate_angle=10, max_random_s=50, max_random_scale=1, max_random_shear_ratio=0.1, min_random_scale=1, model_prefix='/home/code/wangxijun/model/vgg1/vgg', mom=0.9, monitor=0, network='vgg', num_classes=10575, num_epochs=400, num_examples=477362, num_layers=16, optimizer='sgd', pad_size=0, random_crop=1, random_mirror=1, rgb_mean='123.68,116.779,103.939', test_io=0, top_k=0, wd=0.0001)
[17:56:13] src/io/iter_image_recordio_2.cc:153: ImageRecordIOParser2: /home/code/wangxijun/data/webface/scene_train.rec, use 5 threads for decoding..
[17:56:23] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
INFO:root:Epoch[0] Batch [20]   Speed: 33.19 samples/sec        cross-entropy=12.565712
INFO:root:Epoch[0] Batch [40]   Speed: 33.20 samples/sec        cross-entropy=10.379888
INFO:root:Epoch[0] Batch [60]   Speed: 32.95 samples/sec        cross-entropy=9.781008
INFO:root:Epoch[0] Batch [80]   Speed: 32.94 samples/sec        cross-entropy=9.505107
INFO:root:Epoch[0] Batch [100]  Speed: 32.50 samples/sec        cross-entropy=9.321202
INFO:root:Epoch[0] Batch [120]  Speed: 32.33 samples/sec        cross-entropy=9.279530
INFO:root:Epoch[0] Batch [140]  Speed: 32.52 samples/sec        cross-entropy=9.288194
INFO:root:Epoch[0] Batch [160]  Speed: 32.27 samples/sec        cross-entropy=9.287867
INFO:root:Epoch[0] Batch [180]  Speed: 32.19 samples/sec        cross-entropy=9.276982
INFO:root:Epoch[0] Batch [200]  Speed: 31.90 samples/sec        cross-entropy=9.267534
INFO:root:Epoch[0] Batch [220]  Speed: 32.26 samples/sec        cross-entropy=9.266759

after apply depthwise convolution speed:

INFO:root:start with arguments Namespace(batch_size=32, benchmark=0, data_nthreads=8, data_train='/home/code/wangxijun/data/webface/scene_train.rec', data_val=None, disp_batches=20, dtype='float32', gpus='3', image_shape='3,224,224', kv_store='device', load_epoch=None, lr=0.01, lr_factor=0.1, lr_step_epochs='20,50,90', max_random_aspect_ratio=0.25, max_random_h=36, max_random_l=50, max_random_rotate_angle=10, max_random_s=50, max_random_scale=1, max_random_shear_ratio=0.1, min_random_scale=1, model_prefix='/home/code/wangxijun/model/vgg2/vgg', mom=0.9, monitor=0, network='vgg', num_classes=10575, num_epochs=400, num_examples=477362, num_layers=16, optimizer='sgd', pad_size=0, random_crop=1, random_mirror=1, rgb_mean='123.68,116.779,103.939', test_io=0, top_k=0, wd=0.0001)
[17:57:20] src/io/iter_image_recordio_2.cc:153: ImageRecordIOParser2: /home/code/wangxijun/data/webface/scene_train.rec, use 5 threads for decoding..
[17:57:34] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
INFO:root:Epoch[0] Batch [20]   Speed: 31.45 samples/sec        cross-entropy=14.358711
INFO:root:Epoch[0] Batch [40]   Speed: 32.21 samples/sec        cross-entropy=10.281333
INFO:root:Epoch[0] Batch [60]   Speed: 32.16 samples/sec        cross-entropy=9.439612
INFO:root:Epoch[0] Batch [80]   Speed: 32.09 samples/sec        cross-entropy=9.352679
INFO:root:Epoch[0] Batch [100]  Speed: 31.95 samples/sec        cross-entropy=9.296035
INFO:root:Epoch[0] Batch [120]  Speed: 31.97 samples/sec        cross-entropy=9.260700
INFO:root:Epoch[0] Batch [140]  Speed: 32.08 samples/sec        cross-entropy=9.323341
INFO:root:Epoch[0] Batch [160]  Speed: 31.94 samples/sec        cross-entropy=9.229615
INFO:root:Epoch[0] Batch [180]  Speed: 32.03 samples/sec        cross-entropy=9.285774
INFO:root:Epoch[0] Batch [200]  Speed: 31.97 samples/sec        cross-entropy=9.259684
INFO:root:Epoch[0] Batch [220]  Speed: 31.92 samples/sec        cross-entropy=9.250173

Hardware :
K80
Software :
cuda8.0 + cudnn5.1

[lawrencewxj]

Can you give me some suggestions? I was trapped by it. @piiswrong @crazy-cat

[edmBernard]

which version of mxnet did you use ?

[lawrencewxj]

mxnet.version = '0.11.0' @edmBernard

[edmBernard]

Depthwise convolution support has been merge the 16 August #7393
Maybe noob question but are you sure depthwise convolution is faster than regular convolution for training ? (I don't find comparison)

[lawrencewxj]

I remember that i install mxnet after the 16 August. And i calculated the mult-adds in theory as follow:
input_feature_map: DDC
filter: KKC
filter number: N
standard convolution: DDCNKK
depthwise convolution: DDCKK + CNDD
ratio = (DDCKK + CNDD)/(DDCNKK) = 1/N + 1/k^2
In VGG16, there are 13 convolution layers, I calculate the 13 layer that standard convolutions cost 15346.63 millions mult-adds and depthwise convolutions cost only 1786.92 millions mult-adds.
I really wanna figure out if there is something wrong. thank you very much.
@edmBernard

[leleamol]

Proposed Labels: "Performance", "CUDA", "Question"

[QueensGambit]

Hi @lawrencewxj @edmBernard
For a certain model architecture which heavily uses depthwise separable convolutions, I achieved a 1.4x speed-up on GPU ( MXNET-cu10 1.4.1,CUDA 10.0, cuDNN v7.5.1.10), but a 3x speed-up on CPU (MXNET-mkl 1.4.1).
The main reason for this is because grouped convolutions cause memory fraction and are naturally not well suited for GPUs.

The following paper conducted an experiment on this (see Table 2).
ShuffleNet V2: Practical Guidelines for EfficientCNN Architecture Design (Ma et al., 2018)

• https://arxiv.org/pdf/1807.11164.pdf

Recent cuDNN-Versions are improving the performance for group-convolutions:

• https://docs.nvidia.com/deeplearning/sdk/pdf/cuDNN-Release-Notes.pdf

It could be good to verify that MXNet makes use of all these recent optimizations.

Best,
~QueensGambit