SyncBatchNorm problems

[kaleidoscopical]

It runs into a fail of asnumpy() when I simply replace BatchNorm() to contrib.SyncBatchNorm(). Could anyone explain how to use the new function?

[haojin2]

@zhanghang1989

[zhanghang1989]

The SyncBN is implemented in a blocking way. Please use DataParallelModel in https://github.com/dmlc/gluon-cv/blob/master/gluoncv/utils/parallel.py#L11 for debugging, printing or asnumpy.

[Roshrini]

@nswamy Can you please add label: Question

[kaleidoscopical]

@zhanghang1989 It seems unsuitable for non-gluon users. Any suggestions?

[zhanghang1989]

I am not familiar with Symbol API. Can someone help this?

[safrooze]

@kaleidoscopical I haven't personally used SyncBatchNorm, but the operator is available in both nd and sym format. What are you using?

[kaleidoscopical]

@safrooze I have tried both of them. While the nd version seems to perform similarly as the original BatchNorm, the sym version fails when calling asnumpy().

[zhanghang1989]

If using asnumpy() or print() please use DataParallelModel https://github.com/dmlc/gluon-cv/blob/master/gluoncv/utils/parallel.py#L11

Training example of using SyncBatchNorm can be found at https://github.com/dmlc/gluon-cv/blob/master/scripts/segmentation/train.py

[zhanghang1989]

If using standard BatchNorm works, change it to SyncBatchNorm and set num_device correctly, it will work as well.

[piyushghai]

@kaleidoscopical Were you able to get a suitable answer to your question ?

[cctgem]

@kaleidoscopical Did you find a way to use syncbn by symbol api?

[zhanghang1989]

You can't call asnumpy() or do print() with SyncBatchNorm. Please use standard BatchNorm for debugging and change it to SyncBatchNorm with correct num_device after debugging.

[kaleidoscopical]

According to a quick test, the described failure in this issue has disappeared in the newest version of MXNet. Further report about efficiency and performance will be updated if time is available.

@zhanghang1989 Thanks for your kind help. The asnumpy() function will be eventually called in the metric evaluation part, e.g. mx.metric.CrossEntropy(). This is the place where the fail happened as I described in the beginning of this issue. I am not sure what has been changed in the newest version, and whether the SyncBatchNorm really is executed because of ignorable speed drop (great if it is really executed).

[tranvanhoa533]

@kaleidoscopical Which version of mxnet did you use ? I use mxnet version 1.3.1 but It still runs into a fail of asnumpy().

[kaleidoscopical]

@tranvanhoa533 which error message it shows?

Hi @zhanghang1989 !
I think it still has a problem when training with multiple precision, leading to

mxnet.base.MXNetError: [17:38:53] include/mxnet/././tensor_blob.h:236: Check failed: mshadow::DataType<DType>::kFlag == type_flag_ TBlob.get_with_shape: data type do not match specified type.Expected: 2 v.s. given 0

This error message disappears when adding an explicit cast of type right before (16 to 32) and after (32 to 16) mx.sym.contrib.SyncBatchNorm using mx.sym.cast.

It is tedious and sacrifices much more memory and speed. Any suggestion to solve it?

[zhanghang1989]

SyncBN does not support fp16 training yet.

[cccorn]

I met the same problem with mxnet version 1.5.0, and I found a solution from L_xiaoming in https://discuss.gluon.ai/t/topic/7842

The solution is to specify the parameter 'key' in the SyncBatchNorm layer, and you can just use the same string as the layer's name.

But I don't know why it works.

[pengwangucla]

I tried with all the solutions, and neither of them is working. It is really weird When I only use

I met the same problem with mxnet version 1.5.0, and I found a solution from L_xiaoming in https://discuss.gluon.ai/t/topic/7842

The solution is to specify the parameter 'key' in the SyncBatchNorm layer, and you can just use the same string as the layer's name.

But I don't know why it works.

The problem still exists, after given key. There is no matching error, however, the program running is jammed and never continue. The max number of syncbn I can use is 2. Don't know why

[zhanghang1989]

Gluon-CV is using syncBN for training all segmentation model and YoloV3. Please take a look at how to use it.