Question about implementation of gradient compression strategies

[shuo-ouyang]

Hi guys, I am currently studying gradient compression for distributed communication reduction. I found that some strategies have been implemented in MXNet, such as Signum and 2bit quantization. However, such methods are implemented at different levels. Signum is an optimizer while 2bit quantization only works in kvstore. So I am confused what is a best way to implement these gradient compression strategies? How do we determine where (python frontend or kvstore or others) to implement these?

[shuo-ouyang]

@mxnet-label-bot add [Question]

[szhengac]

Recently, we provide a gradient compression api based on BytePS. See bytedance/byteps#225 and https://github.com/bytedance/byteps/blob/master/docs/gradient-compression.md. Currently, it supports 1-bit, top-k, random-k, and dithering. The example of training on ImageNet with mxnet is provided in https://github.com/bytedance/byteps/blob/master/example/mxnet/train_gluon_imagenet_byteps_gc.py

[shuo-ouyang]

Thanks very much for your response. It looks like that byteps#225 is a really good design and implementation of various gradient compression strategies.