Exponential Moving Average/Polyak Averaging in MXNet

[anirudhacharya]

How can I implement Polyak averaging of as part of my network, in the wavenet model - https://arxiv.org/abs/1711.10433

I am not sure it can be implemented as an optimizer, it is more like a post-processing step on the weights after the optimizer update.

Tensorflow has it here - https://www.tensorflow.org/api_docs/python/tf/train/ExponentialMovingAverage

[anirudhacharya]

@mxnet-label-bot add [Feature Request, Question, Optimizer]

[Ishitori]

I have an implementation of Polyak averaging. Take a look here to find out if it is the same version they used in WaveNet: https://github.com/dmlc/gluon-nlp/blob/bidaf/scripts/question_answering/utils.py#L277

[anirudhacharya]

I will check. Thanks!

[sojiadeshina]

@safrooze also implemented Polyak Averaging a la wavenet and may be able to help.