Clear optimizer state in batch.end.callback

[KhurramPirov]

Description

I want to clear SGD optimizer state during some steps of training.
Package used:
Python 3.7

What have you tried to solve it?

1. I tried to delete opt and then create new, inside the function batch callback

[lanking520]

@KhurramPirov Could you please provide code to reproduce?

[KhurramPirov]

@lanking520, @linhaibin
this is the step where I want to clear optimizer state, I want to make momentums of 'fc_7_weight' equal to zero in order to start model.fit with new parameters.

`def _batch_callback(param):
avr_loss.append(param.eval_metric.get_name_value()[1][1])
avr_acc.append(param.eval_metric.get_name_value()[0][1])

  if len(avr_loss) == 20:
      avloss = np.mean(avr_loss)
      avacc = np.mean(avr_acc)
      sw.add_scalar(
          tag='loss',
          value=avloss,
          global_step=global_step[0]
      )
      sw.add_scalar(
          tag='accuracy',
          value=avacc,
          global_step=global_step[0]
      )
      avr_loss.clear()
      avr_acc.clear()

  global_step[0] += 1
  mbatch = global_step[0]
  
  for step in lr_steps:
    if mbatch == step:
      opt.lr *= 0.1
      print('lr change to', opt.lr)
      break

  _cb(param)

  if mbatch % 1000 == 0:
      print('lr-batch-epoch:', opt.lr, param.nbatch, param.epoch)
      model.set_params(arg_params, aux_params)
      opt.create_state_multi_precision('fc7_weight', arg_params['fc7_weight'])`