Access model weights during training

[vokcow]

Hello and thanks for this amazing library.

When training models I find useful checking which parameters change after backpropagation.

In vanilla Pytorch I would do something like

parameters_before = model.parameters()

...

loss.backward()
optimizer.step()
...
parameters_after = model.parameters()

[print( (parameter_before != parameter_after).mean() ) for parameter_before, parameter_after
in zip(parameters_before, parameters_after)]

And I should see a bunch of ones. Some zeroes can indicate branches of the network where the gradients are not flowing back.

I was thinking of a callback that does the same within PL. After reading the docs, I have came up with this:

class CheckParamsUpdatedCallback(Callback):
    def __init__(self):
        self.prms_for_check = dict.fromkeys(('before', 'after'))

    def on_train_epoch_start(self, trainer, pl_module):
        self.prms_for_check['before'] = pl_module.parameters()

    def on_validation_epoch_start(self, trainer, pl_module):
        self.prms_for_check['after'] = pl_module.parameters()
        for v in self.prms_for_check.values():
            if v is None:  # handles validation sanity check
                return None
        print([(pbef != paft).float().mean().item()
               for pbef, paft in zip(self.prms_for_check['before'], self.prms_for_check['after'])])

Nevertheless, I believe that pl_module.parameters() is not returning the actual parameters used in the optimization, as I am always getting zeroes printed out, even if other indicators (loss/metrics) suggest the model is being trained well.

How can I access the model parameters during training process?

Thanks a lot and best wishes,

Victor

[rohitgr7]

And I should see a bunch of ones.

you should see or you did see. I think the parameters saved in the dict are references to the same object hence you can't see any difference there.

pl_module.parameters()

also, this is a generator.

you can deepcopy the state_dict instead to verify it.

[vokcow]

Hi rohitgr7, thanks for your answer.

Indeed, I have checked what I was doing in other projects and I was copying the paremeters to a list, rather than just picking the generator, thanks for pointing that out.

This works as expected now:

class CheckParamsUpdatedCallback(Callback):
    def __init__(self):
        self.prms_for_check = dict.fromkeys(('before', 'after'))

    def on_train_epoch_start(self, trainer, pl_module):
        self.prms_for_check['before'] = [p.clone() for p in pl_module.parameters()]

    def on_validation_epoch_start(self, trainer, pl_module):
        self.prms_for_check['after'] = [p.clone() for p in pl_module.parameters()]
        for v in self.prms_for_check.values():    # handles validation sanity check
            if v is None:
                return None

        print([torch.not_equal(pbef, paft).mean()
               for pbef, paft in zip(self.prms_for_check['before'], self.prms_for_check['after'])])

[rohitgr7]

great!