Switching model during training

[farahFif]

Hello,

I am experiencing knowledge distillation in Incremental Learning. Basically , at each phase I init a new model and train it on the current data as a student model , and use the old model ( trained on previous phase) as a teacher. So far , there's no error in the code but the problem is that the loss never decrease after each phase. Also, is there a way in PL to init new optimizer in on_train_epoch_start ?

Here's example of the code :

def on_train_epoch_start(self) :
       if self.new_phase:
           self.old_backbone = copy.deepcopy(self.backbone)
           self.old_head = copy.deepcopy(self.head)
           self.backbone = None
           self.head = None
           for p in self.old_backbone.parameters():
               p.requires_grad = False
           for p in self.old_head.parameters():
               p.requires_grad = False
           self.backbone = create_backbone(model_name=self.params.backbone_name,
                                            **self.params.backbone_params)
            # create LINEAR head
           self.params.head_params['in_features'] = self.backbone.num_features
           self.head = HEADS.get(self.params.head_name)(**self.params.head_params)
           self.backbone = self.backbone.to(self.device)
           self.head = self.head.to(self.device)
           self.old_backbone.eval()
           self.old_head.eval()
           self.backbone.train()
           self.head.train()