How to run an additional EvaluationEpochLoop ?

[pclucas14]

Hi,

I'm working in a Continual Learning-like setting, where you have multiple phases, and for each phase you receive a new dataset. In this setting, at every phase I have a training and validation set, just like in regular training. So far my solution has been to replace the FitLoop with a custom one

    class MetaFitLoop(FitLoop):
        # Wraps the regular fit loop to handle data dumps over time
        def __init__(self, trainer, cfg):
            super().__init__(max_epochs=cfg.num_epochs)

            self.cfg = cfg
            self.trainer = trainer
            self.trainer.current_round = 0

        def run(self):
            while self.trainer.current_round < self.cfg.num_rounds:
                super(FitLoop, self).run()
                _reset_progress(self.epoch_progress)
                self.trainer.current_round += 1

Everything seems to work fine, however I want to do one more thing. Whenever I start a new round, I wand to compute the performance of my current model on the training data (this is to mimic online evaluation over a stream of data). In other words, I want to do an EvaluationEpochLoop on the training data. However, it's important that in this loop, the optimizers / schedulers are not used, and that it's run in inference mode (so I use less memory). The solution I found seems pretty hacky, and I'd like to make it better.

I added this in my MetaFitLoop constructor

self.eval_loop = EvaluationEpochLoop()
self.train_ds_fetcher = DataFetcher()

And this in run()

train_ds = self.trainer.datamodule.train_dataloader()
... 
while self.trainer.current_round < self.cfg.num_rounds:
        with torch.no_grad():
            self.trainer.is_online = True         # So that I know to use a special `online_step` rather than the `training_step`
            self.eval_loop.run(self.train_ds_fetcher, len(train_dl), OrderedDict())
            self.trainer.is_online = False
...

How can I make this less hacky ? Any suggestions ?

Thanks :)

[pclucas14]

Also, one thing that's not good about this approach is that trainer/global_step resets when I run super(FitLoop, self).run()