Best practices for customize the evaluation

[tshu-w]

Hi, I need a custom evaluation without val_dataloaders or validation_step, just the model and a custom function.
Somewhat similar to the following code, which uses SentEval

Currently, I define empty step and dataloader and write my own EvaluationLoop:

    def before_run(self):
        fit_loop = self.trainer.fit_loop
        epoch_loop = fit_loop.epoch_loop
        epoch_loop.connect(val_loop=EvaluationLoop(verbose=False))
        fit_loop.connect(epoch_loop=epoch_loop)
        self.trainer.fit_loop = fit_loop

        self.trainer.test_loop = EvaluationLoop()

        self.model.validation_step = self.model.test_step = none_fun
        self.datamodule.val_dataloader = empty_dataloader
        self.datamodule.test_dataloader = empty_dataloader

However since the evaluation logic doesn't require dataloader etc, it's somewhat hacky and I'm struggling with the logic when training with multiple GPUs. I was wondering if there is a better way to customize the evaluation.