Separate Trainer for train() and test()?

[davzaman]

Python: 3.9.12
Pytorch: '1.11.0+cu102'
Pytorch-lightning: 1.6.1

The docs for evaluation say:

It is recommended to test with Trainer(devices=1) since distributed strategies such as DDP use DistributedSampler internally, which replicates some samples to make sure all devices have same batch size in case of uneven inputs. This is helpful to make sure benchmarking for research papers is done the right way.

Similarly I get a warning during runtime:

During `trainer.test()`, it is recommended to use `Trainer(devices=1)` to ensure each sample/batch gets evaluated exactly once. Otherwise, multi-device settings use `DistributedSampler` that replicates some samples to make sure all devices have same batch size in case of uneven inputs.

I would like to use DDP for training on 4 GPUs, does this mean I create an entirely new Trainer for just val/test?

My train Trainer:

trainer = pl.Trainer(
            max_epochs=self.max_epochs,
            logger=logger,
            num_nodes=self.num_nodes,
            # use 1 processes if on cpu
            devices=self.num_gpus if self.num_gpus else 1,
            accelerator="gpu" if self.num_gpus else "cpu",
            strategy=DDPPlugin(find_unused_parameters=False)
            if self.num_gpus > 1
            else None,
            enable_checkpointing=False,
            callbacks=callbacks,
            profiler="simple",  # or "advanced" which is more granular
            fast_dev_run=self.fast_dev_run,  # For debugging
        )
trainer.fit(model, datamodule)

If I want to then run testing would I then do something like:

test_trainer= pl.Trainer(
            max_epochs=self.max_epochs,
            logger=logger,
            num_nodes=self.num_nodes,
            # use 1 device for test
            devices=1,
            accelerator="gpu" if self.num_gpus else "cpu",
            strategy=DDPPlugin(find_unused_parameters=False)
            if self.num_gpus > 1
            else None,
            enable_checkpointing=False,
            callbacks=callbacks,
            profiler="simple",  # or "advanced" which is more granular
            fast_dev_run=self.fast_dev_run,  # For debugging
        )
test_trainer.test(model, test_dataloader)

Am I understanding correctly? I don't think it's possible to set the number of devices dynamically since there's no setter defined for the num_devices property in the Trainer class. I'm not sure if it makes sense to make a separate trainer, so I feel a bit confused.

[rohitgr7]

yes, if you care about the accuracy of the test metrics and don't want to include extra samples, this is the only way. Not just a separate trainer, if using DDP, you should create a separate script all together else the whole script, along with the trainer.test call will be launched on each device during trainer.fit call.

[davzaman]

Gotcha, so there's no workaround if I want to do distributed training? DP, all forms of DDP(1,2), and Horovod all use the DistributedSampler and would suffer the same problem/require the same solution, correct?

[rohitgr7]

yes.
similar discussion happened here if you want to learn more: #12862