How to access the strategy of the trainer #11272
-
Hi, I am trying to make my code invariant to the choice of strategies by being able to compute the global batch size which depends on the strategy. For example, for DDP it is The use case I can think of is using the global batch size to initialize the optimizer. trainer(num_nodes=1, gpus=2, strategy='ddp') # pass the strategy ddp for example
class MyLightningModule(pl.LightningModule):
@property
def global_batch_size(self) -> int:
if self.trainer.strategy is None:
return self.trainer.datamodule.train.loader.batch_size
elif self.trainer.strategy is DDPStrategy:
return self.trainer.num_nodes * self.trainer.gpus *\ # There might be a better way to compute the
self.trainer.datamodule.train.loader.batch_size # number of processes using the strategy
...
def configure_optimizers(self) -> Dict[Any, Any]:
optimizer, scheduler = hydra.utils.instantiate(
self.hparams.optimizer, model=self, batch_size=self.global_batch_size, _recursive_=False)
return {
'optimizer': optimizer,
'lr_scheduler': scheduler
} To do so, I would like to retrieve inside my Lightning module the strategy used by my trainer. I tried to find in the trainer code how to access the strategy and I found the property: # in pytorch_lightning.trainer.trainer.py
class Trainer(...):
...
@property
def strategy(self) -> Strategy:
return self._accelerator_connector.strategy However Weirdly, # in pytorch_lightning.trainer.connectors.accelerator_connector.py
class AcceleratorConnector(...):
def __init__(...):
...
self.strategy = self.final_strategy()
... Is it possible to access the strategy used for training? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 4 replies
-
just out of curiosity, what sort of scheduler/optimizer are you initializing using the global_batch_size? |
Beta Was this translation helpful? Give feedback.
just out of curiosity, what sort of scheduler/optimizer are you initializing using the global_batch_size?