ReduceLROnPlateau with horovod

[BigodF]

When I use torch.optim.lr_scheduler.ReduceLROnPlateau with horovod to train my net, horovod will check weather my lr_scheduler is pytorch_lightning.utilities.types ._LRScheduler or not, just like following (HorovodStrategy.set_up function in pytorch_lightning.strategies.horovod):

        lr_scheduler_configs = self.lr_scheduler_configs
        for config in lr_scheduler_configs:
            scheduler = config.scheduler
            assert isinstance(scheduler, _LRScheduler))
            scheduler.base_lrs = [lr * self.world_size for lr in scheduler.base_lrs]

But，ReduceLROnPlateau does not inherent _torch.optim.lr_scheduler.LRScheduler.
Anyone knows how to use ReduceLROnPlateau with horovod?

this is my optimizer and lr_scheduler in LightningModule's configure_optimizers function:

        optimizer, (sch, sch_val) = get_opt_sch_bertfinetune(self, conf, self.args, None, total_steps=total_steps, val_metric_mode='min')
        # linear warm-up lr scheduler
        sch = {
            'scheduler': sch, # torch.optim.lr_schedule._LRScheduler, is ok with horovod assert check
            'interval': 'step'
        }
        # ReduceLROnPlateau
        sch_val = {
            'scheduler': sch_val, #  torch.optim.lr_scheduler.ReduceLROnPlateau
            'monitor': self.val_metric_name,
            'frequency': 1,
        }

        return [optimizer], [sch, sch_val]

[akihironitta]

It seems the assertion was added in #13570 for improving typing coverage in the codebase as part of #13445, and I believe there isn't any limitation on learning rate scheduler kinds with the horovod strategy.

(I was reading it wrong and just realised that _LRScheduler is pytorch_lightning.utilities.types._LRScheduler but not torch.optim.lr_scheduler._LRScheduler.)

But，ReduceLROnPlateau does not inherent _torch.optim.lr_scheduler.LRScheduler.
Anyone knows how to use ReduceLROnPlateau with horovod?

Do you see any issue with it? You should already be able to use ReduceLROnPlateau with the horovod strategy. The assertion only checks if your lr scheduler object has the following protocol. (so it can become True even if your lr scheduler doesn't subclass _LRScheduler:
https://github.com/Lightning-AI/lightning/blob/fe9e5d55bf7991ba36b76d6adae9075b93dfcaa0/src/pytorch_lightning/utilities/types.py#L105-L113

[BigodF]

Thanks for your repliy!

A small code tested on my PC, ReduceLROnPlateau of torch is not an instance of pytorch_lightning.utilities.types._LRScheduler

'''
envs:
pytorch-lightning       1.7.0
pytorch-pretrained-bert 0.6.2
torch                   1.11.0
torchaudio              0.11.0
torchmetrics            0.9.3
torchvision             0.12.0
'''


import torch
from torch.optim import AdamW

from pytorch_lightning.utilities.types import _LRScheduler
from torch.optim.lr_scheduler import ReduceLROnPlateau, LinearLR


opt = AdamW([torch.zeros((2, 3), requires_grad=True)])
sch0 = LinearLR(opt)
sch1 = ReduceLROnPlateau(opt)
print(isinstance(sch0, _LRScheduler))
print(isinstance(sch1, _LRScheduler))

### output
True
False

I don't know how isinstance works.
I notice that a difference between torch._LRScheduler and torch.ReduceLROnPlateau is _LRScheduler has a attribute named "base_lrs", while _LRScheduler don't. This difference alse causes errors in HorovodStrategy.set_up function in pytorch_lightning.strategies.horovod

        lr_scheduler_configs = self.lr_scheduler_configs
        for config in lr_scheduler_configs:
            scheduler = config.scheduler
            assert isinstance(scheduler, _LRScheduler))
            scheduler.base_lrs = [lr * self.world_size for lr in scheduler.base_lrs] # ReduceLROnPlateau dont have base_lrs

[akihironitta]

@BigodF Sorry for the delay. Let me look into it!

[akihironitta]

@BigodF Just realised that you've cross-posted this!

---

For anyone seeing this discussion, please refer to #14236.