how to use one cyle learning rate?

[talhaanwarch]

My one cycle learning rate curve goes above but did not goes down, i mean cycle is not complete.
lr=1e-4

 def configure_optimizers(self):
        opt=torch.optim.AdamW(params=self.parameters(),lr=self.lr )
        scheduler=OneCycleLR(opt,max_lr=1e-2,epochs=50,steps_per_epoch=len(self.train_split)//self.batch_size)
        return {'optimizer': opt,'lr_scheduler':scheduler}

here is learning rate monitor
lr_monitor = LearningRateMonitor(logging_interval='epoch')
code for trainer

trainer = Trainer(max_epochs=50, auto_lr_find=False, auto_scale_batch_size=False,
                  deterministic=True,
                  gpus=-1,precision=16,
                  accumulate_grad_batches=4,
                  stochastic_weight_avg=False,
                  progress_bar_refresh_rate=0,
                  log_every_n_steps=10,
                  callbacks=[lr_monitor],
                  logger=logger
                  )

Here is curve in logger

[grudloff]

Take a look at issue #9630

[tchaton]

Dear @talhaanwarch,

You should do the following. Your scheduler is being executed on epoch end only.

 def configure_optimizers(self):
        opt=torch.optim.AdamW(params=self.parameters(),lr=self.lr )
        scheduler=OneCycleLR(opt,max_lr=1e-2,epochs=50,steps_per_epoch=len(self.train_split)//self.batch_size)
        return [opt], [{"scheduler": scheduler, "interval": "step"}]

[talhaanwarch]

i think it should be
return {'optimizer': opt,"lr_scheduler": scheduler, "internval": "step"}

[rohitgr7]

none of the above will work.. 😅
just do

def configure_optimizers(self):
   opt=torch.optim.AdamW(params=self.parameters(),lr=self.lr )
    scheduler=OneCycleLR(opt,max_lr=1e-2,epochs=50,steps_per_epoch=len(self.train_split)//self.batch_size)
    lr_scheduler = {'scheduler': scheduler, 'interval': 'step'}
    return {'optimizer': opt, 'lr_scheduler': lr_scheduler}

you need to return it properly in a structured way to make sure it can be easily interpreted internally to configure which scheduler belongs to which optimizer and what all properties need to be set for schedulers. Check docs for more info: https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.html#configure-optimizers

[talhaanwarch]

it still showing me output like this

[rohitgr7]

you have set accumulate_grad_batches=4
steps_per_epoch means total optimization steps per epoch.
try setting steps_per_epoch=len(self.train_split)//self.batch_size//4

if it still doesn't work, a reproducible minimal example would be helpful to check the issue :)

[talhaanwarch]

@rohitgr7 @tchaton i am still unable to fix it. I have create a complete gist, hope it will help
https://gist.github.com/talhaanwarch/52e2bc8118e2fb411277d81c04970ae7

[tchaton]

Hey @rohitgr7 @talhaanwarch

def test_lr_monitor_one_cycle_lr(tmpdir):

    class TestModel(BoringModel):

        def configure_optimizers(self):
            optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
            lr_scheduler = optim.lr_scheduler.OneCycleLR(
                optimizer,
                max_lr=0.01,
                steps_per_epoch=64,
                epochs=10,
                three_phase=True
            )
            return [optimizer], [{'scheduler': lr_scheduler, 'interval': 'step'}]

    model = TestModel()
    lr_monitor = LearningRateMonitor(logging_interval="step")
    trainer = Trainer(
        default_root_dir=tmpdir,
        max_epochs=10,
        callbacks=[lr_monitor],
    )
    trainer.fit(model)
    print(f"tensorboard --logdir {tmpdir}")
    breakpoint()

This seems to work fine.

[tchaton]

And your steps_per_epoch should be len(self.train_set)//(self.batch_size * 8).