Detected call of lr_scheduler.step() before optimizer.step()

[digital-idiot]

• pytorch-lightning version: 1.6.4

---

When tuning the learning_rate using lr_find, I get:

UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate

Related Issue: #5587
Code snippet is given below:

def tune_lr(
        model: LightningModule,
        tuning_params: Dict[str, Any] = None,
        trainer_args: Dict[str, Any] = None
):
    if trainer_args is None:
        trainer_args = dict()
    if tuning_params is None:
        tuning_params = dict()
    dummy_trainer = Trainer(
        **trainer_args
    )
    lr_finder = dummy_trainer.tuner.lr_find(model=model, **tuning_params)
    lr = lr_finder.suggestion()
    del dummy_trainer
    return lr

net.hparams.lr = tune_lr(
    model=net,
    tuning_params={
        "mode": "exponential",
        "datamodule": data_module,
        "min_lr": 1e-08,
        "max_lr": 1.0
    },
    trainer_args={
        "callbacks": [
            StochasticWeightAveraging(swa_lrs=1e-2),
            EarlyStopping(
                monitor="Validation-Mean_Loss",
                mode="min",
                patience=10,
                strict=True,
                check_finite=True,
                min_delta=1e-3,
                check_on_train_epoch_end=False,
            )
        ],
        "accumulate_grad_batches": 1,
        "check_val_every_n_epoch": 10,
        "num_sanity_val_steps": 0,
        "detect_anomaly": False,
        "log_every_n_steps": 1,
        "enable_progress_bar": True,
        "precision": 16,
        "sync_batchnorm": False,
        "enable_model_summary": False,
        "max_epochs": max_epochs,
        "accelerator": "gpu",
        "devices": -1
    }
)

[akihironitta]

@digital-idiot Can I see the trainer args as well?

[digital-idiot]

@akihironitta
Do you mean this?:

trainer args

    trainer_args={
        "callbacks": [
            StochasticWeightAveraging(swa_lrs=1e-2),
            EarlyStopping(
                monitor="Validation-Mean_Loss",
                mode="min",
                patience=10,
                strict=True,
                check_finite=True,
                min_delta=1e-3,
                check_on_train_epoch_end=False,
            )
        ],
        "accumulate_grad_batches": 1,
        "check_val_every_n_epoch": 10,
        "num_sanity_val_steps": 0,
        "detect_anomaly": False,
        "log_every_n_steps": 1,
        "enable_progress_bar": True,
        "precision": 16,
        "sync_batchnorm": False,
        "enable_model_summary": False,
        "max_epochs": max_epochs,
        "accelerator": "gpu",
        "devices": -1
    }

it is already there in the code snippet.

[akihironitta]

@digital-idiot Oh I'm sorry I somehow missed it 😅

---

I think it's due to #5558. Could you quickly check if precision=32 will resolve your issue?

[digital-idiot]

I think it's due to #5558. Could you quickly check if precision=32 will resolve your issue?

@akihironitta you are right, lr_find works as intended when precision=32. Thank you for helping me debugging this. I will use precision=32 to find the initial learning rate untill #5558 is fixed.