How does the accumulate_grad_batches affect the lr_scheduler update frequency?

[rentainhe]

It seems when I set accumulate_grad_batches larger than 1, it will affect my lr_scheduler update frequency

I have some problems with accumulate_grad_batches, it will auto scale the total steps in training.

example
When I set accumulate_grad_batches=16, the total steps equals to 4310 and when I change accumulate_grad_batches to 32, the total steps changed to 8620

[rohitgr7]

what do you mean by total steps here? ideally in the codebase schedulers follow their corresponding optimizers. can you share a minimal example? https://colab.research.google.com/drive/1HvWVVTK8j2Nj52qU4Q4YCyzOm0_aLQF3?usp=sharing#scrollTo=4Dk6Ykv8lI7X

[rentainhe]

oh, I figure it out, but I still have another question. I want to run validation every n steps, but I find out lightning removed run_evaluation function.
Before that, I can use this callbacks to run evaluation every n steps

class ValEveryNSteps(pl.Callback):
    def __init__(self, every_n_step):
        self.every_n_step = every_n_step

    def on_batch_end(self, trainer, pl_module):
        if trainer.global_step % self.every_n_step == 0 and trainer.global_step != 0:
            trainer.run_evaluation(test_mode=False)

But in the latest version it won't work because of there is no run_evaluation in Lightning

Does lightning have some other function to support this?

[rohitgr7]

try val_check_interval argument in Trainer for this. https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html#val-check-interval

[rentainhe]

emmmm, I think val_check_interval not work for this situation

• val_check_interval only works in one epoch, if one epoch has 500 steps and val_check_interval=200, it only run evaluation in steps=200 and steps=400 and the same in next epoch, but what I really want is, if total_steps = 500 in one epoch, and val_check_interval=200, it can run evaluation in Epoch 0 : steps 200, steps 400 and Epoch 1 : steps 100

I want it to be connected to the global steps not training steps

[tchaton]

This won't work, but you could choose a value which makes every validation equidistant from each other. In this example, val_check_interval=100 would work quite nicely.

Calling validation outside of Lightning scope won't always work nicely.

[rentainhe]

This won't work, but you could choose a value which makes every validation equidistant from each other. In this example, val_check_interval=100 would work quite nicely.

Calling validation outside of Lightning scope won't always work nicely.

All right 😭