self.global_step understanding

[ajinkyaambatwar]

I had a doubt about the keyword self.global_step
Does it represent the total number of batches seen so far or total number of samples seen so far.
For eg. If I have batch size of 32 and each epoch has 200 batches(total of 6400 samples). After 2 epochs, what should be the value of self.global_step?

Now I intend to use this concept for scheduling my LR using LambdaLR. I intend to update it after 20000(decay_step) steps. So is my lr_lbmd inside configure_optimizer right? Or should I just use
int(self.global_step/self.hparams["optimizer.decay_step])?

def lr_lbmd(_):
            return max(
                self.hparams["optimizer.lr_decay"]
                ** (
                    int(
                        self.global_step
                        * self.hparams["batch_size"]
                        / self.hparams["optimizer.decay_step"]
                    )
                ),
                lr_clip / self.hparams["optimizer.lr"],
            )

[awaelchli]

self.global_step represents the current index of training batch across all epochs. If you are using multi-gpu, important detail: it represents only the counting for one gpu, not all batches combined for all gpus!

global step is incremented after each training step is fully processed and optimizer has stepped and logger has logged.

For eg. If I have batch size of 32 and each epoch has 200 batches(total of 6400 samples). After 2 epochs, what should be the value of self.global_step

It will be 200 * 2 and it means batches 0 to (200 * 2 - 1) have been processed (total 400) and the next batch will be the index 400.

[awaelchli]

If you set interval="step" in the scheduler dict returned from the configure_optimizers() hook, it will call your lr_scheduler.step() function every global step.

[sevmag]

How does the global step behave for the validation period. I'm trying to log train_loss_step and val_loss_step with the WandbLogger . I tried to plot them on the same graph with global_step on the x-axis but for some reason the val_loss_step extends to far greater global_step values. I'm using log_every_n_steps=1 and val_check_interval=0.1 in the lightning Trainer. I have the suspicion that there is a different global_step used. For testing purposes I used 100 datasamples loaded with a batchsize 10.

[JohnHerry]

Is the global_step means the number of mini-batches, or the number of weight-update operations? what is the change when we set accumulate_grad larger then One?

[ShandyDrm]

https://lightning.ai/docs/pytorch/2.5.0/common/lightning_module.html#global-step

The number of optimizer steps taken (does not reset each epoch). This includes multiple optimizers (if enabled).

I assume that if for example accumulate_grad_batches is set to 4, then global_step will increase after 4 batches.

[stf976]

The value of global_step is the "number of optimizer steps taken (does not reset each epoch). This includes multiple optimizers (if enabled)."

(Source: https://lightning.ai/docs/pytorch/stable/common/lightning_module.html#global-step)

With automatic optimization (the default), there is one optimizer step after each training step.

With multiple optimizers, global_step increases accordingly.

If accumulate_grad_batches is used and set to e.g. 4, there is indeed only one global_step per 4 batches. This can be easily verified by logging the number of batches / training steps and global_step.

During validation, global_step does not increase. Again, this is confirmed by logging its value.

If you want to adjust the learning rate or checkpoint based on another metric, such as the total number of batches or samples, it might be easiest to log that metric yourself and monitor it in the learning rate scheduler or checkpoint callback.

The trainer already keeps track of the total number of batches. You could log it like this:

    def training_step(self, batch, batch_idx, dataloader_idx=0):
        loss = ...
        self.log('total_batch_idx', self.trainer.fit_loop.epoch_loop.total_batch_idx, on_step=True, on_epoch=False, logger=True)
        return loss

and access it in the lr scheduler config like this:

    def configure_optimizers(self):        
        optimizer = ...
        scheduler = ...

        lr_scheduler_config = {
            "scheduler": scheduler,
            "interval": "step",
            "frequency": 1,
            "monitor": "total_batch_idx",
        }

        return {
            'optimizer': optimizer,
            'lr_scheduler': lr_scheduler_config,
        }