What is trainer/global_step in wandb logging?

[edmcman]

What does the value trainer/global_step mean in wandb logging?

I am not using distributed training, and only looking during the first epoch. I would think that this is the number of batches processed, but it doesn't seem like it. What is it supposed to be? For example, the latest logged value is 1199. But my progress bar shows the current batch is 5000. I thought maybe the wandb logger is lagging a bit, but I doubt it is lagging that much. So where is this factor of about four difference coming from?

To make things concrete, I took these two screenshots at roughly the same time:

[edmcman]

Now that it's past the first epoch, it's clear that it wasn't the logger "falling behind". The second epoch started at approximately global step 7049, and according to the progress bar there are 27907 batches per epoch. So, again, a factor of four. Is this a bug?

[profPlum]

I figured out what trainer/global_step is and it seems to be correct (see my answer).
That being said it is still possible that there is/was a bug with the progress bar idk.

[N-Demir]

Yeah I really don't get how wandb comes up with its steps either

[N-Demir]

I think it's probably a better question for wandb. But I found this in their docs that I think explains it: https://docs.wandb.ai/guides/integrations/add-wandb-to-any-library/#prevent-x-axis-misalignments

[ringohoffman]

lightning's WandbLogger automatically creates trainer/global_step and configures it as the global x-axis:

pytorch-lightning/src/lightning/pytorch/loggers/wandb.py

Lines 409 to 414 in 4eda5a0

	# define default x-axis
	if isinstance(self._experiment, (Run, RunDisabled)) and getattr(
	self._experiment, "define_metric", None
	):
	self._experiment.define_metric("trainer/global_step")
	self._experiment.define_metric("*", step_metric="trainer/global_step", step_sync=True)

This is actually how I found out this was even possible LOL. I am planning a PR to default to tensorboard's global_step if sync_tensorboard=True.

Truly though, I would actually just prefer that lightning did nothing at all.

[ringohoffman]

Also, this is where they are setting it, in WandbLogger.log_metrics:

pytorch-lightning/src/lightning/pytorch/loggers/wandb.py

Lines 431 to 440 in 1f5add3

	@override
	@rank_zero_only
	def log_metrics(self, metrics: Mapping[str, float], step: Optional[int] = None) -> None:
	assert rank_zero_only.rank == 0, "experiment tried to log from global_rank != 0"
	metrics = _add_prefix(metrics, self._prefix, self.LOGGER_JOIN_CHAR)
	if step is not None:
	self.experiment.log(dict(metrics, **{"trainer/global_step": step}))
	else:
	self.experiment.log(metrics)

[profPlum]

@ringohoffman No! It is important that W&B is logging this because it is the only reliable step counter for W&B and possibly other logging api too. See my answer.

[profPlum]

I figured out what trainer/global_step is! It is exactly as they say: the total number of training optimization steps taken. The problem isn't with PytorchLightning the problem is with W&B's "step" which is probably why Lightning created their own step metric. Basically I calculated the number of steps that I was supposed to have in my training and it exactly matched trainer/global_step whereas W&B step was way off base (1/10 of the actual number of steps). In addition I changed my code in a way that I calculated shouldn't change the total number of optimization steps, despite this it halved the number of W&B "steps" but kept trainer/global_step constant!

It is possible that Lightning internally calls W&B in some weird way which mangles the W&B's step counter. But regardless trainer/global_step should be trusted and W&B step shouldn't this is in pytorch_lightning==2.4.0 and wandb==0.19.11

P.S. I think this discussion is on the same topic: #8007
P.P.S. It is still possible there is/was a bug with the progress bar, idk.