Why raising UserWarning is logging from on_validation_epoch_end and only in rank0

[pengzhenghao]

I am trying to log from on_validation_epoch_end and only in rank 0:

    def on_validation_epoch_end(self):
        # get some data...
        if self.trainer.is_global_zero:
            # do some eval
            self.log_dict(result, rank_zero_only=True)

If I understand correctly, I am not suppose to set rank_zero_only=True with sync_dist=True together because they are conflicting.

However, I got the below warning message flushing the screen.

PossibleUserWarning: It is recommended to use `self.log('eval/XXX', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices

So would that be a issue if I don't do sync_dist=True? If that's OK, then anyone know how to suppress the UserWarning? Thanks！

[ZhaoGe666]

I met exactly the same issue as yours, all we can do by far is to suppress it. Here is an example:
#10182 (comment)