ModelCheckpointCallback.best_model_score is None after distributed training

[hikushalhere]

For single GPU training, ModelCheckpointCallback.best_model_score is reported correctly along with ModelCheckpointCallback.best_model_path. However, when I use DDP to do distributed training, ModelCheckpointCallback.best_model_score is still None after training as if it was not recorded. However, ModelCheckpointCallback.best_model_path is recorded and valid.

Is this expected? What might be wrong given this observation?

Some more information:

1. I am using PL 1.6.0 with PyTorch 1.10.0
2. I am using MLFlow for logging and tracking. MLFlow 1.26.1
3. I am using strategy="ddp"

[Abhishaike]

Running into this exact issue, any update?

[tshu-w]

@Abhishaike I suggest you open an issue with a reproducible script using the BoringModel.