train_time_interval checkpoint and metric value

[DA-L3]

Hi,

I am currently trying to save a checkpoint after some period of time using train_time_interval
PT Lightning version: 1.6.4

mc = ModelCheckpoint(
            dirpath=checkpoint_dir,
            filename='{step}',
            train_time_interval=timedelta(**time_delta),
            save_last=True,
        )

I will call this checkpoint time.
Further I have another checkpoint callback monitoring my metric named val_ca, I call it monitor.
But I want to resume from my time checkpoint.

In the validation_step loop, I use self.log('val_ca', value, on_step=False, on_epoch=True, logger=True). I only validate this value at the end of an epoch.
But since the time checkpoint might save in between epochs, I would still like to carry that value into the checkpoint because resuming from time yields:
ModelCheckpoint(monitor='val_ca') could not find the monitored key in the returned metrics: ['epoch', 'step']. HINT: Did you call log('val_ca', value) in the LightningModule?
when the monitor checkpoint is considered.

Even saving at a time after at least one epoch finished (hence the self.log for the val_ca was executed at least one time), did not resolve the issue.

Does anyone have any ideas how I can carry this val_ca?

Thanks!