move_metrics_to_cpu does not work

[MeteorsHub]

I used torchmetrics.AveragePrecision as the metric in PL module. This metric takes a lot of gpu memory as it save all data in buffer.
So I wanted to move the metric to cpu and set move_metrics_to_cpu True in PL Trainer. But the metric's buffer was still on gpu.
Any ideas of this?

[justusschock]

I think move_metrics_to_cpu does not move the state of the metric modules but just the logged values.

cc @tchaton for confirmation.

@SkafteNicki @tchaton should we maybe extend this? I see that this can be misleading.

@MeteorsHub The issue we do it that way is mainly to avoid having the same thing in memory twice. But I think especially for metrics like AP the only option would be to manually move it to cpu (since this will be moved to gpu together with the model). This will introduce some synchronization points though and might slow down your training!

[tchaton]

Yes, exactly. It would just move the logged values on CPU.
We are currently refactoring the Logging API. We can think about it once this is done. ETA: 1-2 weeks from now.