Relationship between Distributed Training and Metric Calculations

[astrobdr]

I was under the impression that the only thing which must be done when logging metrics in a distributed setting (say ddp), is average the metric computed by each rank. I understood that this is done when doing pl.log with sync_dist=True.

What else should be done? Are there special cases for some metrics? And where is this handled in this library?

[justusschock]

Hey,
For most metrics simply averaging might produce a good estimate, but it does not provide the correct way to aggregate metrics in a distributed setting.
torchmetrics handles this under the hood (If don't disable it) to provide correct results.

To support all kinds of metrics, the implementation is rather complicated, but the sync is done by the Metric class (where we wrap the implemented compute function to first sync the states and then compute afterwards). The best starting point is https://github.com/Lightning-AI/metrics/blob/master/src/torchmetrics/metric.py#L523

[astrobdr]

Thanks @justusschock ! Thats exactly the starting point I needed :)