Would a PR for a general MultioutputWrapper be welcome?

[an1lam]

We've been using torchmetrics internally and enjoying it, but two things we've had to work around are:

1. Some metrics like SpearmanCorrcoef don't support multioutput mode (unlike R2Score for example).
2. Even some (all?) metrics that do support multioutput mode, don't have support for missing data (NaN labels for certain features is our use-case).

As a simple solution to this, we've implemented a MultioutputWrapper class (similar to BootstrapAggregator) that takes a base_metric and copies it num_outputs times, updating each underlying on each metric call. While we've tried to make our internal version as generic as possible, we've admittedly only tested it with the metrics we currently use. Examples include R2Score, SpearmanCorrcoef, and Accuracy with (N, C, O) (where O is number of outputs/tasks) dimensional predictions and (N, O) dimensional labels.

Since we've found this wrapper quite helpful, we'd like to contribute it back to the base package so others can use it. However, we expect it'll take a little work/feedback from the maintainers to get it in a place where it's ready for merging. For that reason, before even creating an issue/PR, I wanted to feel out whether this sort of class seems worth having in the base package and makes sense conceptually.

Thanks in advance!

[SkafteNicki]

Hi @an1lam,
I think that sounds like a great addition. Even if we implemented this kind of feature in all our own metrics, I still see this kind of wrapper having some value for users that have custom metrics where they are also looking to extend to multi-output.
Please feel free to send a PR with what you already have. Then we can begin the process of reviewing :]

[an1lam]

Awesome! Will do some time this week!

[an1lam]

@SkafteNicki FYI: I created an issue (#508) and PR (#510) for this.