Logging Multi-Label Metrics

[deepbakes]

Hey, I was wondering if there was a correct way to evaluate and log multi-class metrics at the end of an epoch, such that the metrics could be evaluated across multiple GPUs using ddp?

Typically, if I was looking at a image classification problem I could use the following:

self.train_average_accuracy = torchmetrics.Accuracy(num_classes=self.num_classes)

x, y = batch
outputs = self(x)  # Evaluate logits.
sigmoid_outputs = torch.sigmoid(outputs)  # Produce sigmoid outputs for multi-label metrics.
loss = self.compute_loss(outputs, y)
average_acc = self.train_average_accuracy(sigmoid_outputs, y)

I could log the either with single GPU or multi-GPU accuracy using:

self.log(
"train_average_acc",
average_acc,
on_step=False,
on_epoch=True,
prog_bar=True,
 sync_dist=self.multi_gpu)

However, if I try modifying the accuracy metric to use multi-label accuracy using:

torchmetrics.Accuracy(threshold=0.5, average=None, num_classes=self.num_classes)

I find that the metric is logged as nan, which is I'm guessing due to the tensor which contains the class wide accuracies not updating between batches. I've also tried to get around this I ended up passing y and y_pred into the def validation_epoch_end(self, outputs) and evaluating the metric using concatenated batches. However, I still find nan values logged. Is there any other way of logging the multi-label metrics other than, say iterating over the tensor containing the class level metrics?

[deepbakes]

As an update, I managed to develop a work around by iterating over the tensor that contained multi-label metrics that are evaluated on a class level. I.e, this function coulds do it

    for idx, metric_value in enumerate(class_metrics):  # Iterate over tensor elements.
        if mode == "debugging":
            print(f"{phase}_{metric}_{class_labels[idx]}: {metric_value}")

        else:  # Note we cannot have forward slashes within class_labels when logging.
            self.log(
                f"{phase}_{metric}_{class_labels[idx]}",
                metric_value,
                on_step=False,
                on_epoch=True,
                sync_dist=self.multi_gpu,
            )

Where the metrics are evaluated on a class wise level using.

average_acc = self.train_average_accuracy(sigmoid_outputs, y)

and the logging function can be placed within the training or validation step using

log_class_metric(self, class_metrics=class_acc, metric="acc", phase="train")