[Metrics] help with the use of AUROC for multi-label classification

[hkmztrk]

Hello all,

I am using pytorch-lightning==1.2.7 and I need a little bit guidance on using PL metrics. I was previously using sklearn metrics, but they seemed to be problematic because of numpy arrays, so I decided to switch PL metrics. Below is my current implementation. I only want to compute the metrics at epoch ends and I am using self.log to monitor the metrics but I am also using logging to print out the results at epoch ends.

According to the documentation:

From v1.2 onward compute() will no longer automatically call reset(), and it is up to the user to reset metrics between epochs, except in the case where the metric is directly passed to LightningModules self.log.

My question is whether my calling compute() function below means that I need to reset() them right after. I'd appreciate a lot if you can help me with this.

        logging.info(f'val auroc: {self.valid_auroc.compute()}')
        logging.info(f'val acc: {self.valid_acc.compute()}')

Another question is that since I'm doing multi-label classification (binary classification for n labels), e.g. [0, 1, 0, 0, 1], should I update initialisation as below (i.e. including pos_label tag)?

self.valid_auroc = pl.metrics.AUROC(num_classes=n_labels, pos_label=1)

Thanks a lot!

    def __init__(...):

        self.train_acc = pl.metrics.Accuracy()
        self.valid_acc = pl.metrics.Accuracy()
        self.valid_auroc = pl.metrics.AUROC(num_classes=n_labels) 


    def training_step(...):
        ... 
        self.train_acc(y_pred, y_true)
        self.log('train_acc', self.train_acc, on_step=True, on_epoch=True)
       # return loss

    def training_epoch_end(...):
           # self.log  loss

   def validation_step(...):
        ... 
        return {'val_loss': loss, 'y_pred': y_pred, 'y_true': y_true, 'pred_prob': pred_prob}

   def validation_epoch_end(self, outputs):
        avg_val_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        all_y_pred = torch.stack([x['y_pred'] for x in outputs]).view(-1)
        all_y_true = torch.stack([x['y_true'] for x in outputs]).view(-1)
        all_pred_prob = torch.stack([x['pred_prob'] for x in outputs]).view(-1)
        
        self.valid_acc(all_y_pred, all_y_true)
        self.valid_auroc(all_pred_prob, all_y_true)
        
        self.log('valid_acc', self.valid_acc, on_step=False, on_epoch=True)
        self.log('valid_auroc', self.valid_auroc, on_step=False, on_epoch=True)
        self.log('avg_val_loss', avg_val_loss, on_epoch=True, on_step=False)

        logging.info(f'mean val loss: {avg_val_loss}')
        logging.info(f'val auroc: {self.valid_auroc.compute()}')
        logging.info(f'val acc: {self.valid_acc.compute()}')
 

[edgarriba]

@hkmztrk According to the source code,
https://github.com/PyTorchLightning/metrics/blob/c65af2cfb8f12db718d016aa072d891030c91067/torchmetrics/classification/auroc.py#L169-L183

compute() takes the cached predictions and targets after the update and then computes the metric.