Can not log metric which is a tensor

[sivakhno]

🐛 Bug

I get an error The metric val_acc_0_step/epoch_0 does not contain a single element thus it cannot be converted to float. when running training loop.

Full stack trace

 File "main.py", line 76, in <module>
    run_training()
  File "main.py", line 69, in run_training
    trainer.fit(model, dm)
  File "/opt/conda/lib/python3.6/site-packages/mlflow/utils/autologging_utils/safety.py", line 484, in safe_patch_function
    patch_function(call_original, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/mlflow/utils/autologging_utils/safety.py", line 241, in patch_with_managed_run
    result = patch_function(original, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/mlflow/pytorch/_pytorch_autolog.py", line 296, in fit
    return _run_and_log_function(self, original, args, kwargs)
  File "/opt/conda/lib/python3.6/site-packages/mlflow/pytorch/_pytorch_autolog.py", line 288, in _run_and_log_function
    result = original(self, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/mlflow/utils/autologging_utils/safety.py", line 440, in call_original
    original_result = original(*og_args, **og_kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 499, in fit
    self.dispatch()
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 546, in dispatch
    self.accelerator.start_training(self)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/accelerators/accelerator.py", line 73, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 114, in start_training
    self._results = trainer.run_train()
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 637, in run_train
    self.train_loop.run_training_epoch()
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/training_loop.py", line 577, in run_training_epoch
    self.trainer.run_evaluation(on_epoch=True)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 732, in run_evaluation
    self.evaluation_loop.log_evaluation_step_metrics(output, batch_idx)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 336, in log_evaluation_step_metrics
    self.__log_result_step_metrics(step_log_metrics, step_pbar_metrics, batch_idx)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 351, in __log_result_step_metrics
    self.trainer.logger_connector.log_metrics(metrics_by_epoch, {}, step=batch_idx)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py", line 222, in log_metrics
    scalar_metrics = self.trainer.metrics_to_scalars(metrics)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/logging.py", line 51, in metrics_to_scalars
    f"The metric `{k}` does not contain a single element"
pytorch_lightning.utilities.exceptions.MisconfigurationException: The metric `val_acc_0_step/epoch_0` does not contain a single element thus it cannot be converted to float. Found `tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0.])`

To Reproduce

I train the model below, which is multi-class / multi-dimensional classifier (have multiple class categories trained in one hierarchical model). The error seem to come from the following piece (running in loop is so that I can specify number of classes, which is different for different dimensions):

for i in range(len(y_true)):
            self.valid_acc[i](preds[i], y_true[i])
            self.log(f'val_acc_{i}', self.valid_acc[i], on_step=True, on_epoch=True)    

I was getting the same error when trying to save confusion matrix.

Code sample

class OntologyTaggerModel(pl.LightningModule):
    def __init__(self,
                 num_classes,
                 model_name='bert-base-cased',
                 learning_rate=3e-6,
                 **kwargs):

        super().__init__()
        self.save_hyperparameters()
        self.learning_rate = learning_rate
        self.model = BertForMulticlassSequenceClassification.from_pretrained(model_name, num_classes=num_classes)
        self.valid_acc = nn.ModuleList([torchmetrics.Accuracy(average=None, num_classes=num_class) for num_class in torch.tensor(num_classes)])
        self.valid_f1 = torchmetrics.F1(multiclass=True, mdmc_average='samplewise')
        self.cm = nn.ModuleList([torchmetrics.ConfusionMatrix(num_classes=num_class) for num_class in torch.tensor(num_classes)])
        

    def forward(self, *input, **kwargs):
        return self.model(*input, **kwargs)

    def training_step(self, batch, batch_idx):
        x, y_true = batch
        loss, _ = self(x, labels=y_true)
        return loss

    def validation_step(self, batch, batch_idx):
        x, y_true = batch
        _, y_preds = self(x, labels=y_true)
        preds = [torch.argmax(y_pred, axis=1) for y_pred in y_preds]
        for i in range(len(y_true)):
            self.valid_acc[i](preds[i], y_true[i])
            self.log(f'val_acc_{i}', self.valid_acc[i], on_step=True, on_epoch=True)        
        self.valid_f1(torch.stack(preds), torch.stack(y_true))
        self.log('f1', self.valid_f1, on_step=True, on_epoch=True)

    def configure_optimizers(self):
        'Prepare optimizer and schedule (linear warmup and decay)'
        return torch.optim.Adam(params=self.parameters(), lr=self.learning_rate)

    def training_epoch_end(self, training_step_outputs):
        avg_loss = torch.tensor([x['loss']
                                 for x in training_step_outputs]).mean()
        self.log('train_loss', avg_loss)

        print(f'###score: train_loss### {avg_loss}')

    def validation_epoch_end(self, val_step_outputs):
        for i in range(len(self.valid_acc)):
            acc = self.valid_acc[i].compute()
            self.log(f'val_score_{i}', acc)
        f1 = self.valid_f1.compute()
        self.log('f1', f1)
        print(f'###score: val_score### {acc}')

Expected behavior

Metrics should be rendered irrespective of dimension

Environment

pytorch-lightning==1.2.7
datasets==1.4.1
mlflow==1.16.0
torchmetrics=0.3.1
torch=1.7.1
python=3.6

[edgarriba]

@sivakhno according to the torchmetrics documentation, the Accuracy object can be instantiated with an explicit flag multiclass. Is there any specific reason why you are not doing in that way ? /cc @Borda @SkafteNicki

[sivakhno]

@edgarriba - good point. If I were to get one accuracy number across class this would be my option. But I'd like to get a breakdown by class labels and if I do so I need to specify num_classes but in my case these are multi-dimensional and hence num_classes is a tensor (the same logic would apply to confusion matrix).

[SkafteNicki]

since you set average=None when you initialize the Accuracy metric, the output will be the accuracy score calculated per class. As this is a non-scalar tensor and self.log is only intended to be used with scalar tensors, you get this error.

[sivakhno]

Thanks for the note @SkafteNicki - that explains the error, I though tensors can also be logged. So what could be my options if I want to log tensor? Could there be a patch I can add atop PL, like a wrapper function? I use MLflow Logger as a metrics backend with

set_mlflow_experiment()
mlflow.pytorch.autolog()
with mlflow.start_run() as run: 
    trainer.fit(model, dm)

and would like to keep it.
Thanks :)

[SkafteNicki]

assuming that you still want to log the accuracy per class (else you should really just set average='micro or macro), you logging code could look something like this:

for i in range(len(y_true)):
    self.valid_acc[i](preds[i], y_true[i])
    for j in range(self.num_classes):
        self.log(f'val_acc_{i}_{j}', self.valid_acc[i][j], on_step=True, on_epoch=True)